PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2011-07-11

• 17:00 Serge: o5753g0178o stuck at warp stage. I ran perl

  runwarpskycell.pl --warp_id 220303 --skycell_id skycell.2615.054
  

• 17:04 Serge: o5753g0096o stuck at fake stage. pantasks.stdout.log reported crash. I ran the same command again and it seems to be working:

  fake_imfile.pl --exp_id 360038 --fake_id 217175 --class_id XY42 --chiproot=neb://ipp026.0/gpc1/MD07.nt/2011/07/11//o5753g0096o.360038/o5753g0096o.360038.ch.255145
     --camroot=neb://any/gpc1/MD07.nt/2011/07/11//o5753g0096o.360038/o5753g0096o.360038.cm.232378 --camera GPC1 
     --outroot neb://ipp026.0/gpc1/MD07.nt/2011/07/11//o5753g0096o.360038/o5753g0096o.360038.fk.217175 --dbname gpc1 --verbose
  

Tuesday : 2011.07.12

• 08:00: Roy: Everything downloaded from summit, but quite a few faults. Will investigate...
• 09:55: Serge: Manually reverted failed publish to IPP-MOPS-TEST -> pubtool -revert -dbname gpc1 -client_id 5
• 10:03: Roy: stdscience died. No idea why. Restarted.
• 14:30: Roy: ipp026 load up to 130....

Comment from Serge: From ​http://ipp004.ifa.hawaii.edu/clusterMonitor/top-20110712T1500.html the load can clearly be attributed to nfs

• 14:55: Serge: Ran a bunch of commands to unstuck MOPS data (difftool -revertdiffskyfile -fault 2 -label ThreePi.nightlyscience -dbname gpc1; warptool -revertwarped -fault 4 -label ThreePi.nightlyscience -dbname gpc1; pubtool -revert -dbname gpc1 -client_id 5)

• 15:00 heather restarted stack because no LAP.ThreePi's were getting queued. They still aren't, and further investigations (through labeltool) show them to not be in active state. The labels are in stdsci, but they won't be processed until the state changes to active.
• 15:05: Roy: Stopped all pantasks servers
• 15:11 Serge: diff log complains about missing warp files I tried: perl runwarpskycell.pl --warp_id 220738 --skycell_id skycell.1996.132
• 15:25: Gene: Rebooted ipp026
• 15:30: Roy: ipp026 backup, Started all pantasks servers
• 15:55: Roy: attempted to revert destreaks with:

magicdstool -clearstatefaults -state failed_revert -set_state new -label 'Threepi.nightlyscience' -dbname gpc1

Didn't seem to do anything. Followed instructions ​here instead. They seem to be clearing.

• 16:30: Roy: warps have finally been reverted and are being processed again.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD