PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2011-07-11

• 17:00 Serge: o5753g0178o stuck at warp stage. I ran perl

  runwarpskycell.pl --warp_id 220303 --skycell_id skycell.2615.054
  

• 17:04 Serge: o5753g0096o stuck at fake stage. pantasks.stdout.log reported crash. I ran the same command again and it seems to be working:

  fake_imfile.pl --exp_id 360038 --fake_id 217175 --class_id XY42 --chiproot=neb://ipp026.0/gpc1/MD07.nt/2011/07/11//o5753g0096o.360038/o5753g0096o.360038.ch.255145
     --camroot=neb://any/gpc1/MD07.nt/2011/07/11//o5753g0096o.360038/o5753g0096o.360038.cm.232378 --camera GPC1 
     --outroot neb://ipp026.0/gpc1/MD07.nt/2011/07/11//o5753g0096o.360038/o5753g0096o.360038.fk.217175 --dbname gpc1 --verbose
  

Tuesday : 2011.07.12

• 08:00: Roy: Everything downloaded from summit, but quite a few faults. Will investigate...
• 09:55: Serge: Manually reverted failed publish to IPP-MOPS-TEST -> pubtool -revert -dbname gpc1 -client_id 5
• 10:03: Roy: stdscience died. No idea why. Restarted.
• 14:30: Roy: ipp026 load up to 130....

Comment from Serge: From ​http://ipp004.ifa.hawaii.edu/clusterMonitor/top-20110712T1500.html the load can clearly be attributed to nfs

• 14:55: Serge: Ran a bunch of commands to unstuck MOPS data (difftool -revertdiffskyfile -fault 2 -label ThreePi.nightlyscience -dbname gpc1; warptool -revertwarped -fault 4 -label ThreePi.nightlyscience -dbname gpc1; pubtool -revert -dbname gpc1 -client_id 5)

• 15:00 heather restarted stack because no LAP.ThreePi's were getting queued. They still aren't, and further investigations (through labeltool) show them to not be in active state. The labels are in stdsci, but they won't be processed until the state changes to active.
• 15:05: Roy: Stopped all pantasks servers
• 15:11 Serge: diff log complains about missing warp files I tried: perl runwarpskycell.pl --warp_id 220738 --skycell_id skycell.1996.132
• 15:25: Gene: Rebooted ipp026
• 15:30: Roy: ipp026 backup, Started all pantasks servers
• 15:55: Roy: attempted to revert destreaks with:

magicdstool -clearstatefaults -state failed_revert -set_state new -label 'Threepi.nightlyscience' -dbname gpc1

Didn't seem to do anything. Followed instructions ​here instead. They seem to be clearing.

• 16:30: Roy: warps have finally been reverted and are being processed again.
• 18:00 heather reverted stacks - it says fault 4 but I don't believe it.
• 18:00 heather neb-rm -m'd gpc1/ThreePi.nt/2011/07/12/o5754g0471o.360705/o5754g0471o.360705.wrp.220745.skycell.1320.071.log, which had a nebulous entry but no corresponding file.

Wednesday : 2011.07.13

• 07:15: Roy: Only 88 of 496 science exposures downloaded from last night, even though summitcopy is apparently running. Unfinished postage stamp requests continues to grow....
• 08:11: CZW: regpeek.pl suggested that o5755g0120o/XY43 was stuck. Reverted using regtool -updateprocessedimfile -exp_id 360887 -class_id XY43 -set_state pending_burntool. Watching pantasks showed continuing errors, and attempts to read the burntool log (gpc1/20110713/o5755g0120o/o5755g0120o.ota43.burn.log) resulted in nebulous unavailable instance errors. I didn't check if there was a nebulous host down, but simply moved the log and table (also unavailable) to FILE.bak to allow burntool to write to that location. Exposures seem to be registering now.
• 08:30: Roy: Gene rebooted ipp026. It had been down for 10 hours

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD