PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2011-07-11

• 17:00 Serge: o5753g0178o stuck at warp stage. I ran perl

  runwarpskycell.pl --warp_id 220303 --skycell_id skycell.2615.054
  

• 17:04 Serge: o5753g0096o stuck at fake stage. pantasks.stdout.log reported crash. I ran the same command again and it seems to be working:

  fake_imfile.pl --exp_id 360038 --fake_id 217175 --class_id XY42 --chiproot=neb://ipp026.0/gpc1/MD07.nt/2011/07/11//o5753g0096o.360038/o5753g0096o.360038.ch.255145
     --camroot=neb://any/gpc1/MD07.nt/2011/07/11//o5753g0096o.360038/o5753g0096o.360038.cm.232378 --camera GPC1 
     --outroot neb://ipp026.0/gpc1/MD07.nt/2011/07/11//o5753g0096o.360038/o5753g0096o.360038.fk.217175 --dbname gpc1 --verbose
  

Tuesday : 2011.07.12

• 08:00: Roy: Everything downloaded from summit, but quite a few faults. Will investigate...
• 09:55: Serge: Manually reverted failed publish to IPP-MOPS-TEST -> pubtool -revert -dbname gpc1 -client_id 5
• 10:03: Roy: stdscience died. No idea why. Restarted.
• 14:30: Roy: ipp026 load up to 130....

Comment from Serge: From ​http://ipp004.ifa.hawaii.edu/clusterMonitor/top-20110712T1500.html the load can clearly be attributed to nfs

• 14:55: Serge: Ran a bunch of commands to unstuck MOPS data (difftool -revertdiffskyfile -fault 2 -label ThreePi.nightlyscience -dbname gpc1; warptool -revertwarped -fault 4 -label ThreePi.nightlyscience -dbname gpc1; pubtool -revert -dbname gpc1 -client_id 5)

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD