PS1 IPP Czar Logs for the week 2013.05.27 - 2013.06.02

(Up to PS1 IPP Czar Logs)

Monday : 2013.05.27

• 02:30 MEH: ipp033 crashed/unresponsive ~4hrs ago.. holding up nightly download/processing.. rebooting
• 03:00 MEH: system still wedged.. restarting summitcopy and registration to clear queues and logs to watch
• 03:20 MEH: still wedged.. nfs trouble on ipp033 still, restarting and looks to be clearing
• 08:50 MEH: stdscience struggling.. doing a regular restart to finish up nightly science
• 13:00 MEH: odd SMF full but MIA stalling warp -- manually remaking

  neb entry neb://any/gpc1/ThreePi.nt/2013/05/27//o6439g0376o.615201/o6439g0376o.615201.cm.816310.smf not found
  
  perl ~ipp/src/ipp-20130307/tools/runcameraexp.pl --redirect-output --cam_id 816310
  

Tuesday : 2013.05.28

Bill is czar today

• 12:08 queued STS exposures from 2009 and 2010 to be processed continuing to use label STS.rp.20130508
  • MEH: since not running deep stacks so might as well add compute3 back into stdscience
• 20:00 MEH: looks like CZW is running stacks on 2x wave3, removing from normal processing, registration was struggling to get things through

Wednesday : 2013.05.29

Bill is czar today

• 08:40 turned chip off to let some of the backed up warps make better progress. The sts data is actually moving through rather well.
• 08:45-09:05 repaired a number of broken raw files and burntool tables

  perl ~bills/ipp/tools/fixburntool -e 98964 -c XY14
  deneb-locate.py gpc1/20100621/o5368g0450o/o5368g0450o.ota52.fits
  perl ~bills/ipp/tools/fixburntool -e 190536 -c XY53
  perl ~bills/ipp/tools/fixburntool -e 190599 -c XY52
  deneb-locate.py gpc1/20100711/o5388g0173o/o5388g0173o.ota40.fits
  perl ~bills/ipp/tools/fixburntool -e 193379 -c XY65
  deneb-locate.py gpc1/20100711/o5388g0173o/o5388g0173o.ota40.fits
  deneb-locate.py gpc1/20100804/o5412g0105o/o5412g0105o.ota57.fits
  deneb-locate.py gpc1/20100804/o5412g0141o/o5412g0141o.ota37.fits
  deneb-locate.py gpc1/20100815/o5423g0285o/o5423g0285o.ota37.fits
  

• 09:06 dropped stack 2231496 a skycell from May 3 that repeatedly faults

Thursday : 2013.05.30

Friday : 2013.05.31

Saturday : 2013.06.01

Sunday : 2013.06.02