PS1 IPP Czar Logs for the week 2013.05.27 - 2013.06.02

(Up to PS1 IPP Czar Logs)

Monday : 2013.05.27

  • 02:30 MEH: ipp033 crashed/unresponsive ~4hrs ago.. holding up nightly download/processing.. rebooting
  • 03:00 MEH: system still wedged.. restarting summitcopy and registration to clear queues and logs to watch
  • 03:20 MEH: still wedged.. nfs trouble on ipp033 still, restarting and looks to be clearing
  • 08:50 MEH: stdscience struggling.. doing a regular restart to finish up nightly science
  • 13:00 MEH: odd SMF full but MIA stalling warp -- manually remaking
    neb entry neb://any/gpc1/ThreePi.nt/2013/05/27//o6439g0376o.615201/ not found
    perl ~ipp/src/ipp-20130307/tools/ --redirect-output --cam_id 816310

Tuesday : 2013.05.28

Bill is czar today

  • 12:08 queued STS exposures from 2009 and 2010 to be processed continuing to use label STS.rp.20130508
    • MEH: since not running deep stacks so might as well add compute3 back into stdscience
  • 20:00 MEH: looks like CZW is running stacks on 2x wave3, removing from normal processing, registration was struggling to get things through. wave3 back in after stacks stopped

Wednesday : 2013.05.29

Bill is czar today

  • 08:40 turned chip off to let some of the backed up warps make better progress. The sts data is actually moving through rather well.
  • 08:45-09:05 repaired a number of broken raw files and burntool tables
    perl ~bills/ipp/tools/fixburntool -e 98964 -c XY14 gpc1/20100621/o5368g0450o/o5368g0450o.ota52.fits
    perl ~bills/ipp/tools/fixburntool -e 190536 -c XY53
    perl ~bills/ipp/tools/fixburntool -e 190599 -c XY52 gpc1/20100711/o5388g0173o/o5388g0173o.ota40.fits
    perl ~bills/ipp/tools/fixburntool -e 193379 -c XY65 gpc1/20100711/o5388g0173o/o5388g0173o.ota40.fits gpc1/20100804/o5412g0105o/o5412g0105o.ota57.fits gpc1/20100804/o5412g0141o/o5412g0141o.ota37.fits gpc1/20100815/o5423g0285o/o5423g0285o.ota37.fits
  • 09:06 dropped stack 2231496 a skycell from May 3 that repeatedly faults
  • 09:09 restarted distribution pantasks which died about 20 minutes ago with "controller still not responding, giving up"
  • 09:15 set data with label like ps_ud% to be cleaned
  • 09:25 restarted stdscience pantasks with chip.on
  • 11:40 MEH: won't be starting deepstacks for a bit still, manually adding compute3 into stdsci again

Thursday : 2013.05.30

Friday : 2013.05.31

Saturday : 2013.06.01

Sunday : 2013.06.02