== PS1 IPP Czar Logs for the week 2013.05.27 - 2013.06.02 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013.05.27 ===
 * 02:30 MEH: ipp033 crashed/unresponsive ~4hrs ago.. holding up nightly download/processing.. rebooting
 * 03:00 MEH: system still wedged.. restarting summitcopy and registration to clear queues and logs to watch
 * 03:20 MEH: still wedged.. nfs trouble on ipp033 still, restarting and looks to be clearing
 * 08:50 MEH: stdscience struggling.. doing a regular restart to finish up nightly science
 * 13:00 MEH: odd SMF full but MIA stalling warp -- manually remaking
{{{
neb entry neb://any/gpc1/ThreePi.nt/2013/05/27//o6439g0376o.615201/o6439g0376o.615201.cm.816310.smf not found

perl ~ipp/src/ipp-20130307/tools/runcameraexp.pl --redirect-output --cam_id 816310
}}}
=== Tuesday : 2013.05.28 ===
Bill is czar today
 * 12:08 queued STS exposures from 2009 and 2010 to be processed continuing to use label STS.rp.20130508
  * MEH: since not running deep stacks so might as well add compute3 back into stdscience 
 * 20:00 MEH: looks like CZW is running stacks on 2x wave3, removing from normal processing, registration was struggling to get things through
=== Wednesday : 2013.05.29 ===
Bill is czar today
 * 08:40 turned chip off to let some of the backed up warps make better progress. The sts data is actually moving through rather well. 
 * 08:45-09:05 repaired a number of broken raw files and burntool tables
{{{
perl ~bills/ipp/tools/fixburntool -e 98964 -c XY14
deneb-locate.py gpc1/20100621/o5368g0450o/o5368g0450o.ota52.fits
perl ~bills/ipp/tools/fixburntool -e 190536 -c XY53
perl ~bills/ipp/tools/fixburntool -e 190599 -c XY52
deneb-locate.py gpc1/20100711/o5388g0173o/o5388g0173o.ota40.fits
perl ~bills/ipp/tools/fixburntool -e 193379 -c XY65
deneb-locate.py gpc1/20100711/o5388g0173o/o5388g0173o.ota40.fits
deneb-locate.py gpc1/20100804/o5412g0105o/o5412g0105o.ota57.fits
deneb-locate.py gpc1/20100804/o5412g0141o/o5412g0141o.ota37.fits
deneb-locate.py gpc1/20100815/o5423g0285o/o5423g0285o.ota37.fits
}}}
 * 09:06 dropped stack 2231496 a skycell from May 3 that repeatedly faults

=== Thursday : 2013.05.30 ===

=== Friday : 2013.05.31 ===

=== Saturday : 2013.06.01 ===

=== Sunday : 2013.06.02 ===