== PS1 IPP Czar Logs for the week 2011-04-25 - 2011-05-01 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2011-04-25 ===
Bill is czar today.

 * 09:00 The 576 STS.2010.b exposures queued for reprocessing so far have finished and have been posted to the distribution server. Queued the chip, warp, and diff runs for cleanup
 * 11:58 set pantasks' to run. They were stopped to allow Chris to make some changes to nebulous.
 * 13:56 CZW: ATRC storage nodes completely added to nebulous. replication pantasks is stopped, as we want to do some replication tests to send data to these disks.
=== Tuesday : 2011-04-26 ===
 * 08:00 Bill burntool was stuck due to a database error that left a rawImfile in an incorrect state. Fixed by running the command that failed by hand.
{{{
regtool -updateprocessedimfile -dbname gpc1 -exp_id 330078 -class_id 'XY46' -burntool_state 0 -set_state pending_burntool

}}}
 * 08:05 Bill warp was stuck because the book warpPendingSkyCell was full of entries in pantaskState DONE. Fixed by running warp.reset stdscience.
 * 09:32 ipp041 is generating lots of faults because it cannot see /data/ippb02.0. force.umount hung. Running  sudo umount -f -v /data/ippb02.2 hung as well. Set ipp041 to off in stdscience.
 * 10:56 Heather snuck in some more jt stacks. (MD02.jtrp)
 * 11:15 Serge tries to replicate ippc02.gpc1 onto ipp001. There might be unusual activity between them.

=== Wednesday : 2011-04-27 ===
 * 07:20 Bill noticed that warp was stuck. warpPendingSkyCell book full of DONE jobs. warp.off wait for any running jobs to finish ; warp.reset ; warp.on
 * 07:30 interestingly the czartool graph for warp did not flatten out even though there were no warp jobs running. Perhaps I caught it just before it would have flat lined.
 * 10:27 Bill set label STS.2010.b to inactive to prevent chip processing from slowing down last night's science processing.
 * 10:31 Bill set ipp012 and ipp020 nebulous state to up (they were set to repair over the weekend)
 * 13:45 gpc1 database restarted, stdscience restarted (to avoid timeout counts disturbing Nrun counts)
=== Thursday : 2011-04-28 ===

=== Friday : 2011-04-29 ===

=== Saturday : 2011-04-30 ===

=== Sunday : 2011-05-01 ===