== PS1 IPP Czar Logs for the week 2012.04.23 - 2012.04.29 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.04.23 ===
 Mark czar tagging in for Roy
 * 08:08 Bill: two chip_imfile processes have been running since around midnight making no progress. Killed the processes and reverted the faults. They quickly finished. These are the case of the 2 three pi exposures that haven't finished.
 * 09:20 Mark: covering for Roy, stare03 out of processing for disk install
 * 11:15 stare03 disks installed, back into processing. 
 * 12:00 with ganglia DOA, manually keeping eye on systems while MD08.GR0-z chip-warp runs with
{{{
~ipp/who_uses_the_cluster/sshToNodes.py ipp 'uptime'
}}}
 * 17:10 going to restart stdscience once MD08.GR0-z finished, pcontrol running 100% cpu but still peaked @100 chip exposures/hr. will queue up MD08.GR0-y afterwards to run along nightly science and finish up.
 * 20:30 stdscience dragging even after restart, try restart again. looked to be a pileup of jobs wanting ipp039.
 * 22:30 with the new nodes, able to run MD08.GR0-y processing alongside nightly science and finish ~500 exposures chip->warp before morning.
=== Tuesday : 2012.04.24 ===
 Mark is czar
 * 07:00 nightly science still downloading, full night of observations but seems slower than normal. been seeing chip_imfile/chip stage taking >1ks on various machines, also seems longer than normal -- look like some crowded fields. 
 * 09:30 nightly science 99.9% done
 * 10:45 destreak_restore has been holding up the remaining 3PI nightly science and time sensitive MD SSdiffs from distribution.. setting destreak.revert.off until nightly science out.
 * 11:15 nightly science distribution finished, flipping destreak.revert.on
 * 14:30 MD08.GR0 is supposed to have >3k stacks running..
=== Wednesday : YYYY.MM.DD ===
 * 06:00 Mark: registration stuck ~half the night, had to run 
{{{
regtool -updateprocessedimfile -exp_id 480743 -class_id XY03 -set_state pending_burntool -dbname gpc1
}}}
=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===