== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : YYYY.MM.DD ===
Happy Discoverers Day!

 * 06:50 Bill: ipp055 is down which has ground the entire cluster to a halt. Set to down in nebulous
   * Users were waiting for the postage stamp server so I restarted it, and the update pantasks.
   * stdscience was a backlogged mess so I shut it down. The space by nodes chart is a disaster. Other pantasks set to stop to let the jobs clear
 * 07:05 Bill: 
   * Gene got ipp055 up again. set it to repair in nebulous
   * "who uses cluster" list looks clean. Probably safe to restart the pantasks. Restarted stdscience.
=== Tuesday : 2013.10.15 ===
 * 14:10 MEH: in case missed it last week czarlog, deepstack is running and will be running for quite some time unless priorities shift. was setup w/ 2x compute3 from stack and uses 40-70% RAM on these machines (48GB), it may be possible to add 1x compute 3 back to stack (for nightly and LAP stacks) -- will manually add and monitor.

 * heather turned off pantasks at 11:37 so rita could restart ipp037, she then turned pantasks back to run
 * heather notes something happened at 12:30ish and loaders stalled...  ?
 * mid-afternoon (heather+MEH) ipp062 was in a funky state (it flatlined about noonish?). MEH got nfs going again, heather restarted loaders.
 * 19:00 MED restarting stdscience
 * 22:00 MEH cleared backlog of 30 exposures in registration (fault+check_burntool)

=== Wednesday : YYYY.MM.DD ===

 * 7:30am HAF unsticks registration and restarts it
 * 7:45 Gavin reboots ipp063 - this is why registration was stuck. Heather needs to check/restart loaders 
  * MEH: ipp063 logs show it didn't stop responding until ~06:38, mostly after observing and doesn't seem to be the direct cause of stalling registration since >200 exposures were backed up. 
 * 8:25 Bill (the czar) just woke up.
   * fixed (the latest?) cause for registration stalling with:  regtool -updateprocessedimfile -set_state pending_burntool -burntool_state 0 -exp_id 665227 -class_id XY27
=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===