== PS1 IPP Czar Logs for the week 2013.01.14 - 2013.01.20 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2013.01.14 ===
 * 00:25 MEH: removing stare from stdscinece for lap_staticsky_notgp may be too many nodes lost for stdscience to keep up with observations
 * 09:05 Bill: set pantasks lap_staticsky_notgp to stop. The galactic plane skycells are done and we can resume standard operation of deepstack
 * 09:?? Bill: set label for the staticsky jobs with label LAP.ThreePi.20120706.gp back to LAP.ThreePi.20120706
 * 11:48 Bill: there are 6 staticsky jobs that are at the "write the outputs" step but haven't made progress since ~9:15-9:30
 * 13:40 Bill: the staticsky jobs were all stuck in a soap call trying to create their output cmf files. Killed off the processes
 * 13:45 Bill: started deepstack pantasks with 3 x compute3 nodes
 * 15:30 MEH: throwing in the 2x wave4 to deepstack for staticsky as well
 * 20:25 MEH: removing 2x wave4 from deepstack to run in other pantasks to redo MD staticsky
 * 23:05 MEH: looks like ipp013 crashed ~21:33 with GPF [wiki:ipp013-crash-20130114], power cycle
=== Tuesday : 2013.01.15 ===
 * 10:30 MEH: stealing stare nodes from stdscience (turned off), since it isn't doing anything, for MD staticsky. might as well do all of wave4 as well..
 * 14:25 MEH: some odd missing files from ipp028 found today, Gene looking at links to fix. stdscience+cleanup stopped, chip.revert.off for a bit. Gene fixed and MOPSreq.czw chips moving through
 * 15:30 Serge: Stopped cleanup so that ippdb02 can catch up with its master
 * 16:30 Serge: Queued a bunch of diffs to be reprocessed for mops (warp label = czw.mopstest.rerun)
 * 18:50 MEH: returning wave4,stare to stdscience -- keeping the 2x wave4 from stack however
=== Wednesday : 2013-01-16 ===
Bill is czar today
 * 06:55 Serge: ippdb02 caught up -> Restarted cleanup
 * 09:30 set deepstack pantasks to stop. There are a couple of skycells in 267 < RA < 303 abs(glat) < 15 that need to run so I'm going to reduce the number of hosts and change the label.
 * 10:40 Determined that there are also runs with abs(glat) > 15 that need to run so decided to let those run first (along with the others queued with ra < 267. We'll rerun the galactic plane skycells later, hopefully after the psLib performance problem gets fixed.

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===