[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2012.12.31 ===
 * 11:40 MEH: no MD SSdiffs made, oddly MD nightly stacks weren't made last night. ns.stacks.run many errors, restarting stdscience. still getting ''failure for: nightly_science.pl --queue_stacks --date 2012-12-31 --dbname gpc1''
  * ESS.nightlyscience is having trouble and blocking? removing label -- no help, must be over-riding/hardcoded somewhere..
  * stacktool command from nightly_science.pl is using -skycell and not -select_skycell.. looks like hadn't used the skycell selection option before?
  * fixing nightly_science.pl to use -select_skycell_id.. 
  * many timeouts but finally queued stacks @13:55, will take ~1-2 hrs before MD SSdiffim are ready, 300 ESS stacks to make first. tweak_ssdiff for 1500-1700 before nightly science..
=== Tuesday : 2013.01.01 ===
 * 11:35 MEH: again no MD SSdiffims, looks like nightly_science.pl is still getting hung up for nightly stacks (and time critical SSdiffims) with a timeout now
  * raised timeout for ns.stack.run to 480 and queued stacks -- finished @14:00
  * tweak_ssdiff for 1400--1600 before nightly science again
=== Wednesday : 2013.01.02 ===
 * 16:00 MEH: queuing up new MD07 refstack exposures to keep the nodes busy
=== Thursday : 2013.01.03 ===
 * 08:35 MEH: chip.off to throw all nodes at remaining nightly warps.
 * 09:30 MEH: chip.revert.off to fix MD07 problems
 * 09:50 MEH: need to restart mysql on ippb machines to search for missing/0 size files
=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===