(Up to PS1 IPP Czar Logs)

Monday : 2012.12.31

• 11:40 MEH: no MD SSdiffs made, oddly MD nightly stacks weren't made last night. ns.stacks.run many errors, restarting stdscience. still getting failure for: nightly_science.pl --queue_stacks --date 2012-12-31 --dbname gpc1
  • ESS.nightlyscience is having trouble and blocking? removing label -- no help, must be over-riding/hardcoded somewhere..
  • stacktool command from nightly_science.pl is using -skycell and not -select_skycell.. looks like hadn't used the skycell selection option before?
  • fixing nightly_science.pl to use -select_skycell_id..
  • many timeouts but finally queued stacks @13:55, will take ~1-2 hrs before MD SSdiffim are ready, 300 ESS stacks to make first. tweak_ssdiff for 1500-1700 before nightly science..

Tuesday : 2013.01.01

• 11:35 MEH: again no MD SSdiffims, looks like nightly_science.pl is still getting hung up for nightly stacks (and time critical SSdiffims) with a timeout now
  • raised timeout for ns.stack.run to 480 and queued stacks -- finished @14:00
  • tweak_ssdiff for 1400--1600 before nightly science again

Wednesday : 2013.01.02

• 16:00 MEH: queuing up new MD07 refstack exposures to keep the nodes busy

Thursday : 2013.01.03

• 08:35 MEH: chip.off to throw all nodes at remaining nightly warps.
• 09:30 MEH: chip.revert.off to fix MD07 problems
• 09:50 MEH: need to restart mysql on ippb machines to search for missing/0 size files

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD