PS1 IPP Czar Logs for the week 2011-04-04 2011-04-10

(Up to PS1 IPP Czar Logs)

Monday : 2011.04.04

Serge is czar. Almost no data last night.

  • 06:26: distribution crashed. I restarted it
  • 10:49: rebuild ippScripts to catch summit_copy.pl change that should fix undownloadable image problem that popped up over the weekend.
  • 13:07: Bills queued 54 diffs with label ThreePi?.rerun. When finished these should also generate some magic and destreak runs.

Tuesday : 2011.04.05

  • No data last night. ipp021 crashed.
  • Bill is running masktest data out ~bills/masktest from a pantasks on ippc25
  • 15:20 dist.revert off. There are some repeating failures due to corrupt files that will take some time to repair
  • 19:40 queued massive cleanup of data updated for the postage stamp server.
  • 21:55 I, Snorkel Bob - I mean, the czar - happened to check and found that stdscience had crashed. Restarted it.
  • 22:15 set the distRuns that are broken to $label.wait and turned dist.revert back on

Wednesday : 2011.04.06

  • production build switched to ipp-20110406

Thursday : 2011.04.07

  • running STS.2010.a

Friday : 2011.04.08

Saturday : 2011.04.09

  • 02:09: CZW: unstuck registration. See ticket #1477.
  • 07:00 Bill: update pantasks was stuck. Lots of books with entries and no jobs running. Restarted pantasks and the jobs finished.

Sunday : 2011.04.10