PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

Happy Discoverers Day!

• 06:50 Bill: ipp055 is down which has ground the entire cluster to a halt. Set to down in nebulous
  • Users were waiting for the postage stamp server so I restarted it, and the update pantasks.
  • stdscience was a backlogged mess so I shut it down. The space by nodes chart is a disaster. Other pantasks set to stop to let the jobs clear
• 07:05 Bill:
  • Gene got ipp055 up again. set it to repair in nebulous
  • "who uses cluster" list looks clean. Probably safe to restart the pantasks. Restarted stdscience.

Tuesday : 2013.10.15

• 14:10 MEH: in case missed it last week czarlog, deepstack is running and will be running for quite some time unless priorities shift. was setup w/ 2x compute3 from stack and uses 40-70% RAM on these machines (48GB), it may be possible to add 1x compute 3 back to stack (for nightly and LAP stacks) -- will manually add and monitor.

• heather turned off pantasks at 11:37 so rita could restart ipp037, she then turned pantasks back to run
• heather notes something happened at 12:30ish and loaders stalled... ?
• mid-afternoon (heather+MEH) ipp062 was in a funky state (it flatlined about noonish?). MEH got nfs going again, heather restarted loaders.
• 19:00 MED restarting stdscience
• 22:00 MEH cleared backlog of 30 exposures in registration (fault+check_burntool)

Wednesday : YYYY.MM.DD

• 7:30am HAF unsticks registration and restarts it
• 7:45 Gavin reboots ipp063 - this is why registration was stuck. Heather needs to check/restart loaders
  • MEH: ipp063 logs show it didn't stop responding until ~06:38, mostly after observing and doesn't seem to be the direct cause of stalling registration since >200 exposures were backed up.

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD