PS1_IPP_Czarlog_20131014 – Pan-STARRS IPP

Context Navigation

Version 14 (modified by bills, 13 years ago) ( diff )
--

Happy Discoverers Day!

06:50 Bill: ipp055 is down which has ground the entire cluster to a halt. Set to down in nebulous
- Users were waiting for the postage stamp server so I restarted it, and the update pantasks.
- stdscience was a backlogged mess so I shut it down. The space by nodes chart is a disaster. Other pantasks set to stop to let the jobs clear
07:05 Bill:
- Gene got ipp055 up again. set it to repair in nebulous
- "who uses cluster" list looks clean. Probably safe to restart the pantasks. Restarted stdscience.

14:10 MEH: in case missed it last week czarlog, deepstack is running and will be running for quite some time unless priorities shift. was setup w/ 2x compute3 from stack and uses 40-70% RAM on these machines (48GB), it may be possible to add 1x compute 3 back to stack (for nightly and LAP stacks) -- will manually add and monitor.

heather turned off pantasks at 11:37 so rita could restart ipp037, she then turned pantasks back to run
heather notes something happened at 12:30ish and loaders stalled... ?
mid-afternoon (heather+MEH) ipp062 was in a funky state (it flatlined about noonish?). MEH got nfs going again, heather restarted loaders.
19:00 MED restarting stdscience
22:00 MEH cleared backlog of 30 exposures in registration (fault+check_burntool)

7:30am HAF unsticks registration and restarts it
7:45 Gavin reboots ipp063 - this is why registration was stuck. Heather needs to check/restart loaders
- MEH: ipp063 logs show it didn't stop responding until ~06:38, mostly after observing and doesn't seem to be the direct cause of stalling registration since >200 exposures were backed up.
8:25 Bill (the czar) just woke up.
- fixed (the latest?) cause for registration stalling with: regtool -updateprocessedimfile -set_state pending_burntool -burntool_state 0 -exp_id 665227 -class_id XY27