PS1 IPP Czar Logs for the week 2015.01.26 - 2015.02.01

(Up to PS1 IPP Czar Logs)

Monday : 2015.01.26

• 04:30 EAM: ipp044 crashed, nothing on console. rebooting.
• 04:45 EAM: ipp044 having trouble on reboot. I set it down in nebulous.
• 05:00 EAM: ipp044 still not returning. i've shutdown stdscience and i'm clearing hung nfs mounts
• 05:40 EAM: ipp044 mounts on the storage nodes are clear, stdscience restart. clearing computes as well
• 05:50 EAM: looks like everything is clear, ipp044 is still down. stdscience, stdlocal, stdlanl are all back up and running.
• 07:30 MEH: manually had to clear exposure fault in registration for o7048g0547o

  regtool -dbname gpc1 -revertprocessedexp -exp_id 863126
  

• 08:00 EAM: stopping and restarting stdlocal (110k warps)
  • there were various hung jobs due to hung ipp044 mount points on the stdlocal x-nodes. I've cleared them out.
• 10:05 MEH: nightly finished, adding storage nodes to stdlocal storage.hosts.on
• 11:35 MEH: ipp094 Haydn releasing for IPP use now, BBU not ready for data targeting but will put into group s5 for processing.
  • also adding for neb-host, may be ok off and on for non-targeted data
• 14:15 MEH: ipp090,093,095,082,080 typically have heavy loads and also seem to be doing some sort of raid verification, put to repair for now
  • is a read process, so unclear if that will help -- 080 almost finished, others still <60% since saturday midnight?
• 15:45 MEH: modified nightly targeting in ipphosts.mhpcc.config to remove ipp083,084,086 since BBU replacement will be few more days at least -- restarted summitcopy, registration, stdsci
• 15:55 MEH: ipp094 neb-host up but non-targeted, same for ipp083,084,086,092 -- monitor for loads but should be fine for non-targeted w/o BBU WB cache

Tuesday : 2015.01.27

• 05:40 EAM: restarted stdlocal
• 08:00 EAM: bumped up poll to 1200 for now -- there are lots of failed warps due to ipp044 being offline. at the default of 600, there are not enough chip jobs to keep pcontrol busy.
• 16:00 CZW: killed leftover ippsky/staticsky jobs, and moved the x-node power back to stdlocal.
• 17:15 CZW: I've noticed that stsci volumes (not full nodes) are dropping out of nebulous today. They reappear after a bit, and can be forced back by logging into ippdb00 and manually checking the mount (ls /data/stsci13.1/ and the like resolve the issue). I don't see anything definitive in the dmesg output that would explain this.

Wednesday : YYYY.MM.DD

• 06:50 EAM: ippc20 crashed with kernel panics, rebooting.
• 14:00 CZW: Pulled back an assortment of LANL exposures/stacks for local processing to attempt to clear them and skip through endless fail-revert-fails.

Thursday : YYYY.MM.DD

• 05:30 EAM: stdscience needs a restart (too sluggish). restarting now.

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD