== PS1 IPP Czar Logs for the week 2015.01.26 - 2015.02.01 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2015.01.26 ===
 * 04:30 EAM: ipp044 crashed, nothing on console.  rebooting.
 * 04:45 EAM: ipp044 having trouble on reboot.  I set it down in nebulous.
 * 05:00 EAM: ipp044 still not returning.  i've shutdown stdscience and i'm clearing hung nfs mounts
 * 05:40 EAM: ipp044 mounts on the storage nodes are clear, stdscience restart.  clearing computes as well
 * 05:50 EAM: looks like everything is clear, ipp044 is still down.  stdscience, stdlocal, stdlanl are all back up and running.
 * 07:30 MEH: manually had to clear exposure fault in registration for o7048g0547o
{{{
regtool -dbname gpc1 -revertprocessedexp -exp_id 863126
}}}
 * 08:00 EAM: stopping and restarting stdlocal (110k warps)
   * there were various hung jobs due to hung ipp044 mount points on the stdlocal x-nodes.  I've cleared them out.
 * 10:05 MEH: nightly finished, adding storage nodes to stdlocal storage.hosts.on
 * 11:35 MEH: ipp094 Haydn releasing for IPP use now, BBU not ready for data targeting but will put into group s5 for processing.
  * also adding for neb-host, may be ok off and on for non-targeted data 
 * 14:15 MEH: ipp090,093,095,082,080 typically have heavy loads and also seem to be doing some sort of raid verification, put to repair for now
  * is a read process, so unclear if that will help -- 080 almost finished, others still <60% since saturday midnight?
 * 15:45 MEH: modified nightly targeting in ipphosts.mhpcc.config to remove ipp083,084,086 since BBU replacement will be few more days at least -- restarted summitcopy, registration, stdsci
 * 15:55 MEH: ipp094 neb-host up but non-targeted, same for ipp083,084,086,092 -- monitor for loads but should be fine for non-targeted w/o BBU WB cache

=== Tuesday : 2015.01.27 ===
 * 05:40 EAM: restarted stdlocal
 * 08:00 EAM: bumped up poll to 1200 for now -- there are lots of failed warps due to ipp044 being offline.  at the default of 600, there are not enough chip jobs to keep pcontrol busy.  
 * 16:00 CZW: killed leftover ippsky/staticsky jobs, and moved the x-node power back to stdlocal.
 * 17:15 CZW: I've noticed that stsci volumes (not full nodes) are dropping out of nebulous today.  They reappear after a bit, and can be forced back by logging into ippdb00 and manually checking the mount (ls /data/stsci13.1/ and the like resolve the issue).  I don't see anything definitive in the dmesg output that would explain this.
=== Wednesday : YYYY.MM.DD ===
 * 06:50 EAM: ippc20 crashed with kernel panics, rebooting.
 * 14:00 CZW: Pulled back an assortment of LANL exposures/stacks for local processing to attempt to clear them and skip through endless fail-revert-fails.

=== Thursday : YYYY.MM.DD ===
 * 05:30 EAM: stdscience needs a restart (too sluggish).  restarting now.

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===