== PS1 IPP Czar Logs for the week 2014-03-17 - 2014-03-23 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2014-03-17 ===
 * 09:30 Bill storage.hosts.on, storage.hosts ignore in staticsky pantasks
 * 11:15 at suggestion of serge and gene, kicked a diff to have a quality code of 42
=== Tuesday : 2014-03-18 ===
mark is czar
 * 07:05 MEH: clearing WS diffim fault 5 (532198,532231,532259,532292 skycell.1089.062; 532381 skycell.2608.044; 533099 skycell.1084.087, skycell.1084.096)
 * 07:50 MEH: nightly finished, staticsky storage.hosts.on, storage.hosts.ignore
 * 08:20 MEH: after ~4 days, restarting regular pantasks for the week
 * 10:30 MEH: seeing load on ipp032 for mops and ippc30 for PSS, setting off in staticsky adds manually
  * ippc30 has MOPS stamps taking ~1min each rather than ~3/min normally, running staticsky on ippc30 probably isn't helping
 * 18:00 MEH: extra nodes for staticsky off in time for nightly, storage.hosts.off 
 * 19:30 MEH: ipp053 looks to have gone unresponsive ~19:20.. seriously.. -- [wiki:ipp053_log]
  * nothing on console, power cycling since non-responsive for over 10 min, ganglia showing drop of cpu to only ~20% cpu_wait @1900 so something probably just crashed
  * failed to boot on first power cycle, finally kicked over on second power cycle and one of the long boot screen systems for ram check.. 
  * nothing in logs except USB drive faults/[sr0] Unhandled sense code after reboot -- leaving in repair and out of processing for tonight
=== Wednesday : 2014-03-19 ===
mark is czar
 * 07:50 MEH: fault 5 diffim to clear (533929,533896 skycell.2518.043), nightly mostly done but for ~100 WS diffs
 * 08:40 MEH: WS diffims finished, switching nodes to staticsky except for ipp032 for Serge/MOPS
 * 13:30 MEH: ipp053 down again after running staticsky load but in neb-host repair. not booting.
=== Thursday : 2014-03-20 ===

=== Friday : 2014-03-21 ===

=== Saturday : 2014-03-22 ===

=== Sunday : 2014-03-23 ===