== PS1 IPP Czar Logs for the week 2017-11-06 - 2017-11-13 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2017-11-06 ===
  
=== Tuesday : 2017-11-07 ===
 * 12:00 CZW: Starting MD database based shuffle check, using ~ipptest/replication as the pantasks server location (20 jobs using stare nodes).  Should have minor impact as only newly created files/repaired files will receive md5sum calculations.

=== Wednesday : 2017.11.08 ===
 * MEH: extra QUB processing running on nightly nodes once nightly processing finished for next few days
 * MEH: ipp121 and ippc118 not back up after afternoon failure of AC@ITC -- tried power cycle and no boot, power off in console and neb-host down for ipp121 and turning ippc118 in nightly pantasks
 * MEH: ippc34,c39,c42,c43,c51 -- gen prot faults or stalled nfs requiring reboot -- ippc43 some reason doing full mem scan by default and takes a very long time
 * MEH: ippc58 kernel panic -- power cycled and back up again
=== Thursday : 2017.11.09 ===
 * MEH: ippc36 gen prot fault and unresponsive -- power cycled and back up again
 * MEH: ippc33,c40 as well -- power cycled and back up again
 * 12:20 CZW: I'm going to power cycle stare02 and stare04, which have stopped responding to ssh.
  * And also stare00, which I didn't notice was doing the same thing.
 * MEH: more compute nodes with kernel panics and gen prot faults showing up and need power cycling -- c32,c34,c35,c38,c44,c45,c46,c47,c48,c49,c50,c52,c56,c57
  * ippc39,c43,c51 now twice <24hr
=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===