Changes between Version 10 and Version 11 of PS1_IPP_Czarlog_20150921

Timestamp:

Sep 25, 2015, 2:14:27 PM (11 years ago)

Author:

eugene

Comment:

--

PS1_IPP_Czarlog_20150921

@@ -18 +18 @@
 
 === Monday : 2015.09.21 ===
+
  * 08:40 EAM : looks like we had a power event on cab1 : all machines in that cabinet rebooted spontaneously in the past 20 min.  ipp013 is still coming up slowly.
  * 18:15 EAM : we lost all power around 9:30 this morning.  UPS failure.  it took until around noon for power to come back.  I am now restarting all ipp pantasks 
@@ -23 +24 @@
   * nebdiskd started on ippdb08
   * WS diff pantasks run under ippmops stdscience also needed to be started
+
 === Tuesday : 2015.09.22 ===
+
  * 11:30 MEH: c2 nodes being used for extra processing for couple hours -- off in stdsci
   * c2 nodes back on in ipp:stdscience and ippmops:stdscience
-=== Wednesday : YYYY.MM.DD ===
 
-=== Thursday : YYYY.MM.DD ===
+=== Wednesday : 2015.09.23 ===
 
-=== Friday : YYYY.MM.DD ===
+=== Thursday : 2015.09.24 ===
 
-* 13:20 EAM: another power outage for the IPP.  the cluster is mostly back up, but I need to shutdown mysql on ippdb05 and make a dump.  the ippdb03 replication backup is not starting -- it looks like the binlogs got corrupted during the shutdown.  If I can recover from the existing binlogs I will do that and stop the dump, but for now please leave all services off.
+=== Friday : 2015.09.25 ===
+
+ * 13:20 EAM: another power outage for the IPP.  the cluster is mostly back up, but I need to shutdown mysql on ippdb05 and make a dump.  the ippdb03 replication backup is not starting -- it looks like the binlogs got corrupted during the shutdown.  If I can recover from the existing binlogs I will do that and stop the dump, but for now please leave all services off.
+ * 14:20 EAM: ~ipp pantasks, ~ippmops stdscience pantasks started.  '''note''' pstamp has not been restarted: I am dumping mysql @ ippc17 to restart the replicant.
 
 === Saturday : YYYY.MM.DD ===