Changes between Version 31 and Version 32 of PS1_IPP_Czarlog_20121119

Timestamp:

Nov 23, 2012, 5:41:55 PM (14 years ago)

Author:

bills

Comment:

--

PS1_IPP_Czarlog_20121119

@@ -35 +35 @@
 === Thursday : 2012.11.22 ===
  * 09:53 Serge: rsync to ippdb04 complete. Restarted nebulous replication on ippc63.
-=== Friday : 2012-11-23 ===
+=== Friday : 2012.11.23 ===
 Bill is czar today
  * 08:00 one of the nodes has been stuck for 60,000 seconds stopped all pantasks 
@@ -50 +50 @@
  * 15:36 set ipp016 to repair and all pantasks to stop. glockfiles are hanging for files on that node. sigh
  * 15:50 looks like a false alarm. staticsky has stopped making progress because I reverted a number of runs that are faulting again.
-=== Saturday : YYYY.MM.DD ===
+ * 17:30 Found the problem causing many of the staticsky runs to crash. Stopped everything and rebuilt psphot.
+ * 17:40 turning chip back on.
+=== Saturday : 2012.11.24 ===
 
-=== Sunday : YYYY.MM.DD ===
+=== Sunday : 2012.11.25 ===