Changes between Version 3 and Version 4 of Production_Cluster_Status

Timestamp:

Nov 2, 2009, 3:18:38 PM (17 years ago)

Author:

eugene

Comment:

--

Production_Cluster_Status

@@ -11 +11 @@
 === Notes === 
 
+ * ipp008 raid is being completely rebuilt
+ * ipp014 is now using the chassis from ipp017 (which had been ipp037, but was sent back to ASA for repair)
+ * ipp037 is using the chassis from ipp014
+ * ipp037 
  * ipp015 is rebuilding disk !#16
 
 === Known issues === 
 
+ * ipp005 - occasional crashes -- we suspect the memory may be bad.  '''note: as of 2009.10.30, the memory has been replaced; we need to stress this machine and see if it continues to crash'''
+ * ipp008 - complete raid failure 2009.10.28 -- disks have been replaced and raid needs to be re-built.  ipp008 has not been used for storage for some time, so we have not lost any vital data.
+ * ipp009 - several recent crashes with coincident complaints from CPU !#3.
+ * ipp013 - dead fans, but it is not the fans themselves (tried replacing them).  It appears to be the cabling that leads up to the fan modules that's gone bad.
+ * ipp014 - has a newer version of the Wave !#1 motherboard which requires older kernel drivers 
+ * ipp016 - dead fans, but it is not the fans themselves (tried replacing them).  It appears to be the cabling that leads up to the fan modules that's gone bad.
+ * ipp037 - fails to boot; suspect motherboard
+ * ipp025 - suspect bad memory module (system is usable)
+
+=== Old Issues ===
+
  * random system crashes under heavy load of nodes, occasionally with a printk() of "do_IRQ: X.XXX" which appears to be caused by a hardware interrupt that does not have a handler
-  (device driver) for it.
- * ipp004 - disk bay #12 dead (system is usable)
- * ipp005 - seems to have random instability issues beyond the do_IRQ issue of the other nodes
- * ipp008 - disk bay #7 dead (system is usable)
- * ipp013 & ipp016 have dead fans but it is not the fans themselves (tried replacing them).  It appears to be the cabling that leads up to the fan modules that's gone bad.
- * forcedeth.c max_interrupt_work is still too conservative (requires a cluster restart to change).
- * ipp025 - suspect bad memory module (system is usable)
- * Our contact at MHPCC, Brad Thomas, notified me that an alarm from cabinet 4 managed power strip had been set off last evening due to high loads on nodes (ipp017, 030-036).
-  Please consider distributing your jobs evenly across ipp production nodes.
+  (device driver) for it
+  * these were only present on Wave !#1 hardware
+  * these seem to have largely stopped with the current forcedeth driver and kernel
+
+ * ipp004 - disk bay !#12 was dead, but started working again after the disk was reseated (ipp004 has device labels inverted, and it is suspected that previous re-seat attempts were addressing the wrong disk)
+
+ * cabinet 4 managed power strip was being overloaded.  With the wave 3 delivery, we rebalanced the power load to avoid this problem.
 
 ===  Layout at MHPCC ===