== IPP MHPCC Production Cluster Status ==

(Up to [wiki:IPP_for_PS1 IPP for PS1])

'''Status as of 2009-12-30'''

=== Nodes down ===

 * ipp008 : motherboard problems -- CPUs were changed on 2010.02.17, but still no boot
 * ipp018 : motherboard problems? power supply problems?
 * ipp037 : long history of return, etc.

=== Under-Used Nodes : Status 2010.02.18 ===

 * ipp004 : currently used for addstar / dvo databases
 * ipp005 : recently replaced CPU, has been used for processing since 2/16, but not yet for nebulous
 * ipp008 : dead hardware
 * ipp009 : had several crashes in 2009, only used for nebulous since ~2009.10.29
 * ipp014 : had several crashes in 2009, only 'available' in nebulous, not 'allocated' (since ~2009.10.29)
 * ipp016 : some crashes in 2009, only nebulous for some time
 * ipp018 : dead hardware
 * ipp025 : paul raised suspicions about memory problems; only used for nebulous since then.
 * ipp037 : dead hardware
 * ipp049 : seems to have somewhat slow RAID I/O; has not been used for processing, and only 'available' in nebulous
 * ippc18 : battery temperature warning
 * ippc19 : battery temperature warning
 * ippdb02 : series of crashes, motherboard swapped.  currently unused.

=== Other suspicious Wave 1 machines ===

 * ipp006 : a few crashes in 2009
 * ipp010 : several crashes, kernel panics in 2009
 * ipp017 : several high-load, high-memory crashes 2009-2010 (not a Wave 1 machine!)

=== Notes === 

 * ipp005 - The memory was completely swapped and crashes continued.  CPUs were replaced (2010.02.09). In production since 2010.02.16.
 * ipp014 - using the chassis from ipp017 with disks from ipp014 (chassis had been ipp037, but was sent back to ASA for repair, ASA shipped back with replacement motherboard (Tyan S2912G2NR-E) Tyan S2912G2NR is EOL - kernel 2.6.31.5 is required.)
 * ipp017 - using the chassis from ipp037 with disks from ipp017
 * ipp018 - replaced 8GB of memory (2009-12-30)
 * ipp027 - original areca card replaced with a newer 3ware card; using this raid as manoa backup
 * ipp034 - using IPP037 Areca RAID Controller (IPP034 original Areca Controller appears to be defected -> Controller#1(PCI) H/W MONITOR DRAM 1-Bit ECC)
 * ipp037 - using the chassis from ipp014 with disks from ipp037

 * ippdb00 - in production as nebulous server
 * ippdb02 - memtest86 4 passes completed no errors. (2009-12-30)

=== Known issues === 

 * ipp008 - complete raid failure 2009.10.28 -- disks have been replaced and raid needs to be re-built.  ipp008 has not been used for storage for some time, so we have not lost any vital data.
 * ipp009 - several recent crashes with coincident complaints from CPU !#3.  '''note: we will swap CPU 1 and 3 and then stress-test this machine to see if the effect moves with the CPU'''
 * ipp013 - dead fans, but it is not the fans themselves (tried replacing them).  It appears to be the cabling that leads up to the fan modules that's gone bad.
 * ipp014 - has a newer version of the Wave !#1 motherboard which requires older kernel drivers or kernel 2.6.31.5.
 * ipp016 - dead fans, but it is not the fans themselves (tried replacing them).  It appears to be the cabling that leads up to the fan modules that's gone bad.
 * ipp018 - suspect bad memory module (system is usable)
 * ipp025 - suspect bad memory module (system is usable)
 * ipp027 - suspect bad memory module (system is usable)
 * ipp037 - fails to boot; serial device not detected by acs48, suspect motherboard
 * ipp034 - Areca reported: 2009-11-17  13:42:23 Controller#1(PCI) H/W MONITOR DRAM 1-Bit ECC (online events logs states to Check the DRAM module and replace with new one if required.)

 * ipp008, ipp046, ippdb02 - after mhpcc power outage (2009-11-27), the nodes fail to retain BIOS settings when power is totally lost. PDU outlets are powered off after shutdown procedure to prevent electrical surge when power resumes.  

=== Old Issues ===

 * random system crashes under heavy load of nodes, occasionally with a printk() of "do_IRQ: X.XXX" which appears to be caused by a hardware interrupt that does not have a handler
  (device driver) for it
  * these were only present on Wave !#1 hardware
  * these seem to have largely stopped with the current forcedeth driver and kernel (early 2009?)

 * ipp004 - disk bay !#12 was dead, but started working again after the disk was reseated (ipp004 has device labels inverted, and it is suspected that previous re-seat attempts were addressing the wrong disk)

 * cabinet 4 managed power strip was being overloaded.  With the wave 3 delivery, we rebalanced the power load to avoid this problem.
 * cab2pdu0 - cab2 down (2009-11-14) PDU sockets 5 & 9 reports on but PSU's on ipp005 & ipp009 not powered. Plan to return for repair.

 * Entire IPP Production Cluster. MHPCC is currently experiencing a power outage. (2009-11-27T08:59)
===  Layout at MHPCC ===
{{{
  cabinet0: ipp020-21
  cabinet1: ipp012-013,037,008,014,016,018-019
  cabinet2: ipp004-007,015,009-011
  cabinet3: ipp023-029
  cabinet4: ipp030-036,017
  cabinet5: ipp038-043,ippdb00-02 (cab5con.ipp)
  cabinet6: ipp044-048,ippc00-09  (cab6con.ipp)
  cabinet7: ipp049-053,ippc10-19  (cab7con.ipp)
}}}