IPP MHPCC Production Cluster Status

(Up to IPP for PS1)

Status as of 2008-11-14

Nodes down

None.

Notes

• ipp008 raid is being completely rebuilt
• ipp015 is rebuilding disk #16

• ipp014 - using the chassis from ipp017 with disks from ipp014 (chassis had been ipp037, but was sent back to ASA for repair)
• ipp017 - using the chassis from ipp037 with disks from ipp017
• ipp027 - original areca card replaced with a newer 3ware card; using this raid as manoa backup
• ipp037 - using the chassis from ipp014 with disks from ipp037

• ippdb00 - IPP needs to stress-test this machine

Known issues

• ipp005 - occasional crashes -- we suspect the memory may be bad. note: as of 2009.10.30, the memory has been replaced; we need to stress this machine and see if it continues to crash
• ipp008 - complete raid failure 2009.10.28 -- disks have been replaced and raid needs to be re-built. ipp008 has not been used for storage for some time, so we have not lost any vital data.
• ipp009 - several recent crashes with coincident complaints from CPU #3. note: we will swap CPU 1 and 3 and then stress-test this machine to see if the effect moves with the CPU
• ipp013 - dead fans, but it is not the fans themselves (tried replacing them). It appears to be the cabling that leads up to the fan modules that's gone bad.
• ipp014 - has a newer version of the Wave #1 motherboard which requires older kernel drivers
• ipp016 - dead fans, but it is not the fans themselves (tried replacing them). It appears to be the cabling that leads up to the fan modules that's gone bad.
• ipp018 - suspect bad memory module (system is usable)
• ipp025 - suspect bad memory module (system is usable)
• ipp027 - suspect bad memory module (system is usable)
• ipp037 - fails to boot; suspect motherboard

Old Issues

• random system crashes under heavy load of nodes, occasionally with a printk() of "do_IRQ: X.XXX" which appears to be caused by a hardware interrupt that does not have a handler (device driver) for it
  • these were only present on Wave #1 hardware
  • these seem to have largely stopped with the current forcedeth driver and kernel

• ipp004 - disk bay #12 was dead, but started working again after the disk was reseated (ipp004 has device labels inverted, and it is suspected that previous re-seat attempts were addressing the wrong disk)

• cabinet 4 managed power strip was being overloaded. With the wave 3 delivery, we rebalanced the power load to avoid this problem.

Layout at MHPCC

  cabinet0: ipp020-21
  cabinet1: ipp012-014,008,016,018-019
  cabinet2: ipp004-007,015,009-011
  cabinet3: ipp023-029
  cabinet4: ipp030-036,017