IPP MHPCC Production Cluster Status

(Up to IPP for PS1)

Status as of 2008-11-14

Nodes down

None.

Notes

• ipp015 is rebuilding disk \#16

Known issues

• random system crashes under heavy load of nodes, occasionally with a printk() of "do_IRQ: X.XXX" which appears to be caused by a hardware interrupt that does not have a handler (device driver) for it.
• ipp004 - disk bay #12 dead (system is usable)
• ipp005 - seems to have random instability issues beyond the do_IRQ issue of the other nodes
• ipp008 - disk bay #7 dead (system is usable)
• ipp013 & ipp016 have dead fans but it is not the fans themselves (tried replacing them). It appears to be the cabling that leads up to the fan modules that's gone bad.
• forcedeth.c max_interrupt_work is still too conservative (requires a cluster restart to change).
• ipp025 - suspect bad memory module (system is usable)
• Our contact at MHPCC, Brad Thomas, notified me that an alarm from cabinet 4 managed power strip had been set off last evening due to high loads on nodes (ipp017, 030-036). Please consider distributing your jobs evenly across ipp production nodes.

Layout at MHPCC

  cabinet0: ipp020-21
  cabinet1: ipp012-014,008,016,018-019
  cabinet2: ipp004-007,015,009-011
  cabinet3: ipp023-029
  cabinet4: ipp030-036,017