== IPP MHPCC Production Cluster Status ==

(Up to [wiki:IPP_for_PS1 IPP for PS1])

'''Status as of 2008-11-14'''

=== Nodes down ===

None.

=== Notes === 

 * ipp015 is rebuilding disk \#16

=== Known issues === 

 * random system crashes under heavy load of nodes, occasionally with a printk() of "do_IRQ: X.XXX" which appears to be caused by a hardware interrupt that does not have a handler
  (device driver) for it.
 * ipp004 - disk bay #12 dead (system is usable)
 * ipp005 - seems to have random instability issues beyond the do_IRQ issue of the other nodes
 * ipp008 - disk bay #7 dead (system is usable)
 * ipp013 & ipp016 have dead fans but it is not the fans themselves (tried replacing them).  It appears to be the cabling that leads up to the fan modules that's gone bad.
 * forcedeth.c max_interrupt_work is still too conservative (requires a cluster restart to change).
 * ipp025 - suspect bad memory module (system is usable)
 * Our contact at MHPCC, Brad Thomas, notified me that an alarm from cabinet 4 managed power strip had been set off last evening due to high loads on nodes (ipp017, 030-036).
  Please consider distributing your jobs evenly across ipp production nodes.

===  Layout at MHPCC ===
{{{
  cabinet0: ipp020-21
  cabinet1: ipp012-014,008,016,018-019
  cabinet2: ipp004-007,015,009-011
  cabinet3: ipp023-029
  cabinet4: ipp030-036,017
}}}