• 2008-07-15T16:20:00 ethernet became unresponsive in manner consistent with forcedeth. problem power cycled by bills
• 2008-07-15T17:15-30 (approx) system became unresponsive. power cycled by gavin
• 2008-07-16T15:00 system became unresponsive power cycled by bills
• 2008-07-17T20:45:32 IDE Channel 9 Device Removed
• 2008-07-17T21:08:59 IDE Channel 9 Device Failed
• 2008-07-17T23:31:08 IDE Channel 9 Device Removed
• 2008-07-28T10:10:08 system became unresponsive. power cycled by gavin
• 2008-07-28T11:02:00 system became unresponsive. (alerted by nagios) power cycled by gavin
• 2008-08-18T17:15:00 system lost network connection. power cycled by EAM
• 2008-08-31T06:00:00 (approx) system became unresponsive, no console log. (alerted by nagios) power cycled by eam
• 2008-09-12T16:55:00 total deadlock, power cycled
• 2008-09-15T07:04 system unresponsive all night. console unresponsive. power cycled by bills
• 2008-09-16T09:19:05 system unresponsive; no msg on console. notified by nagios. power cycled by gavin
• 2008-09-16T16:40 system deadlocked during fsck, power cycled
• 2008-09-30T11:28 kernel BUG in dmesg
• 2008-10-14T06:00 system hung, no response on console, power cycled.
• 2008-10-14T15:46 system hung, no response on console, power cycled.
• 2008-10-15T09:30 system hung, no response on console, power cycled.
• 2008-10-19T10:46 system hung, no response on console, no output power cycled by bills
• 2008-12-11T12:17 system hung, no response on console, power cycled by gavin
• 2009-04-16T14:08 system hung, no response on console, power cycled by bills
• 2009-05-21T06:50 system hung, no response on console, power cycled by bills
• 2009-07-23T08:40 system hung, no response on console, power cycled by bills
• 2009-07-28T16:39:54 system hung, no response on console, power cycled by gavin
• 2009-07-31T?? system hung, no response on console, power cycled by bills
• 2009-08-03 03:13:34 system hung, no response on console, power cycled by cindy
• 2009-08-04T12:30 system hung, no response on console, power cycled by bills
• 2009-08-17T16:33:57 system hung, no response on console, power cycled by gavin
• 2009-08-28T09:41:15 system hung, no response on console, power cycled by gavin
• 2009-08-28T11:33 IDE Channel 13 Device replaced
• 2009-09-03T09:40:51 system hung, no response on console, power cycled by gavin
• 2009-09-06T14:28 system became unresponsive power cycled by bills
• 2009-09-10 17:03:47 system hung, no response on console, power cycled by cindy
• 2009-09-11T16:52:13 system hung, no response on console, power cycled by gavin
• 2009-09-16 13:44:04 HST system hung, no response on console, power cycled by cindy. ganglia shows high memory usage
• 2009-09-17T11:26:34 system hung, no response on console, memory & CPU spike before crash ; power cycled by gavin
• 2009-09-21T19:46:40 system hung, no response on console, ganglia reported high memory usage before crash ; power cycled by gavin
• 2009-09-22T09:35:20 IDE Channel 3 Device replaced
• 2009-09-23T16:26:51 system hung, no response on console, ganglia memory usage rising before crash ; power cycled by gavin
• 2009-09-30T1948 system hung, no response on console, ganglia memory usage rising before crash ; power cycled by bills at 0405
• 2009-10-30T11:00 replaced memory modules w/ 4 x 2GB DIMMS.
• 2009-10-30T14:29:15 IDE Channel 16 Deviced failed (replaced 2009-10-30T16:26:16)
• 2009-11-02T09:12 system I/O error msg. RAID/volume set in rebuild state from 10/30/2009 (95.5%)
• 2009-11-05T12:28:04 system crash, no response on console, cpu​, doesn't appear to be memory​ issue ; power cycled by gavin
• 2009-11-10T13:00:04 unresponsive, nothing on console, power cycled by gavin
• 2009-11-10T15:34:54 unresponsive, nothing on console, slight network​ / memory​ change, power cycled by gavin
• 2009-11-12T17:24:46 unresponsive, nothing on console, ganglia memory​, /var/log/messages: Nov 12 15:14:49 ipp005 [169533.514255] ppStack[32254]: segfault at 11 ip 000000000041179c sp 00007fff14990a40 error 4 in ppStack[400000+17000] ; power cycled by gavin
• 2009-11-17T01:34:43 unresponsive, nothing on console, power cycled by gavin (2009-11-17T10:00)