• 2008-09-22T04:00 (approx) kernel oops
• 2009-10-02T13:48:32 IDE Channel 10 Reading Error (reseated)
• 2009-11-12T22:27:07 IDE Channel 10 Reading Error (replaced 2009-11-13T09:55:36)
• 2010-01-05T16:26:21 system unresponsive, ganglia load​, memory​, cpu​, power cycled by gavin
• 2010-08-04T15:25:03 IDE Channel 2 Reading Error (replaced 2010-08-04T16:31:53)
• 2011-06-28T11:50:04 network unresponsive, nothing on console, power cycled by gavin
• 2011-7-5 17:17:21 Controller # 1(PCI) IDE Channel # 4 Reading Error Drive replaced
• 2011-11-10 6:26:56 Controller 1(PCI) IDE Channel 9 Reading Error NOT REPLACED
• 2011-9-17 12:32:49 Controller#1(PCI) IDE Channel # 7 Device Failed REPLACED
• 2012-2-17 14:21:24 Controller#1(PCI) IDE Channel #16 Device Failed REPLACED
• Oct 6 05:50:05 ipp033 archttp64: [WARNING]: ArecaRaidCtrl Event(URGENT): 2012-10-6 4:40:43 Controller#1(PCI) IDE Channel #24 Device Failed (replaced: Oct 6 11:28:09)
• Note: a 1TB WD drive failed. We replaced it with a 1TB Seagate drive, but it didn't recognize that drive, so I replaced with a new 2TB drive.
• Note: the time reported by the Areca RAID controller is off by more than an hour!
• 2012-11-05 ~19:00 system crash, no info on display or in /var/log/messages ipp033-crash-20121105. NFS did not come up right, had to restart. ganglia didn't recognize as back up for ~10 minutes..
• 2012-11-16T01:45:03 system unresponsive, console messages, power cycled by gavin (2012-11-16T07:56:25)
• 2013-05-27 02:30 unresponsive, nothing on console. power-cycled
• 2013-08-23 22:50 unresponsive, nothing on console. power-cycled
• 2013-08-31 00:30 unresponsive, nothing on console. power-cycled
• 2013-09-07 23:40 unresponsive, nothing on console. power-cycled.. seems to be a common problem, overloaded or midnight gremlins?
• 2013-09-10 02:00 unresponsive, nothing on console. power-cycled..
• 2014-05-10 01:30 unresponsive, nothing on console, power-cycled -- 80% loaded, no high temperature/load/mem use..