* 2008-09-22T04:00 (approx) [wiki:Ipp033-crash-2008-09-22 kernel oops] * 2009-10-02T13:48:32 IDE Channel 10 Reading Error (reseated) * 2009-11-12T22:27:07 IDE Channel 10 Reading Error (replaced 2009-11-13T09:55:36) * 2010-01-05T16:26:21 system unresponsive, ganglia [attachment:ipp033-loadgraph-2010-01-05.png load], [attachment:ipp033-memorygraph-2010-01-05.png memory], [attachment:ipp033-cpugraph-2010-01-05.png cpu], power cycled by gavin * 2010-08-04T15:25:03 IDE Channel 2 Reading Error (replaced 2010-08-04T16:31:53) * 2011-06-28T11:50:04 network unresponsive, nothing on console, power cycled by gavin * 2011-7-5 17:17:21 Controller # 1(PCI) IDE Channel # 4 Reading Error Drive replaced * 2011-11-10 6:26:56 Controller 1(PCI) IDE Channel 9 Reading Error NOT REPLACED * 2011-9-17 12:32:49 Controller#1(PCI) IDE Channel # 7 Device Failed REPLACED * 2012-2-17 14:21:24 Controller#1(PCI) IDE Channel #16 Device Failed REPLACED * Oct 6 05:50:05 ipp033 archttp64: [WARNING]: ArecaRaidCtrl Event(URGENT): 2012-10-6 4:40:43 Controller#1(PCI) IDE Channel #24 Device Failed (replaced: Oct 6 11:28:09) * Note: a 1TB WD drive failed. We replaced it with a 1TB Seagate drive, but it didn't recognize that drive, so I replaced with a new 2TB drive. * Note: the time reported by the Areca RAID controller is off by more than an hour! * 2012-11-05 ~19:00 system crash, no info on display or in /var/log/messages [wiki:ipp033-crash-20121105]. NFS did not come up right, had to restart. ganglia didn't recognize as back up for ~10 minutes.. * 2012-11-16T01:45:03 system unresponsive, console [wiki:ipp033-crash-20121116 messages], power cycled by gavin (2012-11-16T07:56:25) * 2013-05-27 02:30 unresponsive, nothing on console. power-cycled * 2013-08-23 22:50 unresponsive, nothing on console. power-cycled * 2013-08-31 00:30 unresponsive, nothing on console. power-cycled * 2013-09-07 23:40 unresponsive, nothing on console. power-cycled.. seems to be a common problem, overloaded or midnight gremlins? * 2013-09-10 02:00 unresponsive, nothing on console. power-cycled.. * 2014-05-10 01:30 unresponsive, nothing on console, power-cycled -- 80% loaded, no high temperature/load/mem use..