• 2008-07-15T16:20:00 ethernet became unresponsive in manner consistent with forcedeth. problem power cycled by bills
• 2008-07-15T17:15-30 (approx) system became unresponsive. power cycled by gavin
• 2008-07-16T15:00 system became unresponsive power cycled by bills
• 2008-07-17T20:45:32 IDE Channel 9 Device Removed
• 2008-07-17T21:08:59 IDE Channel 9 Device Failed
• 2008-07-17T23:31:08 IDE Channel 9 Device Removed
• 2008-07-28T10:10:08 system became unresponsive. power cycled by gavin
• 2008-07-28T11:02:00 system became unresponsive. (alerted by nagios) power cycled by gavin
• 2008-08-18T17:15:00 system lost network connection. power cycled by EAM
• 2008-08-31T06:00:00 (approx) system became unresponsive, no console log. (alerted by nagios) power cycled by eam
• 2008-09-12T16:55:00 total deadlock, power cycled
• 2008-09-15T07:04 system unresponsive all night. console unresponsive. power cycled by bills
• 2008-09-16T09:19:05 system unresponsive; no msg on console. notified by nagios. power cycled by gavin
• 2008-09-16T16:40 system deadlocked during fsck, power cycled
• 2008-09-30T11:28 kernel BUG in dmesg
• 2008-10-14T06:00 system hung, no response on console, power cycled.
• 2008-10-14T15:46 system hung, no response on console, power cycled.
• 2008-10-15T09:30 system hung, no response on console, power cycled.
• 2008-10-19T10:46 system hung, no response on console, no output power cycled by bills
• 2008-12-11T12:17 system hung, no response on console, power cycled by gavin
• 2009-04-16T14:08 system hung, no response on console, power cycled by bills
• 2009-05-21T06:50 system hung, no response on console, power cycled by bills
• 2009-07-23T08:40 system hung, no response on console, power cycled by bills
• 2009-07-28T16:39:54 system hung, no response on console, power cycled by gavin
• 2009-07-31T?? system hung, no response on console, power cycled by bills
• 2009-08-03 03:13:34 system hung, no response on console, power cycled by cindy
• 2009-08-04T12:30 system hung, no response on console, power cycled by bills
• 2009-08-17T16:33:57 system hung, no response on console, power cycled by gavin
• 2009-08-28T09:41:15 system hung, no response on console, power cycled by gavin
• 2009-08-28T11:33 IDE Channel 13 Device replaced
• 2009-09-03T09:40:51 system hung, no response on console, power cycled by gavin
• 2009-09-06T14:28 system became unresponsive power cycled by bills
• 2009-09-10 17:03:47 system hung, no response on console, power cycled by cindy
• 2009-09-11T16:52:13 system hung, no response on console, power cycled by gavin
• 2009-09-16 13:44:04 HST system hung, no response on console, power cycled by cindy. ganglia shows high memory usage
• 2009-09-17T11:26:34 system hung, no response on console, memory & CPU spike before crash ; power cycled by gavin
• 2009-09-21T19:46:40 system hung, no response on console, ganglia reported high memory usage before crash ; power cycled by gavin
• 2009-09-22T09:35:20 IDE Channel 3 Device replaced
• 2009-09-23T16:26:51 system hung, no response on console, ganglia memory usage rising before crash ; power cycled by gavin
• 2009-09-30T1948 system hung, no response on console, ganglia memory usage rising before crash ; power cycled by bills at 0405
• 2009-10-30T11:00 replaced memory modules w/ 4 x 2GB DIMMS.
• 2009-10-30T14:29:15 IDE Channel 16 Deviced failed (replaced 2009-10-30T16:26:16)
• 2009-11-02T09:12 system I/O error msg. RAID/volume set in rebuild state from 10/30/2009 (95.5%)
• 2009-11-05T12:28:04 system crash, no response on console, cpu​, doesn't appear to be memory​ issue ; power cycled by gavin
• 2009-11-10T13:00:04 unresponsive, nothing on console, power cycled by gavin
• 2009-11-10T15:34:54 unresponsive, nothing on console, slight network​ / memory​ change, power cycled by gavin
• 2009-11-12T17:24:46 unresponsive, nothing on console, ganglia memory​, /var/log/messages: Nov 12 15:14:49 ipp005 [169533.514255] ppStack[32254]: segfault at 11 ip 000000000041179c sp 00007fff14990a40 error 4 in ppStack[400000+17000] ; power cycled by gavin
• 2009-11-17T01:34:43 unresponsive, nothing on console, power cycled by gavin (2009-11-17T10:00)
• 2009-11-17T11:09:53 unresponsive, nothing on console, power cycled by gavin
• 2009-11-17T18:01:22 unresponsive, nothing on console, power cycled by gavin
• 2009-11-19T18:40:24 unresponsive, nothing on console, power cycled by gavin
• 2009-11-20T23:20 unresponsive, nothing on console, power cycled by blls at 2009-11-21T730
• Nov 23 21:59:36 HST 2009 unresponsive, nothing on console, power cycled by cindy at Nov 24 08:12:46 HST 2009
• Dec 2 20:51:27 HST 2009 unresponsive, nothing on console or logs, power cycled by cindy
• 2009-12-04T06:53:35 unresponsive, nothing on console, power cycled by gavin (2009-12-04T08:40)
• 2009-12-08T09:31 unresponsive, [281585.924355] journal commit I/O error [348308.075629] Kernel panic - not syncing: Attempted to kill init! ; power cycled by gavin
• 2009-12-09T14:45:14 system unresponsive, no response on console, power cycled by gavin
• 2009-12-12T22:04:11 system unresponsive, no response on console, power cycled by gavin (2009-12-13T11:09)
• 2009-12-14T19:40:52 system unresponsive, no response on console, power cycled by gavin (2009-12-15T06:44)
• 2009-12-15T14:20:52 system unresponsive, no response on console, ganglia memory​/load​, power cycled by gavin
• 2009-12-15T14:58:12 system unresponsive, no response on console, ganglia load​, power cycled by gavin
• 2009-12-22T21:31:16 system unresponsive, nothing on console, ganglia load​, power cycled by gavin (2009-12-23T10:19)
• 2009-12-29T11:38:33 system unresponsive, nothing on console, power cycled by gavin (2009-12-29T12:15)
• 2009-12-29T18:00:14 system unresponsive, nothing on console, ganglia load​, power cycled by gavin
• 2009-12-30T10:20:37 system unresponsive, nothing on console, ganglia load​, memory​, power cycled by gavin
• 2009-12-30T11:36 system unresponsive, ganglia load​, memory​ power cycled by gavin
• 2010-01-06T18:25 system unresponsive, nothing on console, power cycled by gavin
• 2010-02-09T09:00 system down for maintenance, Bill Giebink to test IPP018 CPU's in IPP005.
• 2010-07-02T09:30 system down for upgrade, Steph Mohr installed 24 x 2TB HDD's.
• 2010-08-10T13:39:20 IDE Channel 10 Device Failed (replace 2010-08-10T15:01:54)
• 2010-10-30T0606 ipp005 load very high able to log in. Follow this link for more info. ipp005.20101030
• 2011-02-09T13:10:58 IDE Channel 4 Device Removed (re-seated 2011-02-09T13:11:24)
• 2011-02-09 motherboard upgrade. done.
• 2011-02-18T13:00:00 kernel panic, rebooted by EAM
• 2011-03-05T01 (time approximate) kernel panic still responds to ganglia but not on console. power cycled by bills ~6:15 am
• 2011-03-07T11:21 same error I think. Console says:
  • <Mar/07 07:36 am>
  • [185287.241326] journal commit I/O error
  • [189999.638383] Kernel panic - not syncing: Attempted to kill init!
• 2011-03-15T12:45:45 system unresponsive, [701247.783908] Kernel panic - not syncing: Attempted to kill init! ; power cycled by gavin
• 2011-03-29T15:20 system unresponsive, not mounted in nebulous but ganglia thinks it's alive. Power cycled by bills. Console said:
  • <Mar/29 08:20 am>ipp005 login: [1196647.444703] journal commit I/O error
  • [1213444.565970] Kernel panic - not syncing: Attempted to kill init!
• 2011-04-05 14:11 INIT: Id "s0" r[609679.616849] Kernel panic - not syncing: Attempted to kill init! Power cycled by bills.
• 2011-04-12
  • <Apr/12 07:35 am> [579766.406844] journal commit I/O error
  • <Apr/12 07:35 am> [579766.406876] journal commit I/O error
  • [584297.232145] Kernel panic - not syncing: Attempted to kill init!

• 2011-04-21 [761674.464312] Kernel panic - not syncing: Attempted to kill init! Power cycled by bills