• [4551966.920235] Kernel panic - not syncing: Attempted to kill init!
• 20090814T1227 System hung. Following messages on console Power cycled by bills

       [229478.284767] BUG: spinlock lockup on CPU#1, scsi_eh_5/2771, ffff88083a9b17c0
       [229363.893762] BUG: spinlock lockup on CPU#3, syslog-ng/7829, ffff88083a9b17c0
       [229464.858611] BUG: spinlock lockup on CPU#6, sshd/12753, ffff88083a9b17c0
  
  

• 20090819T2000 appprox Kernel panic - not syncing: Attempted to kill init! power cycled by bills
• 20090825T2040 or so system became unresponsive. Not noticed until 0439 by iqanalysis folks. No response on console at 0641. Power cycled by bills. At the time of the failure summit copy operations were firing up. Also, at the same time cronjob mysql dumps running to back up database on other nodes. Times were changed to start at 4AM.
• 2009-09-08T16:00:37 system became unresponsive ; nothing on console ; 3ware logs have been removed after reboot (unable to to tell if auto verify caused the unresponsiveness. power cycled by gavin.
• 2009-09-12T00:48 system became unresponsive, load extremely high ; nothing on console ; power cycled by gavin
• 2009-09-14T08:06:24 system became unresponsive, load extremely high ; nothing on console ; power cycled by gavin.
• 2009-09-15 12:32:04 HST 2009 3ware controller Firmware upgrade done, node required reboot by cindy
• 2009-09-28T08:12:54 system unresponsive, extremely high load, nothing on console, power cycled by gavin
• 2009-11-04T console msg: [3209490.099538] Kernel panic - not syncing: Attempted to kill init!, ippdb02_crash-20091104, power cycled by gavin (2009-11-12T10:20:36)
• 2009-11-12T04:26:35 system unresponsive, nothing displayed on console and messages, ganglia reported high load​ / memory​ / CPU-wait​, power cycled by gavin
• Sat Nov 14 21:15:49 HST 2009 system unresponsive, nothing displayed on console and messages, power cycled by cindy
• Sun Nov 22 21:28:03 HST 2009 nagios started complaining about problems communicating with ippdb02 Host down alert at 22:07 console shows hardware error See ippdb02_error20091122 System did not come back up after cycling power twice. Left it with power on. Bill Sweeney
• 2009-12-03T03:34:27 high load​, cpu​, memory​, 504 processes, nothing on console ; power cycled by gavin
• 2010-01-27T10:30:00 Motherboard swap - Aberdeen
• 2010-04-22T00:06:25 system became unresponsive, nothing on console, power cycled by gavin (rebooted 2010-04-22T05:59)