• 2007-03-22T14:36:28 IDE Channel 15 Device Removed
• 2009-12-09T08:53:50 IDE Channel 13 Device Failed (replaced 2009-12-09T13:42:44)
• 2010-01-20T09:55:10 IDE Channel 4 Media Error Count (reseated 2010-01-20T?)
• 2010-07-30T03:46 System became unresponsive over network. Console responsive login for bills fails due to yp. Looks like network is broken. I hate wave 1 machines. Power cycled by bills at 0700
• 2010-08-26T11:45 System became unresponsive over network. Console accepts input and scrolls but no other output power cycled by bills. Note this occurred during a nebulous no space emergency.
• 2010-09-18T14:30 Same symptoms as last entry. (except for space emergency part) Power cycled by bills.
• 2010-9-21 15:9:2 Channel #1 Device Failed :replaced and rebuilt
• 2010-09-23T10:12:16 system unresponsive, nothing on console, power cycled by gavin
• 2010-09-27T06:25 system uresponsive, nothing on console, console scrolls to <CR> but outputs nothing. power cycled by bills 08:45 messages
• 2010-10-07T11:27:32 system unresponsive, nothing on console, power cycled by gavin
• 2010-10-14T21:12 or so system unresponsive, nothing on console, power cycled by bills
• 2010-10-27T10:50:01 system unresponsive, nothing on console. /var/log/messages contains lots of "Oct 27 10:52:01 ipp012 [606071.296158] nfs: server ipp030 not responding, timed out" messages. This probably indicates a problem with the network interface. power cycled by bills at 11:01
• 2010-11-01T23:35 system unresponsive. power cycled by bills at 2010-11-02T00848 'nfs: server ipp* not responding messages' in /var/log/meesages began at 23:35
• 2010-11003T13:15 system unresponsive. console scrolls for <CR>. power cycled by bills server not repsonding messages started at 13:15
• 2010-11-8 2:22:2 Channel #20 Device Failed replaced rebuilt
• 2010-11-20 7:36:50 #10 Device Failed replaced rebuilt
• 2010-11-25 19:22:6 Channel #17 Device Failed replaced rebuilt
• 2010-12-10 9:27:24 Channel #5 Device Failed
• 2011-01-20 Wave1 Refurb - mobo + cpu + memory + 2TB HDD (IPP009,012,014)
• 2011-01-26T08:54:57 system unresponsive, INIT: Id "s0" r[46526.771988] Kernel panic - not syncing: Attempted to kill init!, power cycled by gavin
• 2011-01-27T10:21:44 system unresponsive, [90738.124591] Kernel panic - not syncing: Attempted to kill init!, power cycled by gavin
• 2011-4-22 22:42:11 Controller #1(PCI) IDE Channel #14 Reading Error
• 2011-6-7 1:14:15 Controller #1(PCI) IDE Channel #18 Reading Error
• 2012-8-5 5:13:16 Controller #1(PCI) IDE Channel #10 Reading Error Drive replaced
• 2012-9-22T6:49:33 Controller #1(PCI) IDE Channel #1 Reading Error Drive replaced
• 2012-9-24T17:39:46 Controller #1(PCI) IDE Channel #12 Reading Error - Drive replaced on 9/25/12 in AM
  • 2012-9-25 9:54:59 Controller #1(PCI) ARC-1280-VOL#00 Start Rebuilding
• 2012-9-25 Sep 25 01:41:49 ipp012 [ 77.058077] iTCO_wdt: failed to reset NO_REBOOT flag, reboot disabled by hardware
  • system was rebooted with: reboot -f (to force the reboot) load was over 50 and was not allowing remote logins. Because of the drive replacement that was started a few hours earlier we did not want to power cycle while the RAID was rebuilding. System rebooted without problems.
• 2012-09-27 05:00 rebooted due to mount problem (see czarlog PS1_IPP_Czarlog_20120924
• 2012-10-18 09:45 Bill rebooted sick ipp012 (from czar log PS1_IPP_Czarlog_20121015)
• 2012-12-03 10:30 Power-cycled (see wiki:PS1_IPP_Czarlog_20121203 for details)
• 2012-12-22 06:50 unable to log into, glocks hanging. similar problem been having with multiple wave1 systems. power cycled. on console ipp012-crash-20121222
• 2013-5-3 11:21:55 Controller #1(PCI) IDE Channel #15 Reading Error (replaced 05/03/13)
• 2013-05-05 13:15 hung on console (kernel panic, see wiki:ipp012-crash-20130505)