* 2010-01-14T09:32:37   IDE Channel Device 2 Failed (replaced 2009-01-14T?) 
     Areca Controller did not detect replacement disk, re-seated & replaced new disk. no response. 
     (rebooted IPP026 2009-01-15T17:00) Brad was required to re-seat once again after reboot in order for Areca to detect new disk.
   * 2010-01-19T09:42:37   IDE Channel 9 Media/Timeout error (replaced 2010-1-19T09:45:38)
   * 2011-4-2  13:36:31 Controller#1(PCI) IDE Channel # 17 Reading Error disc replaced 
   * 2011-5-13  5:46:38 Controller#1(PCI) IDE Channel #10 Reading Error 
   * 2011-07-06T17:05:14 system unresponsive, [wiki:Ipp026-2011-07-06-kernel kernel panic], power cycled by gavin
   * 2011-07-12T15:00:00 nfs went insane, rebooted by EAM
   * 2011-07-13T08:00:00 kernel panic last night, rebooted by EAM
   * 2011-07-26T13:50:00 kernel panic, rebooted by EAM [wiki:ipp026-2011-07-26-panic]
   * 2011-07-27T12:30 Kernel panic rebooted by Serge
{{{
<Jul/27 10:15 pm>[80552.646321] HARDWARE ERROR
<Jul/27 10:15 pm>[80552.646321] CPU 1: Machine Check Exception:                5 Bank 5: b200001084200e0f
<Jul/27 10:15 pm>[80552.646321] RIP !INEXACT! 33:<00007fcf717506c4> 
<Jul/27 10:15 pm>[80552.646321] TSC ea1511662cd0 
<Jul/27 10:15 pm>[80552.646321] This is not a software problem!
<Jul/27 10:15 pm>[80552.646321] Run through mcelog --ascii to decode and contact your hardware vendor
<Jul/27 10:15 pm>[80552.646321] 
<Jul/27 10:15 pm>[80552.646321] HARDWARE ERROR
<Jul/27 10:15 pm>[80552.646321] CPU 7: Machine Check Exception:                4 Bank 5: b200002008000e0f
<Jul/27 10:15 pm>[80552.646321] TSC ea1511664498 
<Jul/27 10:15 pm>[80552.646321] This is not a software problem!
<Jul/27 10:15 pm>[80552.646321] Run through mcelog --ascii to decode and contact your hardware vendor
<Jul/27 10:15 pm>[80552.646321] Kernel panic - not syncing: Machine check
<Jul/27 10:15 pm>[80552.646321] ------------[ cut here ]------------
<Jul/27 10:15 pm>[80552.646321] WARNING: at kernel/smp.c:333 smp_call_function_mask+0x37/0x1d7()
<Jul/27 10:15 pm>[80552.646321] Modules linked in: coretemp w83627hf w83793 hwmon_vid k8temp autofs4 i2c_i801 iTCO_wdt i2c_core e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
<Jul/27 10:15 pm>[80552.646321] Pid: 7776, comm: ppSub Tainted: G   M  D W  2.6.28-rc7-00105-gfeaf384 #4
<Jul/27 10:15 pm>[80552.646321] Call Trace:
<Jul/27 10:15 pm>[80552.646321]  <#MC>  [<ffffffff802397db>] warn_on_slowpath+0x51/0x6d
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff803eec86>] notify_update+0x2b/0x30
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff8025bc31>] smp_call_function_mask+0x37/0x1d7
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff80264013>] crash_kexec+0x17/0xef

<Jul/27 10:15 pm>[80552.646321]  [<ffffffff802640e2>] crash_kexec+0xe6/0xef
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff8021d53e>] native_smp_send_stop+0x1a/0x26
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff80239896>] panic+0x95/0x13f
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff805b9e66>] __atomic_notifier_call_chain+0x74/0x83
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff805b9df2>] __atomic_notifier_call_chain+0x0/0x83
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff8023969e>] oops_enter+0x9/0x10
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff80217b55>] mce_log+0x0/0x7f
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff80217eef>] do_machine_check+0x2d5/0x378
<Jul/27 10:15 pm>[80552.646321]  [<ffffffff8020d04f>] machine_check+0x7f/0x90
<Jul/27 10:15 pm>[80552.646321]  <<EOE>> <4>---[ end trace 4eaa2a86a8e2da22 ]---
}}}

 * 2011-08-10T03:25 kernel panic [wiki:ipp026-crash-20110810], rebooted by eam
 * 2011-09-13T15:14 kernel panic [wiki:ipp026-crash-20110913], same error as the previous two it seems, rebooted by CZW
 * 2011-09-15T18:10 kernel panic [wiki:ipp026-crash-20110915], same error, rebooted by CZW, removed from processing list
 * 2011-09-30T12:36 kernel panic, same error, rebooted by SC
 * 2012-11-09 New ARECA RAID controller installed and all data remained accessible. After this successful install/test all RAID disks swapped with new 2TB disks. OS installed and all backed up data reloaded.
 * 2012-12-09 10:15 many faults, trap invalid opcode and BUG: unable to handle kernel NULL pointer dereference at 0000000000000000. may want a reboot but MEH doesn't like to do on weekend unless absolutely necessary... not a full crash but.. [wiki:ipp026-crash-20121209]