* 2008-08-26T17:00 forcedeth hang power cycled by bills
  * 2008-09-10T18:48 forcedeth problem power cycled by gavin @ ~19:31
  * 2008-11-05T18:49 load went to 11, console hung. power cycled by eam
  * 2009-11-03T09:40:57  IDE Channel 20 Device Failed (replaced 2009-11-03T15:54:05)
  * 2009-12-31T19:42:44  IDE Channel 16 Device Failed (replaced 2010-01-04T10:53:27)
  * 2010-07-06T10:49:32  IDE Channel 5 Device Failed (replaced 2010-07-06T11:45:33)
  * 2010-09-18T14:30   Ssystem became unresponsive no response on console power cycled by bills
  * 2010-12-4  3:52:14  Channel # 1 Device Failed
  * 2010-12-30T13:54:04  system unresponsive, [5920242.434978] BUG: spinlock lockup on CPU#0, ppSub/18783, ffff88002806b480, power cycled by gavin
  * 2011-01-03T21:26:01  IDE Channel # 3 Device Failed (replaced 2011-01-04T08:43:23)
  * 2011-01-11T19:45 system down. Power cycled by bills. From console:
{{{
<Jan/11 04:44 pm>ipp021 login: [1038205.587029] BUG: spinlock lockup on CPU#3, emacs/7426, ffff88002806b480
<Jan/11 04:44 pm>[1038205.590351] BUG: spinlock lockup on CPU#2, sed/7427, ffff88002806b480
<Jan/11 04:45 pm>[1038205.589518] BUG: spinlock lockup on CPU#0, pclient/30661, ffff88002806b480
}}}
  *2011-1-17  13:59:15 Controller#1(PCI) IDE Channel # 8 Device Failed
  *2011-01-04 system became unresplonsive. Thousands of messages of this form on console. Power cycled by bills 7:30 next morning.
    * <Apr/04 07:07 pm>[5462026.705265] eth0: too many iterations (6) in nv_nic_irq.
    * <Apr/04 07:07 pm>[5462033.776453] eth0: too many iterations (6) in nv_nic_irq.

 * 2011-7-7  18:52:33 Controller#1(PCI) IDE Channel #12 Reading Error drive replaced
 * 2011-8-20  2:0:3 Controller#1(PCI) IDE Channel #21 Reading Error replaced
 * 2011-9-9  13:50    System became unresponsive. Power cycled by bills From console

{{{
<Jun/19 03:41 am>This is ipp021.ifa.hawaii.edu (Linux x86_64 2.6.28-rc7-00105-gfeaf384) 06:05:31

<Sep/09 11:48 pm>ipp021 login: [7112606.049999] BUG: spinlock lockup on CPU#3, sshd/6684, ffff88002806b480
<Sep/09 11:48 pm>[7112606.082910] BUG: spinlock lockup on CPU#2, ppImage/9163, ffff88002806b480

}}}
 * 2011-09-11 13:40 ipp021 had a panic message but kept running. Poser cycled by bills. Console said

{{{
<Sep/11 10:56 pm>ipp021 login: [169209.803118] BUG: unable to handle kernel paging request at 0000000000003138
<Sep/11 10:56 pm>[169209.806473] IP: [<ffffffff80256f86>] __lock_acquire+0x398/0x7f3
<Sep/11 10:56 pm>[169209.806473] PGD 1f840a067 PUD 2018c9067 PMD 0 
<Sep/11 10:56 pm>[169209.806473] Oops: 0002 [#1] SMP 
<Sep/11 10:56 pm>[169209.806473] last sysfs file: /sys/devices/system/cpu/sched_mc_power_savings

<Sep/11 10:56 pm>[169209.830602] CPU 3 
<Sep/11 10:56 pm>[169209.830602] Modules linked in: w83793 hwmon_vid autofs4 e1000e i2c_nforce2 k8temp i2c_core forcedeth tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
<Sep/11 10:56 pm>[169209.830602] Pid: 3168, comm: xfslogd/3 Not tainted 2.6.28-rc7-00105-gfeaf384 #4
<Sep/11 10:56 pm>[169209.859801] RIP: 0010:[<ffffffff80256f86>]  [<ffffffff80256f86>] __lock_acquire+0x398/0x7f3
<Sep/11 10:56 pm>[169209.859801] RSP: 0018:ffff88012e2c1c90  EFLAGS: 00010006
<Sep/11 10:56 pm>[169209.859801] RAX: 0000000000000002 RBX: 0000000000000046 RCX: 0000000000000000
<Sep/11 10:56 pm>[169209.859801] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88016de86070
<Sep/11 10:56 pm>[169209.859801] RBP: 0000000000003000 R08: 0000000000000001 R09: 0000000000000001
<Sep/11 10:56 pm>[169209.859801] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000286
<Sep/11 10:56 pm>[169209.859801] R13: 0000000000000000 R14: ffff88012e18f000 R15: ffff88016de86070
<Sep/11 10:56 pm>[169209.859801] FS:  00007f322a3516f0(0000) GS:ffff88022fa0c780(0000) knlGS:0000000000000000
<Sep/11 10:56 pm>[169209.926472] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<Sep/11 10:56 pm>[169209.926472] CR2: 0000000000003138 CR3: 00000001f7468000 CR4: 00000000000006e0
<Sep/11 10:56 pm>[169209.926472] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Sep/11 10:56 pm>[169209.926472] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Sep/11 10:56 pm>[169209.926472] Process xfslogd/3 (pid: 3168, threadinfo ffff88012e2c0000, task ffff88012e5d9e60)
<Sep/11 10:56 pm>[169209.960600] Stack:
<Sep/11 10:56 pm>[169209.960600]  0000000000000001 0000000000000000 ffff88012e5d9e60 0000000000000046
<Sep/11 10:56 pm>[169209.960600]  ffff88012e2c1d50 0000000000000286 0000000000000000 ffff88012e18f000
<Sep/11 10:56 pm>[169209.960600]  0000000000000003 ffffffff80257d90 0000000000000000 ffffffff8022f028
<Sep/11 10:56 pm>[169209.960600] Call Trace:
<Sep/11 10:56 pm>[169209.960600]  [<ffffffff80257d90>] ? lock_acquire+0x52/0x6b

<Sep/11 10:56 pm>[169209.960600]  [<ffffffff8022f028>] ? __wake_up+0x22/0x4e
<Sep/11 10:56 pm>[169209.960600]  [<ffffffff805b7750>] ? _spin_lock_irqsave+0x2f/0x3f
<Sep/11 10:56 pm>[169209.960600]  [<ffffffff8022f028>] ? __wake_up+0x22/0x4e
<Sep/11 10:56 pm>[169209.960600]  [<ffffffff8022f028>] ? __wake_up+0x22/0x4e
<Sep/11 10:56 pm>[169209.960600]  [<ffffffffa00b042f>] ? xfs_buf_item_unpin+0x25/0xa4 [xfs]
<Sep/11 10:56 pm>[169209.960600]  [<ffffffffa00d50d3>] ? xfs_trans_chunk_committed+0xba/0xf0 [xfs]
<Sep/11 10:56 pm>[169209.960600]  [<ffffffffa00d514d>] ? xfs_trans_committed+0x44/0xd2 [xfs]
<Sep/11 10:56 pm>[169209.960600]  [<ffffffffa00ca407>] ? xlog_state_do_callback+0x169/0x281 [xfs]
<Sep/11 10:56 pm>[169210.056458]  [<ffffffffa00e079c>] ? xfs_buf_iodone_work+0x0/0x74 [xfs]
<Sep/11 10:56 pm>[169210.056458]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
<Sep/11 10:56 pm>[169210.056458]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
<Sep/11 10:56 pm>[169210.056458]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
<Sep/11 10:56 pm>[169210.056458]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
<Sep/11 10:56 pm>[169210.056458]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
<Sep/11 10:56 pm>[169210.056458]  [<ffffffff8024becc>] ? kthread+0x47/0x73
<Sep/11 10:56 pm>[169210.056458]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
<Sep/11 10:56 pm>[169210.056458]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
<Sep/11 10:56 pm>[169210.056458]  [<ffffffff8024be85>] ? kthread+0x0/0x73
<Sep/11 10:56 pm>[169210.056458]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
<Sep/11 10:56 pm>[169210.056458] Code: 24 0c 39 55 28 74 24 e8 3d 77 14 00 85 c0 0f 84 03 fd ff ff 83 3d 72 95 d2 00 00 be 47 03 00 00 0f 84 bd 03 00 00 e9 ec fc ff ff <f0> ff 85 38 01 00 00 48 8b 44 24 10 44 8b b0 70 06 00 00 41 83 
<Sep/11 10:56 pm>[169210.056458] RIP  [<ffffffff80256f86>] __lock_acquire+0x398/0x7f3
<Sep/11 10:56 pm>[169210.056458]  RSP <ffff88012e2c1c90>

<Sep/11 10:56 pm>[169210.056458] CR2: 0000000000003138
<Sep/11 10:56 pm>[169210.056458] ---[ end trace ed4a94ecfe133cc2 ]---
<Sep/11 10:56 pm>[169219.103998] svc: 10.10.20.13, port=41442: unknown version (0 for prog 100003, nfsd)
<Sep/11 10:58 pm>[169302.808386] eth0: too many iterations (6) in nv_nic_irq.

File erased !


This is ipp021.ifa.hawaii.edu (Linux x86_64 2.6.28-rc7-00105-gfeaf384) 13:35:51

ipp021 login: bills
Password: 
Last login: Sat Jun 18 07:48:12 HST 2011 on ttyS0
(
(ipp021:~) bills% sudo shutdown now

Broadcast message from root (ttyS0) (Sun Sep 11 13:38:23 2011):

The system is going down to maintenance mode NOW!

}}}