Checked ganglia this morning. Lots of failed jobs. ipp005 high load. Ganglia shows that started around 11pm last night. Here are the contents of the console.

<Oct/20 08:42 am>This is ipp005.ifa.hawaii.edu (Linux x86_64 2.6.28-rc7-00105-gfeaf384) 11:02:14

<Oct/29 06:08 am>ipp005 login: [768588.516988] 
<Oct/29 06:08 am>[768588.516991] =========================
<Oct/29 06:08 am>[768588.520874] [ BUG: held lock freed! ]
<Oct/29 06:08 am>[768588.520874] -------------------------
<Oct/29 06:08 am>[768588.520874] nfsd/7639 is freeing memory ffff8801a5d9c000-ffff8801a5d9c2b7, with a lock still held there!
<Oct/29 06:08 am>[768588.520874]  (&(&ip->i_lock)->mr_lock){....}, at: [<ffffffffa00c1bbb>] xfs_iget_core+0x3de/0x5ae [xfs]

<Oct/29 06:08 am>[768588.520874] 2 locks held by nfsd/7639:
<Oct/29 06:08 am>[768588.520874]  #0:  (hash_sem){....}, at: [<ffffffff8034f2fd>] nfsd+0x1b7/0x296
<Oct/29 06:08 am>[768588.520874]  #1:  (&(&ip->i_lock)->mr_lock){....}, at: [<ffffffffa00c1bbb>] xfs_iget_core+0x3de/0x5ae [xfs]
<Oct/29 06:08 am>[768588.520874] 
<Oct/29 06:08 am>[768588.520874] stack backtrace:
<Oct/29 06:08 am>[768588.520874] Pid: 7639, comm: nfsd Not tainted 2.6.28-rc7-00105-gfeaf384 #4
<Oct/29 06:08 am>[768588.520874] Call Trace:
                 <Oct/29 06:08 am>[768588.520874]  [<ffffffff80257acf>] debug_check_no_locks_freed+0xda/0x100
<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00c1d70>] xfs_iget_core+0x593/0x5ae [xfs]
<Oct/29 06:08 am>[768588.520874]  [<ffffffff802a5d85>] kmem_cache_free+0x6a/0xab
<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00c1d70>] xfs_iget_core+0x593/0x5ae [xfs]
<Oct/29 06:08 am>[768588.520874]  [<ffffffff805b7388>] _spin_unlock+0x17/0x20
<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00c1e6d>] xfs_iget+0xe2/0x189 [xfs]
<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00e0b78>] xfs_nfs_get_inode+0x39/0x88 [xfs]
<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00e0c62>] xfs_fs_fh_to_dentry+0x69/0x76 [xfs]
<Oct/29 06:08 am>[768588.520874]  [<ffffffff8034e8ff>] exportfs_decode_fh+0x30/0x1cd
<Oct/29 06:08 am>[768588.520874]  [<ffffffff80351f21>] nfsd_acceptable+0x0/0xd3
<Oct/29 06:08 am>[768588.520874]  [<ffffffff803562ca>] exp_find_key+0x96/0xa9
<Oct/29 06:08 am>[768588.520874]  [<ffffffff8050ac9a>] sock_recvmsg+0xd5/0xed
<Oct/29 06:08 am>[768588.520874]  [<ffffffff8022eda5>] place_entity+0x52/0x75
<Oct/29 06:08 am>[768588.520874]  [<ffffffff80356346>] exp_find+0x69/0x74
<Oct/29 06:08 am>[768588.520874]  [<ffffffff803517a4>] fh_verify+0x269/0x54d

<Oct/29 06:08 am>[768588.681209]  [<ffffffff80353053>] nfsd_open+0x1f/0x17b
<Oct/29 06:08 am>[768588.681209]  [<ffffffff80353530>] nfsd_read+0x7f/0xc4
                 <Oct/29 06:08 am>[768588.681209]  [<ffffffff8035a34b>] nfsd3_proc_read+0xfe/0x141
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8034ecd8>] nfsd_dispatch+0xde/0x1c2
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8058febb>] svc_process+0x408/0x6e9
<Oct/29 06:08 am>[768588.681209]  [<ffffffff805b70a2>] __down_read+0x34/0x9e
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8034f305>] nfsd+0x1bf/0x296
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8034f146>] nfsd+0x0/0x296
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8024becc>] kthread+0x47/0x73
<Oct/29 06:08 am>[768588.681209]  [<ffffffff80234852>] schedule_tail+0x27/0x5f
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8020cd19>] child_rip+0xa/0x11
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8024be85>] kthread+0x0/0x73
<Oct/29 06:08 am>[768588.681209]  [<ffffffff8020cd0f>] child_rip+0x0/0x11
<Oct/29 08:42 am>[777846.860683] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 08:49 am>[778280.004590] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 10:18 am>[783590.637909] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 10:50 am>[785500.050813] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 11:53 am>[789307.308652] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 12:04 pm>[789975.432215] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 12:18 pm>[790831.449968] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 12:18 pm>[790835.238038] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 02:20 pm>[798133.388035] eth0: too many iterations (61) in nv_nic_irq.

<Oct/29 03:42 pm>[803028.043815] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 04:21 pm>[805413.358445] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 04:25 pm>[805631.004822] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 04:45 pm>[806813.000336] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 05:12 pm>[808422.702304] eth0: too many iterations (61) in nv_nic_irq.
<Oct/29 08:31 pm>[820402.099219] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
<Oct/29 08:31 pm>[820402.099219] IP: [<ffffffff8022dfe5>] __wake_up_common+0x28/0x74
<Oct/29 08:31 pm>[820402.099219] PGD 1c7d97067 PUD 1c7d50067 PMD 0 
<Oct/29 08:31 pm>[820402.099219] Oops: 0000 [#1] SMP 
<Oct/29 08:31 pm>[820402.099219] last sysfs file: /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
<Oct/29 08:31 pm>[820402.099219] CPU 3 
<Oct/29 08:31 pm>[820402.099219] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt i2c_nforce2 k8temp i2c_core forcedeth tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
<Oct/29 08:31 pm>[820402.099219] Pid: 3168, comm: xfslogd/3 Not tainted 2.6.28-rc7-00105-gfeaf384 #4
<Oct/29 08:31 pm>[820402.099219] RIP: 0010:[<ffffffff8022dfe5>]  [<ffffffff8022dfe5>] __wake_up_common+0x28/0x74
<Oct/29 08:31 pm>[820402.099219] RSP: 0018:ffff88012e06bce0  EFLAGS: 00010097
<Oct/29 08:31 pm>[820402.099219] RAX: 0000000000000286 RBX: ffff880161aa7858 RCX: 0000000000000000
<Oct/29 08:31 pm>[820402.099219] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880161aa7858
<Oct/29 08:31 pm>[820402.099219] RBP: ffff88012e06bd10 R08: ffffffffffffffe8 R09: 0000000000000001
<Oct/29 08:31 pm>[820402.099219] R10: 0000000000000000 R11: ffffffff8039d94e R12: 0000000000000000
<Oct/29 08:31 pm>[820402.099219] R13: 0000000000000000 R14: ffff880161aa7858 R15: 0000000000000000
<Oct/29 08:31 pm>[820402.099219] FS:  0000000040eeb950(0000) GS:ffff88022fa0d780(0000) knlGS:0000000000000000
<Oct/29 08:31 pm>[820402.099219] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b

<Oct/29 08:31 pm>[820402.099219] CR2: 0000000000000000 CR3: 00000001c7d5e000 CR4: 00000000000006e0
<Oct/29 08:31 pm>[820402.099219] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Oct/29 08:31 pm>[820402.099219] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Oct/29 08:31 pm>[820402.099219] Process xfslogd/3 (pid: 3168, threadinfo ffff88012e06a000, task ffff88012e1b0f30)
<Oct/29 08:31 pm>[820402.099219] Stack:
<Oct/29 08:31 pm>[820402.099219]  0000000300000000 ffff880161aa7858 0000000000000000 0000000000000000
<Oct/29 08:31 pm>[820402.099219]  0000000000000286 0000000000000003 ffff88012e06bd50 ffffffff8022f03e
<Oct/29 08:31 pm>[820402.099219]  ffffffffffffff6e ffff880016af5001 0000000000000000 ffff880161aa7800
<Oct/29 08:31 pm>[820402.099219] Call Trace:
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8022f03e>] __wake_up+0x38/0x4e
<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00b042f>] xfs_buf_item_unpin+0x25/0xa4 [xfs]
<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00d50d3>] ? xfs_trans_chunk_committed+0xba/0xf0 [xfs]
<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00d514d>] ? xfs_trans_committed+0x44/0xd2 [xfs]
<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00ca407>] ? xlog_state_do_callback+0x169/0x281 [xfs]
<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00e079c>] ? xfs_buf_iodone_work+0x0/0x74 [xfs]
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8024becc>] ? kthread+0x47/0x73
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f

<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80212496>] ? c1e_idle+0x0/0xe9
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8024be85>] ? kthread+0x0/0x73
<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
<Oct/29 08:31 pm>[820402.099219] Code: e5 c9 c3 55 48 89 e5 41 57 4d 89 c7 41 56 49 89 fe 41 55 41 54 41 89 d4 53 48 83 ec 08 89 75 d4 89 4d d0 4c 8b 47 30 49 83 e8 18 <49> 8b 58 18 eb 29 45 8b 28 4c 89 f9 8b 55 d0 8b 75 d4 4c 89 c7 
<Oct/29 08:31 pm>[820402.099219] RIP  [<ffffffff8022dfe5>] __wake_up_common+0x28/0x74
<Oct/29 08:31 pm>[820402.099219]  RSP <ffff88012e06bce0>
<Oct/29 08:31 pm>[820402.099219] CR2: 0000000000000000
<Oct/29 08:31 pm>[820402.099219] ---[ end trace 06a2b6b5cc51d31c ]---

File erased !