{{{
<Oct/18 02:29 pm>[2166927.212444] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 02:38 pm>[2167453.279670] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 02:44 pm>[2167820.053114] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 02:44 pm>[2167829.825284] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 02:45 pm>[2167843.299182] eth0: too many iterations (61) in nv_nic_irq.

<Oct/18 02:49 pm>[2168083.915312] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 02:50 pm>[2168137.272467] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 02:52 pm>[2168268.603536] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 02:53 pm>[2168328.301594] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 02:55 pm>[2168464.712749] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 03:00 pm>[2168767.112347] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 03:01 pm>[2168822.652157] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 03:01 pm>[2168831.937728] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 03:07 pm>[2169186.736898] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 03:07 pm>[2169192.823571] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 03:10 pm>[2169376.933742] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 03:17 pm>[2169780.440498] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 03:19 pm>[2169914.502023] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 03:20 pm>[2169945.596265] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 03:44 pm>[2171427.495663] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 03:44 pm>[2171427.945067] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 03:44 pm>[2171430.046570] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 03:45 pm>[2171441.611634] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 04:00 pm>[2172362.396403] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 04:01 pm>[2172412.597991] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 04:28 pm>[2174016.949615] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 04:28 pm>[2174050.794724] eth0: too many iterations (61) in nv_nic_irq.

<Oct/18 04:57 pm>[2175763.346162] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 04:57 pm>[2175804.589231] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 04:58 pm>[2175825.360519] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 04:58 pm>[2175827.874278] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 05:07 pm>[2176394.207316] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 05:07 pm>[2176394.224308] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 05:13 pm>[2176729.665140] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 05:24 pm>[2177413.481094] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 05:26 pm>[2177548.386101] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 06:26 pm>[2181141.574758] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 06:27 pm>[2181165.759276] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 06:58 pm>[2183047.815782] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 07:00 pm>[2183165.908889] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 07:00 pm>[2183181.882869] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 07:01 pm>[2183209.592358] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 07:17 pm>[2184212.086125] eth0: too many iterations (61) in nv_nic_irq.
<Oct/18 07:18 pm>[2184269.178471] swapper: page allocation failure. order:0, mod<Oct/18 07:18 pm>
                 <Oct/18 07:18 pm>[2184269.181783] Pid: 0, comm: swapper Not tainted 2.6.28-rc7-00105-gfeaf384 #4
<Oct/18 07:18 pm>[2184269.188379] Call Trace:
<Oct/18 07:18 pm>[2184269.188379]  <IRQ>  [<ffffffff802821c3>] __alloc_pages_internal+0x3ab/0x3c4
<Oct/18 07:18 pm>[2184269.202432]  [<ffffffff802a543d>] new_slab+0xbf/0x274
<Oct/18 07:18 pm>[2184269.202432]  [<ffffffff802a5875>] __slab_alloc+0x207/0x3ed

<Oct/18 07:18 pm>[2184269.212427]  [<ffffffff8052e3d7>] ip_local_deliver+0x15e/0x164
<Oct/18 07:18 pm>[2184269.219094]  [<ffffffff80511430>] dev_alloc_skb+0x16/0x2d
<Oct/18 07:18 pm>[2184269.219094]  [<ffffffff80251709>] getnstimeofday+0x39/0x98
<Oct/18 07:18 pm>[2184269.219094]  [<ffffffff80511430>] dev_alloc_skb+0x16/0x2d
<Oct/18 07:18 pm>[2184269.219094]  [<ffffffff802a6c76>] __kmalloc_node_track_cal<Oct/18 07:18 pm>
                 <Oct/18 07:18 pm>[2184269.219094]  [<ffffffff805110b6>] __alloc_skb+0x69/0x132
<Oct/18 07:18 pm>[2184269.219094]  [<ffffffff80511430>] dev_alloc_skb+0x16/0x2d
<Oct/18 07:18 pm>[2184269.255759]  [<ffffffffa016f891>] nv_alloc_rx_optimized+0x53/0x1b8 [forcedeth]
<Oct/18 07:18 pm>[2184269.255759]  [<ffffffffa0171087>] nv_nic_irq_optimized+0xa1/0x232 [forcedeth]
<Oct/18 07:18 pm>[2184269.255759]  [<ffffffff80277305>] handle_IRQ_event+0x1e/0x4c
<Oct/18 07:18 pm>[2184269.255759]  [<ffffffff80278cbb>] handle_fasteoi_irq+0x90/0xcf
<Oct/18 07:18 pm>[2184269.255759]  [<ffffffff8020e15b>] do_IRQ+0xf7/0x16c
<Oct/18 07:18 pm>[2184269.255759]  [<ffffffff8020c2f6>] ret_from_intr+0x0/0xa
<Oct/18 07:18 pm>[2184269.255759]  <EOI>  [<ffffffff80212496>] c1e_idle+0x0/0xe9
<Oct/18 07:18 pm>[2184269.305793]  [<ffffffff805b9df2>] __atomic_notifier_call_chain+0x0/0x83
<Oct/18 07:18 pm>[2184269.312427]  [<ffffffff8021237c>] default_idle+0x27/0x3b
<Oct/18 07:18 pm>[2184269.312427]  [<ffffffff8021257b>] c1e_idle+0xe5/0xe9
<Oct/18 07:18 pm>[2184269.312427]  [<ffffffff8020ac41>] cpu_idle+0x4a/0x6d
<Oct/18 07:18 pm>[2184269.312427] Mem-Info:
<Oct/18 07:18 pm>[2184269.312427] Node 0 DMA per-cpu:
<Oct/18 07:18 pm>[2184269.312427] CPU    0: hi:    0, btch:   1 usd:   0
<Oct/18 07:18 pm>[2184269.312427] CPU    1: hi:    0, btch:   1 usd:   0

<Oct/18 07:18 pm>[2184269.312427] CPU    2: hi:    0, btch:   1 usd:   0
<Oct/18 07:18 pm>[2184269.312427] CPU    3: hi:    0, btch:   1 usd:   0
<Oct/18 07:18 pm>[2184269.312427] Node 0 DMA32 per-cpu:
<Oct/18 07:18 pm>[2184269.312427] CPU    0: hi:  186, btch:  31 usd: 151
<Oct/18 07:18 pm>[2184269.312427] CPU    1: hi:  186, btch:  31 usd: 171
<Oct/18 07:18 pm>[2184269.312427] CPU    2: hi:  186, btch:  31 usd: 179
<Oct/18 07:18 pm>[2184269.312427] CPU    3: hi:  186, btch:  31 usd: 182
<Oct/18 07:18 pm>[2184269.312427] Node 0 Normal per-cpu:
<Oct/18 07:18 pm>[2184269.312427] CPU    0: hi:  186, btch:  31 usd:  16
<Oct/18 07:18 pm>[2184269.312427] CPU    1: hi:  186, btch:  31 usd: 166
<Oct/18 07:18 pm>[2184269.312427] CPU    2: hi:  186, btch:  31 usd: 184
<Oct/18 07:18 pm>[2184269.312427] CPU    3: hi:  186, btch:  31 usd: 161
<Oct/18 07:18 pm>[2184269.312427] Node 1 Normal per-cpu:
<Oct/18 07:18 pm>[2184269.312427] CPU    0: hi:  186, btch:  31 usd: 185
                 <Oct/18 07:18 pm>[2184269.312427] CPU    1: hi:  186, btch:  31 usd: 156
<Oct/18 07:18 pm>[2184269.312427] CPU    2: hi:  186, btch:  31 usd:  49
<Oct/18 07:18 pm>[2184269.312427] CPU    3: hi:  186, btch:  31 usd:  42
<Oct/18 07:18 pm>[2184269.312427] Active_anon:22076 active_file:11679 inactive_anon:18930
<Oct/18 07:18 pm>[2184269.312427]  inactive_file:1844135 unevictable:0 dirty:26247 writeback:37 unstable:0
<Oct/18 07:18 pm>[2184269.312427]  free:8542 slab:49168 mapped:1900 pagetables:8256 bounce:0
<Oct/18 07:18 pm>[2184269.312427] Node 0 DMA free:2516kB min:0kB low:0kB high:0kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:1136kB pages_scanned:0 all_unreclaimable? yes
<Oct/18 07:18 pm>[2184269.312427] lowmem_reserve[]: 0 3223 3975 3975

<Oct/18 07:18 pm>[2184269.312427] Node 0 DMA32 free:16976kB min:4616kB low:5768kB high:6924kB active_anon:18668kB inactive_anon:13784kB active_file:16328kB inactive_file:3063460kB unevictable:0kB present:3301088kB pages_scanned:0 all_unreclaimable? no
<Oct/18 07:18 pm>[2184269.312427] lowmem_reserve[]: 0 0 751 751
<Oct/18 07:18 pm>[2184269.312427] Node 0 Normal free:1784kB min:1076kB low:1344kB high:1612kB active_anon:9900kB inactive_anon:36320kB active_file:13452kB inactive_file:681840kB unevictable:0kB present:769536kB pages_scanned:0 all_unreclaim<Oct/18 07:18 pm>
                 <Oct/18 07:18 pm>[2184269.312427] lowmem_reserve[]: 0 0 0 0
<Oct/18 07:18 pm>[2184269.312427] Node 1 Normal free:12916kB min:5740kB low:7172kB high:8608kB active_anon:59740kB inactive_anon:25616kB active_file:17072kB inactive_file:3631316kB unevictable:0kB present:4104192kB pages_scanned:0 all_unreclaimable? no
<Oct/18 07:18 pm>[2184269.312427] lowmem_reserve[]: 0 0 0 0
<Oct/18 07:18 pm>[2184269.312427] Node 0 DMA: 3*4kB 3*8kB 3*16kB 4*32kB 6*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2516kB
<Oct/18 07:18 pm>[2184269.312427] Node 0 DMA32: 673*4kB 66*8kB 99*16kB 267*32kB 14*64kB 8*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 17060kB
<Oct/18 07:18 pm>[2184269.312427] Node 0 Normal: 188*4kB 16*8kB 6*16kB 8*32kB 3*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1808kB
<Oct/18 07:18 pm>[2184269.312427] Node 1 Normal: 225*4kB 3*8kB 101*16kB 223*32kB 20*64kB 4*128kB 2*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 13004kB
<Oct/18 07:18 pm>[2184269.312427] 1857164 total pagecache pages
<Oct/18 07:18 pm>[2184269.312427] 1326 pages in swap cache
<Oct/18 07:18 pm>[2184269.312427] Swap cache stats: add 422665, delete 421339, find 29972534/30003223
<Oct/18 07:18 pm>[2184269.312427] Free swap  = 31168016kB
<Oct/18 07:18 pm>[2184269.312427] Total swap = 31999992kB
<Oct/18 07:18 pm>[2184269.312427] 2097136 pages RAM
<Oct/18 07:18 pm>[2184269.312427] 86330 pages reserved
<Oct/18 07:18 pm>[2184269.312427] 1006061 pages shared
<Oct/18 07:18 pm>[2184269.312427] 1023918 pages non-shared
<Oct/18 07:22 pm>[2184425.056152] BUG: spinlock lockup on CPU#3, xfslogd/3/3168, ffff88000ccd8458
<Oct/18 07:22 pm>[2184425.056152] Pid: 3168, comm: xfslogd/3 Not tainted 2.6.28-rc7-00105-gfeaf384 #4
<Oct/18 07:22 pm>[2184425.056152] Call Trace:

<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff803a103f>] _raw_spin_lock+0xfb/0x122
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff805b7758>] _spin_lock_irqsave+0x37/0x3f
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff8022f028>] __wake_up+0x22/0x4e
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff8022f028>] __wake_up+0x22/0x4e
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffffa00b042f>] xfs_buf_item_unpin+0x25/0xa4 [xfs]
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffffa00d50d3>] xfs_trans_chunk_committed+0xba/0xf0 [xfs]
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffffa00d514d>] xfs_trans_committed+0x44/0xd2 [xfs]
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffffa00ca407>] xlog_state_do_callback+0x169/0x281 [xfs]
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffffa00e079c>] xfs_buf_iodone_work+0x0/0x74 [xfs]
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff80248973>] run_workqueue+0xed/0x1ed
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff8024891d>] run_workqueue+0x97/0x1ed
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff80249511>] worker_thread+0xd8/0xe3
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff8024bfda>] autoremove_wake_function+0x0/0x2e
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff80249439>] worker_thread+0x0/0xe3
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff8024becc>] kthread+0x47/0x73
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff80234852>] schedule_tail+0x27/0x5f
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff8020cd19>] child_rip+0xa/0x11
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff80212496>] c1e_idle+0x0/0xe9
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff8024be85>] kthread+0x0/0x73
<Oct/18 07:22 pm>[2184425.056152]  [<ffffffff8020cd0f>] child_rip+0x0/0x11
<Oct/18 07:24 pm>[2184595.389048] BUG: soft lockup - CPU#2 stuck for 61s! [sshd:23859]
<Oct/18 07:24 pm>[2184595.389048] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt k8temp forcedeth i2c_nforce2 i2c_core tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan

<Oct/18 07:24 pm>[2184595.389048] CPU 2:
<Oct/18 07:24 pm>[2184595.389048] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt k8temp forcedeth i2c_nforce2 i2c_core tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
<Oct/18 07:24 pm>[2184595.389048] Pid: 23859, comm: sshd Not tainted 2.6.28-rc7-00105-gfeaf384 #4
<Oct/18 07:24 pm>[2184595.389048] RIP: 0010:[<ffffffff8021d9cd>]  [<ffffffff8021d9cd>] native_flush_tlb_others+0x8e/0xb6
<Oct/18 07:24 pm>[2184595.389048] RSP: 0000:ffff8801a0c01da8  EFLAGS: 00000202
<Oct/18 07:24 pm>[2184595.389048] RAX: 00000000000008f2 RBX: 0000000000000002 RCX: ffff880135a43198
<Oct/18 07:24 pm>[2184595.389048] RDX: 00000000000008f2 RSI: 00000000000000f2 RDI: 0000000000000008
<Oct/18 07:24 pm>[2184595.389048] RBP: 0000000000000002 R08: 0000000000000001 R09: 0000000000000000
<Oct/18 07:24 pm>[2184595.389048] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000058
<Oct/18 07:24 pm>[2184595.389048] R13: ffffe2000842b3a0 R14: 0000000000000246 R15: ffff88017096f680
<Oct/18 07:24 pm>[2184595.389048] FS:  00007f79506856f0(0000) GS:ffff88022fa0d300(0000) knlGS:000000005978eb90
<Oct/18 07:24 pm>[2184595.389048] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Oct/18 07:24 pm>[2184595.389048] CR2: 00007fff586a2fe8 CR3: 0000000187b58000 CR4: 00000000000006e0
<Oct/18 07:24 pm>[2184595.389048] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Oct/18 07:24 pm>[2184595.389048] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Oct/18 07:24 pm>[2184595.389048] Call Trace:
<Oct/18 07:24 pm>[2184595.389048]  [<ffffffff8021d9c9>] ? native_flush_tlb_others+0x8a/0xb6
<Oct/18 07:24 pm>[2184595.389048]  [<ffffffff8021dabd>] ? flush_tlb_page+0x5e/0x65
<Oct/18 07:24 pm>[2184595.389048]  [<ffffffff802299c3>] ? ptep_set_access_flags+0x1b/0x1f
<Oct/18 07:24 pm>[2184595.389048]  [<ffffffff8028ad20>] ? do_wp_page+0x4c8/0x55b
<Oct/18 07:24 pm>[2184595.389048]  [<ffffffff8028c4fa>] ? handle_mm_fault+0x6ca/0x728
<Oct/18 07:24 pm>[2184595.389048]  [<ffffffff805b99e1>] ? do_page_fault+0x424/0x7df

<Oct/18 07:24 pm>[2184595.389048]  [<ffffffff805b7141>] ? lockdep_sys_exit_thunk+0x35/0x67
<Oct/18 07:24 pm>[2184595.389048]  [<ffffffff805b795d>] ? error_exit+0x0/0x9a

File erased !

}}}