20111122T2200 (about) ipp066 went down, from console

<Nov/22 10:04 pm>[1136649.037248] BUG: unable to handle kernel paging request at 000000006aa54080
<Nov/22 10:04 pm>[1136649.044925] IP: [<ffffffff8102fc64>] cpuacct_charge+0x5f/0x96
<Nov/22 10:04 pm>[1136649.051132] PGD 2b800e067 PUD 3eb871067 PMD 0 
<Nov/22 10:04 pm>[1136649.056106] Thread overran stack, or stack corrupted
<Nov/22 10:04 pm>[1136649.061536] Oops: 0000 [#1] SMP 
<Nov/22 10:04 pm>[1136649.065226] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.3/i2c-0/0-002f/temp6_alarm
<Nov/22 10:04 pm>[1136649.074221] CPU 2 
                 <Nov/22 10:04 pm>[1136649.076262] Modules linked in: w83793 hwmon_vid autofs4 ioatdma i2c_i801 i2c_core iTCO_wdt e1000e dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Nov/22 10:04 pm>[1136649.110721] 
<Nov/22 10:04 pm>[1136649.112603] Pid: 8371, comm: nfsd Not tainted 2.6.34.4 #3 S7010/empty
<Nov/22 10:04 pm>[1136649.119537] RIP: 0010:[<ffffffff8102fc64>]  [<ffffffff8102fc64>] cpuacct_charge+0x5f/0x96
<Nov/22 10:04 pm>[1136649.128447] RSP: 0018:ffff88061c81c110  EFLAGS: 00010086
<Nov/22 10:04 pm>[1136649.134235] RAX: 00000000000169f0 RBX: 000000001d1e6120 RCX: 000000001d1e6120
<Nov/22 10:04 pm>[1136649.142065] RDX: ffff880624ca4600 RSI: 000000000000001f RDI: 0000000000000000
<Nov/22 10:04 pm>[1136649.149895] RBP: ffff88061c81c130 R08: 0000000000000006 R09: 0000000000000000
<Nov/22 10:04 pm>[1136649.157736] R10: ffffffff81035927 R11: ffffffff81a9daa0 R12: ffff88061d1e6120
<Nov/22 10:04 pm>[1136649.165594] R13: 00000000000006b5 R14: 0001753e6cc49a37 R15: ffff880002653c00
<Nov/22 10:04 pm>[1136649.173399] FS:  0000000000000000(0000) GS:ffff880002640000(0000) knlGS:0000000000000000
<Nov/22 10:04 pm>[1136649.182186] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Nov/22 10:04 pm>[1136649.188403] CR2: 000000006aa54080 CR3: 00000003eb84a000 CR4: 00000000000006e0
<Nov/22 10:04 pm>[1136649.196239] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

<Nov/22 10:04 pm>[1136649.204011] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Nov/22 10:04 pm>[1136649.211833] Process nfsd (pid: 8371, threadinfo ffff88061c81c000, task ffff88061d1e6120)
<Nov/22 10:04 pm>[1136649.220600] Stack:
<Nov/22 10:04 pm>[1136649.223002]  ffffffff8102fc21 ffff88061d1e6120 ffff88061d1e6158 00000000000006b5
<Nov/22 10:04 pm>[1136649.230789] <0> ffff88061c81c160 ffffffff8102fe1e ffff880623503090 ffff880002653c88
<Nov/22 10:04 pm>[1136649.239241] <0> 0000000000000000 ffff880002653c00 ffff88061c81c1b0 ffffffff8103160d
<Nov/22 10:04 pm>[1136649.248030] Call Trace:
<Nov/22 10:04 pm>[1136649.250892]  [<ffffffff8102fc21>] ? cpuacct_charge+0x1c/0x96
<Nov/22 10:04 pm>[1136649.257022]  [<ffffffff8102fe1e>] update_curr+0x100/0x140
<Nov/22 10:04 pm>[1136649.262910]  [<ffffffff8103160d>] enqueue_task_fair+0x98/0x332
<Nov/22 10:04 pm>[1136649.269220]  [<ffffffff8102b548>] enqueue_task+0x5e/0x69
<Nov/22 10:04 pm>[1136649.274977]  [<ffffffff8102b659>] activate_task+0x2a/0x32
<Nov/22 10:04 pm>[1136649.280846]  [<ffffffff81035666>] pull_task+0x35/0x55
<Nov/22 10:04 pm>[1136649.286343]  [<ffffffff81037767>] load_balance+0xcab/0xf63
<Nov/22 10:04 pm>[1136649.292327]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Nov/22 10:04 pm>[1136649.298676]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
<Nov/22 10:04 pm>[1136649.304290]  [<ffffffff8151f61f>] ? _raw_spin_unlock_irq+0x26/0x2c
<Nov/22 10:04 pm>[1136649.310949]  [<ffffffff8151d35f>] schedule_timeout+0x28/0x24c
<Nov/22 10:04 pm>[1136649.317155]  [<ffffffff8151e73d>] ? __down+0x60/0x95
<Nov/22 10:04 pm>[1136649.322563]  [<ffffffff8151e745>] __down+0x68/0x95
<Nov/22 10:04 pm>[1136649.327808]  [<ffffffff81058e2d>] ? down+0xf/0x3a
<Nov/22 10:04 pm>[1136649.332980]  [<ffffffff812810b4>] ? _xfs_buf_find+0x1bb/0x273

<Nov/22 10:04 pm>[1136649.339197]  [<ffffffff81058e46>] down+0x28/0x3a
<Nov/22 10:04 pm>[1136649.344265]  [<ffffffff8127f8af>] xfs_buf_lock+0x79/0xad
<Nov/22 10:04 pm>[1136649.350034]  [<ffffffff812810b4>] _xfs_buf_find+0x1bb/0x273
<Nov/22 10:04 pm>[1136649.356068]  [<ffffffff812811cc>] xfs_buf_get+0x60/0x17e
<Nov/22 10:04 pm>[1136649.361852]  [<ffffffff81277bc9>] xfs_trans_get_buf+0xe7/0x13e
<Nov/22 10:04 pm>[1136649.368193]  [<ffffffff8124d7b6>] xfs_btree_get_buf_block+0x44/0x60
<Nov/22 10:04 pm>[1136649.374935]  [<ffffffff8124ed73>] xfs_btree_split+0x1ef/0x5d0
<Nov/22 10:04 pm>[1136649.381158]  [<ffffffff8124e223>] ? xfs_btree_lshift+0x69/0x545
<Nov/22 10:04 pm>[1136649.387563]  [<ffffffff8128121a>] ? xfs_buf_get+0xae/0x17e
<Nov/22 10:04 pm>[1136649.393508]  [<ffffffff8125001d>] xfs_btree_make_block_unfull+0x10f/0x14c
<Nov/22 10:04 pm>[1136649.400774]  [<ffffffff81250222>] xfs_btree_insrec+0x1c8/0x4fc
<Nov/22 10:04 pm>[1136649.407107]  [<ffffffff8124ca12>] ? xfs_btree_rec_addr+0xd/0x14
<Nov/22 10:04 pm>[1136649.413497]  [<ffffffff8124f8e8>] ? xfs_btree_lookup+0x256/0x3dc
                 <Nov/22 10:04 pm>[1136649.419986]  [<ffffffff812505b4>] xfs_btree_insert+0x5e/0x14a
<Nov/22 10:04 pm>[1136649.426228]  [<ffffffff8123acb4>] xfs_free_ag_extent+0x4a9/0x6a6
<Nov/22 10:04 pm>[1136649.432747]  [<ffffffff8123c5bc>] xfs_alloc_fix_freelist+0x24c/0x3f2
<Nov/22 10:04 pm>[1136649.439594]  [<ffffffff812734e7>] ? xfs_perag_get+0x59/0x99
<Nov/22 10:04 pm>[1136649.445644]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
<Nov/22 10:04 pm>[1136649.451954]  [<ffffffff812734b7>] ? xfs_perag_get+0x29/0x99
<Nov/22 10:04 pm>[1136649.457988]  [<ffffffff8123cad0>] xfs_alloc_vextent+0x2cd/0x4ad
<Nov/22 10:04 pm>[1136649.464369]  [<ffffffff812488c1>] xfs_bmap_btalloc+0x3c2/0x576
<Nov/22 10:04 pm>[1136649.470704]  [<ffffffff81248a98>] xfs_bmap_alloc+0x23/0x25

<Nov/22 10:04 pm>[1136649.476659]  [<ffffffff8124add5>] xfs_bmapi+0x7b5/0xe73
<Nov/22 10:04 pm>[1136649.482329]  [<ffffffff8126cbeb>] ? xlog_grant_log_space+0x3e0/0x484
<Nov/22 10:04 pm>[1136649.489144]  [<ffffffff8105861d>] ? down_write_nested+0x2a/0x4a
<Nov/22 10:04 pm>[1136649.495556]  [<ffffffff81267363>] xfs_iomap_write_allocate+0x227/0x353
<Nov/22 10:04 pm>[1136649.502553]  [<ffffffff81267eac>] xfs_iomap+0x318/0x39c
<Nov/22 10:04 pm>[1136649.508219]  [<ffffffff8127d1e5>] xfs_map_blocks+0x27/0x2b
<Nov/22 10:04 pm>[1136649.514166]  [<ffffffff8127e160>] xfs_page_state_convert+0x2d8/0x5b2
<Nov/22 10:04 pm>[1136649.521007]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Nov/22 10:04 pm>[1136649.527334]  [<ffffffff8127e71f>] xfs_vm_writepage+0xd9/0x113
<Nov/22 10:04 pm>[1136649.533567]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
<Nov/22 10:04 pm>[1136649.539867]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Nov/22 10:04 pm>[1136649.546197]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
<Nov/22 10:04 pm>[1136649.553149]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
<Nov/22 10:04 pm>[1136649.558997]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
<Nov/22 10:04 pm>[1136649.565812]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
<Nov/22 10:04 pm>[1136649.572466]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
<Nov/22 10:04 pm>[1136649.578699]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Nov/22 10:04 pm>[1136649.585358]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
<Nov/22 10:04 pm>[1136649.592195]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
<Nov/22 10:04 pm>[1136649.598573]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
<Nov/22 10:04 pm>[1136649.604899]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
<Nov/22 10:04 pm>[1136649.611888]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc

<Nov/22 10:04 pm>[1136649.619123]  [<ffffffff810b4993>] ra_submit+0x1c/0x20
<Nov/22 10:04 pm>[1136649.624616]  [<ffffffff810b4cb7>] ondemand_readahead+0x243/0x256
<Nov/22 10:04 pm>[1136649.631111]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
<Nov/22 10:04 pm>[1136649.637810]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
<Nov/22 10:04 pm>[1136649.644751]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
<Nov/22 10:04 pm>[1136649.656339]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Nov/22 10:04 pm>[1136649.662670]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Nov/22 10:04 pm>[1136649.668972]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
<Nov/22 10:04 pm>[1136649.675318]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
<Nov/22 10:04 pm>[1136649.681004]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
                 <Nov/22 10:04 pm>[1136649.687237]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
<Nov/22 10:04 pm>[1136649.694060]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
<Nov/22 10:04 pm>[1136649.700639]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
<Nov/22 10:04 pm>[1136649.706522]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
<Nov/22 10:04 pm>[1136649.713286]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
<Nov/22 10:04 pm>[1136649.720209]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
<Nov/22 10:04 pm>[1136649.726284]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
<Nov/22 10:04 pm>[1136649.732248]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
<Nov/22 10:04 pm>[1136649.737761]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
<Nov/22 10:04 pm>[1136649.743892]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
<Nov/22 10:04 pm>[1136649.749856]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
<Nov/22 10:04 pm>[1136649.755735]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139

<Nov/22 10:04 pm>[1136649.760988]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
<Nov/22 10:04 pm>[1136649.766158]  [<ffffffff810546e2>] kthread+0x7d/0x85
<Nov/22 10:04 pm>[1136649.771492]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Nov/22 10:04 pm>[1136649.777882]  [<ffffffff81054665>] ? kthread+0x0/0x85
<Nov/22 10:04 pm>[1136649.783298]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Nov/22 10:04 pm>[1136649.789871] Code: c7 a0 da a9 81 48 63 58 18 48 c7 04 24 21 fc 02 81 e8 52 47 03 00 49 8b 84 24 a0 0e 00 00 48 89 d9 48 8b 50 48 eb 16 48 8b 42 20 <48> 03 04 cd 80 37 b2 81 4c 01 28 48 8b 92 c8 00 00 00 48 85 d2 
<Nov/22 10:04 pm>[1136649.811185] RIP  [<ffffffff8102fc64>] cpuacct_charge+0x5f/0x96
<Nov/22 10:04 pm>[1136649.817510]  RSP <ffff88061c81c110>
<Nov/22 10:04 pm>[1136649.821450] CR2: 000000006aa54080
<Nov/22 10:04 pm>[1136649.825456] ---[ end trace 43e8e7ba86136bef ]---
<Nov/22 10:04 pm>[1136649.830562] Kernel panic - not syncing: Fatal exception in interrupt
<Nov/22 10:04 pm>[1136649.837415] Pid: 8371, comm: nfsd Tainted: G      D    2.6.34.4 #3
<Nov/22 10:04 pm>[1136649.844096] Call Trace:
<Nov/22 10:05 pm>[1<1036>[711183.86751618.684]5 61B8UG7]:  sBUpGin:l socpkin lloocckku lpo ocnku pCP Uon#5 C, PUn#fs1d, /8ks3w18a,pd f0f/6ff088,8 00ff0f26f58830c00002
<Nov/22 10:05 pm>653c00
<Nov/22 10:05 pm>[113[61711836.785168.189700] 2P75i]d: P 6i0d:8 , 83c1om8,m : cokmswm:a pdnf0s Tda Tinaitentd:e d:G   G        D   D      2. 62..364.3.44<Nov/22 10:05 pm>#3

<Nov/22 10:05 pm>[113[61711836.781568.189183] 0C94al]l C aTrlla ceT:ra
<Nov/22 10:05 pm>ce:
<Nov/22 10:05 pm>[1136[711138.68718880.8175]61 9 5]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
<Nov/22 10:05 pm>[1136718.856199]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Nov/22 10:05 pm>[1136718.856201]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a

<Nov/22 10:05 pm>[1136718.856203]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Nov/22 10:05 pm>[1136718.856206]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Nov/22 10:05 pm>[1136718.856208]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
                 <Nov/22 10:05 pm>[1136718.856211]  [<ffffffff8127e964>] xfsbufd_wakeup+0x49/0x76
<Nov/22 10:05 pm>[1136718.856213]  [<ffffffff810b8876>] shrink_slab+0x69/0x148
                 <Nov/22 10:05 pm>[1136718.856216] BUG: spinlock lockup on CPU#0, swapper/0, ffff880002653c00
<Nov/22 10:05 pm>[1136718.856220]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
<Nov/22 10:05 pm>[1136718.856222] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3
<Nov/22 10:05 pm>[1136718.856223] Call Trace:
                 <Nov/22 10:05 pm>[1136718.856225]  [<ffffffff810b9340>] kswapd+0x640/0x8bb
<Nov/22 10:05 pm>[1136718.856226]  <IRQ>  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Nov/22 10:05 pm>[1136718.856231]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
<Nov/22 10:05 pm>[1136718.856233]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
<Nov/22 10:05 pm>[1136718.856237]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Nov/22 10:05 pm>[1136718.856239]  [<ffffffff810b8d00>] ? kswapd+0x0/0x8bb
<Nov/22 10:05 pm>[1136718.856241]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Nov/22 10:05 pm>[1136718.856243]  [<ffffffff810546e2>] kthread+0x7d/0x85
<Nov/22 10:05 pm>[1136718.856245]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Nov/22 10:05 pm>[1136718.856248]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Nov/22 10:05 pm>[1136718.856250]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Nov/22 10:05 pm>[1136718.856252]  [<ffffffff81036899>] default_wake_function+0xd/0xf
<Nov/22 10:05 pm>[1136718.856255]  [<ffffffff81054665>] ? kthread+0x0/0x85

<Nov/22 10:05 pm>[1136718.856257]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
<Nov/22 10:05 pm>[1136718.856260]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Nov/22 10:05 pm>[1136718.856261]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
<Nov/22 10:05 pm>[1136718.856262] sending NMI to all CPUs:
<Nov/22 10:05 pm>[1136718.856266] NMI backtrace for cpu 0
<Nov/22 10:05 pm>[1136718.890254] BUG: spinlock lockup on CPU#8, kswapd1/609, ffff880002633c00
<Nov/22 10:05 pm>[1136718.890257] Pid: 609, comm: kswapd1 Tainted: G      D    2.6.34.4 #3
<Nov/22 10:05 pm>[1136718.890258] Call Trace:
<Nov/22 10:05 pm>[1136718.890260]  <IRQ>  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
<Nov/22 10:05 pm>[1136718.890269]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
                 <Nov/22 10:05 pm>[1136718.890272]  [<ffffffff81032141>] ? sched_rt_period_timer+0xa4/0x1c1
<Nov/22 10:05 pm>[1136718.890275]  [<ffffffff81032141>] sched_rt_period_timer+0xa4/0x1c1
<Nov/22 10:05 pm>[1136718.890277]  [<ffffffff8103209d>] ? sched_rt_period_timer+0x0/0x1c1
<Nov/22 10:05 pm>[1136718.890280]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
<Nov/22 10:05 pm>[1136718.890282]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
<Nov/22 10:05 pm>[1136718.890287]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
<Nov/22 10:05 pm>[1136718.890293]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
<Nov/22 10:05 pm>[1136718.890294]  <EOI>  [<ffffffff811cae9b>] ? nfs_access_cache_shrinker+0x2c/0x229
<Nov/22 10:05 pm>[1136718.890302]  [<ffffffff812c08b7>] ? delay_tsc+0x2d/0x59
<Nov/22 10:05 pm>[1136718.890304]  [<ffffffff812c08fe>] __delay+0xa/0xc
<Nov/22 10:05 pm>[1136718.890306]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
<Nov/22 10:05 pm>[1136718.890308]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39

<Nov/22 10:05 pm>[1136718.890312]  [<ffffffff8127e930>] ? xfsbufd_wakeup+0x15/0x76
<Nov/22 10:05 pm>[1136718.890314]  [<ffffffff8127e930>] xfsbufd_wakeup+0x15/0x76
<Nov/22 10:05 pm>[1136718.890318]  [<ffffffff810b8876>] shrink_slab+0x69/0x148
<Nov/22 10:05 pm>[1136718.890321]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
<Nov/22 10:05 pm>[1136718.890323]  [<ffffffff810b9340>] kswapd+0x640/0x8bb
<Nov/22 10:05 pm>[1136718.890325]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
<Nov/22 10:05 pm>[1136718.890328]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
<Nov/22 10:05 pm>[1136718.890330]  [<ffffffff810b8d00>] ? kswapd+0x0/0x8bb
<Nov/22 10:05 pm>[1136718.890332]  [<ffffffff810546e2>] kthread+0x7d/0x85
<Nov/22 10:05 pm>[1136718.890335]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
<Nov/22 10:05 pm>[1136718.890337]  [<ffffffff81054665>] ? kthread+0x0/0x85
<Nov/22 10:05 pm>[1136718.890339]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
<Nov/22 10:05 pm>[1136718.890340] sending NMI to all CPUs:
<Nov/22 10:05 pm>[1136719.251114] CPU 0 
<Nov/22 10:05 pm>[1136719.253220] Modules linked in: w83793 hwmon_vid autofs4 ioatdma i2c_i801 i2c_core iTCO_wdt e1000e dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
<Nov/22 10:05 pm>[1136719.288964] 
<Nov/22 10:05 pm>[1136719.290864] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3 S7010/empty
<Nov/22 10:05 pm>[1136719.298782] RIP: 0010:[<ffffffff812bfeb3>]  [<ffffffff812bfeb3>] vsnprintf+0xa9/0x41f
<Nov/22 10:05 pm>[1136719.307405] RSP: 0018:ffff880002603500  EFLAGS: 00000093
<Nov/22 10:05 pm>[1136719.313214] RAX: 0000000000000000 RBX: ffff880002603640 RCX: ffffffff8178117e
<Nov/22 10:05 pm>[1136719.321033] RDX: ffffffff81781100 RSI: 0000000000000003 RDI: ffffffff8178117e
<Nov/22 10:05 pm>[1136719.328874] RBP: ffff880002603560 R08: 0000000000000003 R09: ffffffff8178117b

<Nov/22 10:05 pm>[1136719.336698] R10: ffffffff8103bc99 R11: ffff880002603550 R12: ffffffff81c9ab90
<Nov/22 10:05 pm>[1136719.344548] R13: ffffffff8178117b R14: ffffffff81c9af90 R15: ffffffff81c9ab90
<Nov/22 10:05 pm>[1136719.352403] FS:  0000000000000000(0000) GS:ffff880002600000(0000) knlGS:0000000000000000
<Nov/22 10:05 pm>[1136719.361233] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<Nov/22 10:05 pm>[1136719.367450] CR2: 00007fcd38ff0228 CR3: 0000000386db3000 CR4: 00000000000006f0
<Nov/22 10:05 pm>[1136719.375286] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Nov/22 10:05 pm>[1136719.383151] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Nov/22 10:05 pm>[1136719.390993] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a92020)
<Nov/22 10:05 pm>[1136719.399812] Stack:
<Nov/22 10:05 pm>[1136719.402264]  0000000000000086 ffffffff81a99cd0 0000000000000400 ffffffff8178117e
<Nov/22 10:05 pm>[1136719.410207] <0> 0000000000000000 0000000000000000 ffffffff81a99c58 0000000000000400
<Nov/22 10:05 pm>[1136719.418834] <0> ffff880002603640 ffffffff8178117b 0000000000000004 ffff880002603fc0
<Nov/22 10:05 pm>[1136719.427906] Call Trace:
<Nov/22 10:05 pm>[1136719.430810]  <IRQ> 
<Nov/22 10:05 pm>[1136719.433379]  [<ffffffff812c0388>] vscnprintf+0x11/0x29
<Nov/22 10:05 pm>[1136719.439010]  [<ffffffff8103bc99>] ? vprintk+0x152/0x3a0
<Nov/22 10:05 pm>[1136719.444731]  [<ffffffff8103bcea>] vprintk+0x1a3/0x3a0
<Nov/22 10:05 pm>[1136719.450280]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
<Nov/22 10:05 pm>[1136719.456628]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
<Nov/22 10:05 pm>[1136719.463137]  [<ffffffff8151c589>] printk+0x67/0x69
<Nov/22 10:05 pm>[1136719.468406]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
<Nov/22 10:05 pm>[1136719.474926]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d

<Nov/22 10:05 pm>[1136719.481428]  [<ffffffff8151c589>] ? printk+0x67/0x69
<Nov/22 10:05 pm>[1136719.486881]  [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
<Nov/22 10:05 pm>[1136719.493393]  [<ffffffff810060bb>] print_trace_address+0x4e/0x57
<Nov/22 10:05 pm>[1136719.499822]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
<Nov/22 10:05 pm>[1136719.506332]  [<ffffffff810061e9>] print_context_stack+0x62/0xbd
<Nov/22 10:05 pm>[1136719.512790]  [<ffffffff810054e2>] dump_trace+0x23d/0x327
<Nov/22 10:05 pm>[1136719.518614]  [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
<Nov/22 10:05 pm>[1136719.524943]  [<ffffffff8100606b>] show_trace+0x10/0x12
<Nov/22 10:05 pm>[1136719.530586]  [<ffffffff8151c401>] dump_stack+0x72/0x7b
<Nov/22 10:05 pm>[1136719.536206]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
<Nov/22 10:05 pm>[1136719.542525]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Nov/22 10:05 pm>[1136719.548514]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Nov/22 10:05 pm>[1136719.554497]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Nov/22 10:05 pm>[1136719.560313]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Nov/22 10:05 pm>[1136719.566400]  [<ffffffff81036899>] default_wake_function+0xd/0xf
<Nov/22 10:05 pm>[1136719.572809]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
<Nov/22 10:05 pm>[1136719.578970]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
<Nov/22 10:05 pm>[1136719.584482]  [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
<Nov/22 10:05 pm>[1136719.590840]  [<ffffffff814f1d64>] svc_udp_data_ready+0x4e/0x7b
<Nov/22 10:05 pm>[1136719.597178]  [<ffffffff8144d67d>] sock_queue_rcv_skb+0x14e/0x179
                 <Nov/22 10:05 pm>[1136719.603686]  [<ffffffff81499a17>] __udp_queue_rcv_skb+0x12/0x6f
<Nov/22 10:05 pm>[1136719.610101]  [<ffffffff8149ae9e>] udp_queue_rcv_skb+0x1aa/0x258

<Nov/22 10:05 pm>[1136719.616531]  [<ffffffff8149b610>] __udp4_lib_rcv+0x31c/0x546
<Nov/22 10:05 pm>[1136719.622705]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
<Nov/22 10:05 pm>[1136719.629760]  [<ffffffff8149b84f>] udp_rcv+0x15/0x17
<Nov/22 10:05 pm>[1136719.635097]  [<ffffffff8147a540>] ip_local_deliver_finish+0x111/0x1d0
<Nov/22 10:05 pm>[1136719.642053]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
<Nov/22 10:05 pm>[1136719.649115]  [<ffffffff8147a671>] ip_local_deliver+0x72/0x7a
<Nov/22 10:05 pm>[1136719.655250]  [<ffffffff8147a170>] ip_rcv_finish+0x334/0x350
<Nov/22 10:05 pm>[1136719.661325]  [<ffffffff8147a409>] ip_rcv+0x27d/0x2a3
<Nov/22 10:05 pm>[1136719.666762]  [<ffffffff814570c8>] netif_receive_skb+0x3f4/0x431
<Nov/22 10:05 pm>[1136719.673205]  [<ffffffff81456e72>] ? netif_receive_skb+0x19e/0x431
<Nov/22 10:05 pm>[1136719.679793]  [<ffffffff8145792d>] napi_skb_finish+0x24/0x3c
<Nov/22 10:05 pm>[1136719.685885]  [<ffffffff8145796f>] napi_gro_receive+0x2a/0x2f
<Nov/22 10:05 pm>[1136719.696526]  [<ffffffffa01c28dd>] e1000_receive_skb+0x5c/0x65 [e1000e]
<Nov/22 10:05 pm>[1136719.703567]  [<ffffffffa01c2b32>] e1000_clean_rx_irq+0x24c/0x301 [e1000e]
<Nov/22 10:05 pm>[1136719.710891]  [<ffffffffa01c405a>] e1000_clean+0x75/0x22e [e1000e]
<Nov/22 10:05 pm>[1136719.717524]  [<ffffffff814573ba>] net_rx_action+0xb2/0x1ff
<Nov/22 10:05 pm>[1136719.723494]  [<ffffffff814574c5>] ? net_rx_action+0x1bd/0x1ff
<Nov/22 10:05 pm>[1136719.729771]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
<Nov/22 10:05 pm>[1136719.735692]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
<Nov/22 10:05 pm>[1136719.741485]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
<Nov/22 10:05 pm>[1136719.747128]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
<Nov/22 10:05 pm>[1136719.752603]  [<ffffffff8100457c>] do_IRQ+0xaa/0xc1

<Nov/22 10:05 pm>[1136719.757899]  [<ffffffff8151f913>] ret_from_intr+0x0/0xa
<Nov/22 10:05 pm>[1136719.763622]  <EOI> 
<Nov/22 10:05 pm>[1136719.766199]  [<ffffffff8126aba0>] ? _xfs_log_force_lsn+0x73/0x29c
<Nov/22 10:05 pm>[1136719.772797]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
<Nov/22 10:05 pm>[1136719.779339]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
<Nov/22 10:05 pm>[1136719.785840]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
<Nov/22 10:05 pm>[1136719.792108]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
<Nov/22 10:05 pm>[1136719.797552]  [<ffffffff815072da>] rest_init+0xbe/0xc2
<Nov/22 10:05 pm>[1136719.803083]  [<ffffffff8150721c>] ? rest_init+0x0/0xc2
<Nov/22 10:05 pm>[1136719.808713]  [<ffffffff81b3fd09>] start_kernel+0x389/0x395
<Nov/22 10:05 pm>[1136719.814730]  [<ffffffff81b3f2a1>] x86_64_start_reservations+0xb1/0xb5
<Nov/22 10:05 pm>[1136719.821708]  [<ffffffff81b3f386>] x86_64_start_kernel+0xe1/0xe8
<Nov/22 10:05 pm>[1136719.828145] Code: 89 ef e8 5d e5 ff ff 4c 63 c0 89 c6 4b 8d 54 05 00 48 89 55 b8 8a 55 c0 80 fa 11 0f 87 1c 02 00 00 0f b6 c2 ff 24 c5 c0 40 64 81 <4d> 39 f4 73 20 4c 89 f0 48 63 ce 44 89 f2 4c 29 e0 44 29 e2 4c 
<Nov/22 10:05 pm>[1136719.851413] Call Trace:
<Nov/22 10:05 pm>[1136719.854282]  <IRQ>  [<ffffffff812c0388>] vscnprintf+0x11/0x29
<Nov/22 10:05 pm>[1136719.860606]  [<ffffffff8103bc99>] ? vprintk+0x152/0x3a0
<Nov/22 10:05 pm>[1136719.866344]  [<ffffffff8103bcea>] vprintk+0x1a3/0x3a0
<Nov/22 10:05 pm>[1136719.871919]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
<Nov/22 10:05 pm>[1136719.878249]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
<Nov/22 10:05 pm>[1136719.884768]  [<ffffffff8151c589>] printk+0x67/0x69
<Nov/22 10:05 pm>[1136719.890044]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
<Nov/22 10:05 pm>[1136719.896567]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d

<Nov/22 10:05 pm>[1136719.903120]  [<ffffffff8151c589>] ? printk+0x67/0x69
<Nov/22 10:05 pm>[1136719.908561]  [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
<Nov/22 10:05 pm>[1136719.915058]  [<ffffffff810060bb>] print_trace_address+0x4e/0x57
<Nov/22 10:05 pm>[1136719.921489]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
<Nov/22 10:05 pm>[1136719.927998]  [<ffffffff810061e9>] print_context_stack+0x62/0xbd
<Nov/22 10:05 pm>[1136719.934430]  [<ffffffff810054e2>] dump_trace+0x23d/0x327
<Nov/22 10:05 pm>[1136719.940246]  [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
<Nov/22 10:05 pm>[1136719.946573]  [<ffffffff8100606b>] show_trace+0x10/0x12
<Nov/22 10:05 pm>[1136719.952226]  [<ffffffff8151c401>] dump_stack+0x72/0x7b
<Nov/22 10:05 pm>[1136719.957852]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
<Nov/22 10:05 pm>[1136719.964201]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Nov/22 10:05 pm>[1136719.970224]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Nov/22 10:05 pm>[1136719.976224]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Nov/22 10:05 pm>[1136719.982050]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Nov/22 10:05 pm>[1136719.988127]  [<ffffffff81036899>] default_wake_function+0xd/0xf
<Nov/22 10:05 pm>[1136719.994524]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
<Nov/22 10:05 pm>[1136720.000680]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
<Nov/22 10:05 pm>[1136720.006253]  [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
<Nov/22 10:05 pm>[1136720.012583]  [<ffffffff814f1d64>] svc_udp_data_ready+0x4e/0x7b
                 <Nov/22 10:05 pm>[1136720.018903]  [<ffffffff8144d67d>] sock_queue_rcv_skb+0x14e/0x179
<Nov/22 10:05 pm>[1136720.025429]  [<ffffffff81499a17>] __udp_queue_rcv_skb+0x12/0x6f
<Nov/22 10:05 pm>[1136720.031869]  [<ffffffff8149ae9e>] udp_queue_rcv_skb+0x1aa/0x258

<Nov/22 10:05 pm>[1136720.038318]  [<ffffffff8149b610>] __udp4_lib_rcv+0x31c/0x546
<Nov/22 10:05 pm>[1136720.044480]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
<Nov/22 10:05 pm>[1136720.051537]  [<ffffffff8149b84f>] udp_rcv+0x15/0x17
<Nov/22 10:05 pm>[1136720.056881]  [<ffffffff8147a540>] ip_local_deliver_finish+0x111/0x1d0
<Nov/22 10:05 pm>[1136720.063837]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
<Nov/22 10:05 pm>[1136720.070892]  [<ffffffff8147a671>] ip_local_deliver+0x72/0x7a
<Nov/22 10:05 pm>[1136720.077044]  [<ffffffff8147a170>] ip_rcv_finish+0x334/0x350
<Nov/22 10:05 pm>[1136720.083162]  [<ffffffff8147a409>] ip_rcv+0x27d/0x2a3
<Nov/22 10:05 pm>[1136720.088625]  [<ffffffff814570c8>] netif_receive_skb+0x3f4/0x431
<Nov/22 10:05 pm>[1136720.095042]  [<ffffffff81456e72>] ? netif_receive_skb+0x19e/0x431
<Nov/22 10:05 pm>[1136720.101682]  [<ffffffff8145792d>] napi_skb_finish+0x24/0x3c
<Nov/22 10:05 pm>[1136720.107748]  [<ffffffff8145796f>] napi_gro_receive+0x2a/0x2f
<Nov/22 10:05 pm>[1136720.113916]  [<ffffffffa01c28dd>] e1000_receive_skb+0x5c/0x65 [e1000e]
<Nov/22 10:05 pm>[1136720.120998]  [<ffffffffa01c2b32>] e1000_clean_rx_irq+0x24c/0x301 [e1000e]
<Nov/22 10:05 pm>[1136720.128354]  [<ffffffffa01c405a>] e1000_clean+0x75/0x22e [e1000e]
<Nov/22 10:05 pm>[1136720.134971]  [<ffffffff814573ba>] net_rx_action+0xb2/0x1ff
<Nov/22 10:05 pm>[1136720.140960]  [<ffffffff814574c5>] ? net_rx_action+0x1bd/0x1ff
<Nov/22 10:05 pm>[1136720.147221]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
<Nov/22 10:05 pm>[1136720.153139]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
<Nov/22 10:05 pm>[1136720.158931]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
<Nov/22 10:05 pm>[1136720.164532]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
<Nov/22 10:05 pm>[1136720.169985]  [<ffffffff8100457c>] do_IRQ+0xaa/0xc1

<Nov/22 10:05 pm>[1136720.175275]  [<ffffffff8151f913>] ret_from_intr+0x0/0xa
<Nov/22 10:05 pm>[1136720.180965]  <EOI>  [<ffffffff8126aba0>] ? _xfs_log_force_lsn+0x73/0x29c
<Nov/22 10:05 pm>[1136720.188277]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
<Nov/22 10:05 pm>[1136720.194772]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
<Nov/22 10:05 pm>[1136720.201283]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
<Nov/22 10:05 pm>[1136720.207547]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
<Nov/22 10:05 pm>[1136720.213018]  [<ffffffff815072da>] rest_init+0xbe/0xc2
<Nov/22 10:05 pm>[1136720.218568]  [<ffffffff8150721c>] ? rest_init+0x0/0xc2
<Nov/22 10:05 pm>[1136720.224178]  [<ffffffff81b3fd09>] start_kernel+0x389/0x395
<Nov/22 10:05 pm>[1136720.230153]  [<ffffffff81b3f2a1>] x86_64_start_reservations+0xb1/0xb5
<Nov/22 10:05 pm>[1136720.237133]  [<ffffffff81b3f386>] x86_64_start_kernel+0xe1/0xe8
<Nov/22 10:05 pm>[1136720.243569] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3
<Nov/22 10:05 pm>[1136720.250236] Call Trace:
<Nov/22 10:05 pm>[1136720.253110]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
<Nov/22 10:05 pm>[1136720.259511]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
<Nov/22 10:05 pm>[1136720.265864]  [<ffffffff81520360>] do_nmi+0xb2/0x259
<Nov/22 10:05 pm>[1136720.271220]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
<Nov/22 10:05 pm>[1136720.276251]  [<ffffffff8103bc99>] ? vprintk+0x152/0x3a0
                 <Nov/22 10:05 pm>[1136720.281959]  [<ffffffff812bfeb3>] ? vsnprintf+0xa9/0x41f
<Nov/22 10:05 pm>[1136720.287757]  <<EOE>>  <IRQ>  [<ffffffff812c0388>] vscnprintf+0x11/0x29
<Nov/22 10:05 pm>[1136720.294937]  [<ffffffff8103bc99>] ? vprintk+0x152/0x3a0
<Nov/22 10:05 pm>[1136720.300663]  [<ffffffff8103bcea>] vprintk+0x1a3/0x3a0

<Nov/22 10:05 pm>[1136720.306198]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
<Nov/22 10:05 pm>[1136720.312587]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
<Nov/22 10:05 pm>[1136720.319106]  [<ffffffff8151c589>] printk+0x67/0x69
<Nov/22 10:05 pm>[1136720.324391]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
<Nov/22 10:05 pm>[1136720.330913]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
                 <Nov/22 10:05 pm>[1136720.337431]  [<ffffffff8151c589>] ? printk+0x67/0x69
<Nov/22 10:05 pm>[1136720.342892]  [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
<Nov/22 10:05 pm>[1136720.349413]  [<ffffffff810060bb>] print_trace_address+0x4e/0x57
<Nov/22 10:05 pm>[1136720.355845]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
<Nov/22 10:05 pm>[1136720.362361]  [<ffffffff810061e9>] print_context_stack+0x62/0xbd
<Nov/22 10:05 pm>[1136720.368802]  [<ffffffff810054e2>] dump_trace+0x23d/0x327
<Nov/22 10:05 pm>[1136720.374626]  [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
<Nov/22 10:05 pm>[1136720.380965]  [<ffffffff8100606b>] show_trace+0x10/0x12
<Nov/22 10:05 pm>[1136720.386589]  [<ffffffff8151c401>] dump_stack+0x72/0x7b
<Nov/22 10:05 pm>[1136720.392208]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
<Nov/22 10:05 pm>[1136720.398522]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
<Nov/22 10:05 pm>[1136720.404502]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
<Nov/22 10:05 pm>[1136720.410492]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
<Nov/22 10:05 pm>[1136720.416292]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
<Nov/22 10:05 pm>[1136720.422379]  [<ffffffff81036899>] default_wake_function+0xd/0xf
<Nov/22 10:05 pm>[1136720.428811]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
<Nov/22 10:05 pm>[1136720.435000]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f

<Nov/22 10:05 pm>[1136720.440566]  [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
<Nov/22 10:05 pm>[1136720.446912]  [<ffffffff814f1d64>] svc_udp_data_ready+0x4e/0x7b
<Nov/22 10:05 pm>[1136720.453258]  [<ffffffff8144d67d>] sock_queue_rcv_skb+0x14e/0x179
<Nov/22 10:05 pm>[1136720.459795]  [<ffffffff81499a17>] __udp_queue_rcv_skb+0x12/0x6f
<Nov/22 10:05 pm>[1136720.466223]  [<ffffffff8149ae9e>] udp_queue_rcv_skb+0x1aa/0x258
<Nov/22 10:05 pm>[1136720.472654]  [<ffffffff8149b610>] __udp4_lib_rcv+0x31c/0x546
<Nov/22 10:05 pm>[1136720.478809]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
<Nov/22 10:05 pm>[1136720.485840]  [<ffffffff8149b84f>] udp_rcv+0x15/0x17
<Nov/22 10:05 pm>[1136720.491236]  [<ffffffff8147a540>] ip_local_deliver_finish+0x111/0x1d0
<Nov/22 10:05 pm>[1136720.498193]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
<Nov/22 10:05 pm>[1136720.505283]  [<ffffffff8147a671>] ip_local_deliver+0x72/0x7a
<Nov/22 10:05 pm>[1136720.511442]  [<ffffffff8147a170>] ip_rcv_finish+0x334/0x350
<Nov/22 10:05 pm>[1136720.517546]  [<ffffffff8147a409>] ip_rcv+0x27d/0x2a3
<Nov/22 10:05 pm>[1136720.523016]  [<ffffffff814570c8>] netif_receive_skb+0x3f4/0x431
<Nov/22 10:05 pm>[1136720.529449]  [<ffffffff81456e72>] ? netif_receive_skb+0x19e/0x431
<Nov/22 10:05 pm>[1136720.536072]  [<ffffffff8145792d>] napi_skb_finish+0x24/0x3c
<Nov/22 10:05 pm>[1136720.542155]  [<ffffffff8145796f>] napi_gro_receive+0x2a/0x2f
<Nov/22 10:05 pm>[1136720.548304]  [<ffffffffa01c28dd>] e1000_receive_skb+0x5c/0x65 [e1000e]
<Nov/22 10:05 pm>[1136720.555369]  [<ffffffffa01c2b32>] e1000_clean_rx_irq+0x24c/0x301 [e1000e]
<Nov/22 10:05 pm>[1136720.562695]  [<ffffffffa01c405a>] e1000_clean+0x75/0x22e [e1000e]
<Nov/22 10:05 pm>[1136720.569302]  [<ffffffff814573ba>] net_rx_action+0xb2/0x1ff
<Nov/22 10:05 pm>[1136720.575298]  [<ffffffff814574c5>] ? net_rx_action+0x1bd/0x1ff

<Nov/22 10:05 pm>[1136720.581566]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
<Nov/22 10:05 pm>[1136720.587441]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
<Nov/22 10:05 pm>[1136720.593243]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
<Nov/22 10:05 pm>[1136720.598844]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
<Nov/22 10:05 pm>[1136720.604300]  [<ffffffff8100457c>] do_IRQ+0xaa/0xc1
<Nov/22 10:05 pm>[1136720.609570]  [<ffffffff8151f913>] ret_from_intr+0x0/0xa
<Nov/22 10:05 pm>[1136720.615302]  <EOI>  [<ffffffff8126aba0>] ? _xfs_log_force_lsn+0x73/0x29c
<Nov/22 10:05 pm>[1136720.622589]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
                 <Nov/22 10:05 pm>[1136720.629110]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
<Nov/22 10:05 pm>[1136720.635639]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
<Nov/22 10:05 pm>[1136720.641931]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
<Nov/22 10:05 pm>[1136720.647389]  [<ffffffff815072da>] rest_init+0xbe/0xc2
<Nov/22 10:05 pm>[1136720.652906]  [<ffffffff8150721c>] ? rest_init+0x0/0xc2
<Nov/22 10:05 pm>[1136720.658534]  [<ffffffff81b3fd09>] start_kernel+0x389/0x395
<Nov/22 10:05 pm>[1136720.664534]  [<ffffffff81b3f2a1>] x86_64_start_reservations+0xb1/0xb5
<Nov/22 10:05 pm>[1136720.671513]  [<ffffffff81b3f386>] x86_64_start_kernel+0xe1/0xe8
<Nov/22 10:05 pm>[1136720.677924] NMI backtrace for cpu 8