IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Version 1 and Version 2 of ipp066_log


Ignore:
Timestamp:
Aug 2, 2012, 10:53:25 AM (14 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • ipp066_log

    v1 v2  
    1  20111122T2200 (about) ipp066 went down, from console
    2 {{{
    3 <Nov/22 10:04 pm>[1136649.037248] BUG: unable to handle kernel paging request at 000000006aa54080
    4 <Nov/22 10:04 pm>[1136649.044925] IP: [<ffffffff8102fc64>] cpuacct_charge+0x5f/0x96
    5 <Nov/22 10:04 pm>[1136649.051132] PGD 2b800e067 PUD 3eb871067 PMD 0
    6 <Nov/22 10:04 pm>[1136649.056106] Thread overran stack, or stack corrupted
    7 <Nov/22 10:04 pm>[1136649.061536] Oops: 0000 [#1] SMP
    8 <Nov/22 10:04 pm>[1136649.065226] last sysfs file: /sys/devices/pci0000:00/0000:00:1f.3/i2c-0/0-002f/temp6_alarm
    9 <Nov/22 10:04 pm>[1136649.074221] CPU 2
    10                  <Nov/22 10:04 pm>[1136649.076262] Modules linked in: w83793 hwmon_vid autofs4 ioatdma i2c_i801 i2c_core iTCO_wdt e1000e dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
    11 <Nov/22 10:04 pm>[1136649.110721]
    12 <Nov/22 10:04 pm>[1136649.112603] Pid: 8371, comm: nfsd Not tainted 2.6.34.4 #3 S7010/empty
    13 <Nov/22 10:04 pm>[1136649.119537] RIP: 0010:[<ffffffff8102fc64>]  [<ffffffff8102fc64>] cpuacct_charge+0x5f/0x96
    14 <Nov/22 10:04 pm>[1136649.128447] RSP: 0018:ffff88061c81c110  EFLAGS: 00010086
    15 <Nov/22 10:04 pm>[1136649.134235] RAX: 00000000000169f0 RBX: 000000001d1e6120 RCX: 000000001d1e6120
    16 <Nov/22 10:04 pm>[1136649.142065] RDX: ffff880624ca4600 RSI: 000000000000001f RDI: 0000000000000000
    17 <Nov/22 10:04 pm>[1136649.149895] RBP: ffff88061c81c130 R08: 0000000000000006 R09: 0000000000000000
    18 <Nov/22 10:04 pm>[1136649.157736] R10: ffffffff81035927 R11: ffffffff81a9daa0 R12: ffff88061d1e6120
    19 <Nov/22 10:04 pm>[1136649.165594] R13: 00000000000006b5 R14: 0001753e6cc49a37 R15: ffff880002653c00
    20 <Nov/22 10:04 pm>[1136649.173399] FS:  0000000000000000(0000) GS:ffff880002640000(0000) knlGS:0000000000000000
    21 <Nov/22 10:04 pm>[1136649.182186] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
    22 <Nov/22 10:04 pm>[1136649.188403] CR2: 000000006aa54080 CR3: 00000003eb84a000 CR4: 00000000000006e0
    23 <Nov/22 10:04 pm>[1136649.196239] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    24 
    25 <Nov/22 10:04 pm>[1136649.204011] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
    26 <Nov/22 10:04 pm>[1136649.211833] Process nfsd (pid: 8371, threadinfo ffff88061c81c000, task ffff88061d1e6120)
    27 <Nov/22 10:04 pm>[1136649.220600] Stack:
    28 <Nov/22 10:04 pm>[1136649.223002]  ffffffff8102fc21 ffff88061d1e6120 ffff88061d1e6158 00000000000006b5
    29 <Nov/22 10:04 pm>[1136649.230789] <0> ffff88061c81c160 ffffffff8102fe1e ffff880623503090 ffff880002653c88
    30 <Nov/22 10:04 pm>[1136649.239241] <0> 0000000000000000 ffff880002653c00 ffff88061c81c1b0 ffffffff8103160d
    31 <Nov/22 10:04 pm>[1136649.248030] Call Trace:
    32 <Nov/22 10:04 pm>[1136649.250892]  [<ffffffff8102fc21>] ? cpuacct_charge+0x1c/0x96
    33 <Nov/22 10:04 pm>[1136649.257022]  [<ffffffff8102fe1e>] update_curr+0x100/0x140
    34 <Nov/22 10:04 pm>[1136649.262910]  [<ffffffff8103160d>] enqueue_task_fair+0x98/0x332
    35 <Nov/22 10:04 pm>[1136649.269220]  [<ffffffff8102b548>] enqueue_task+0x5e/0x69
    36 <Nov/22 10:04 pm>[1136649.274977]  [<ffffffff8102b659>] activate_task+0x2a/0x32
    37 <Nov/22 10:04 pm>[1136649.280846]  [<ffffffff81035666>] pull_task+0x35/0x55
    38 <Nov/22 10:04 pm>[1136649.286343]  [<ffffffff81037767>] load_balance+0xcab/0xf63
    39 <Nov/22 10:04 pm>[1136649.292327]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
    40 <Nov/22 10:04 pm>[1136649.298676]  [<ffffffff8151cc0e>] schedule+0x25a/0x86a
    41 <Nov/22 10:04 pm>[1136649.304290]  [<ffffffff8151f61f>] ? _raw_spin_unlock_irq+0x26/0x2c
    42 <Nov/22 10:04 pm>[1136649.310949]  [<ffffffff8151d35f>] schedule_timeout+0x28/0x24c
    43 <Nov/22 10:04 pm>[1136649.317155]  [<ffffffff8151e73d>] ? __down+0x60/0x95
    44 <Nov/22 10:04 pm>[1136649.322563]  [<ffffffff8151e745>] __down+0x68/0x95
    45 <Nov/22 10:04 pm>[1136649.327808]  [<ffffffff81058e2d>] ? down+0xf/0x3a
    46 <Nov/22 10:04 pm>[1136649.332980]  [<ffffffff812810b4>] ? _xfs_buf_find+0x1bb/0x273
    47 
    48 <Nov/22 10:04 pm>[1136649.339197]  [<ffffffff81058e46>] down+0x28/0x3a
    49 <Nov/22 10:04 pm>[1136649.344265]  [<ffffffff8127f8af>] xfs_buf_lock+0x79/0xad
    50 <Nov/22 10:04 pm>[1136649.350034]  [<ffffffff812810b4>] _xfs_buf_find+0x1bb/0x273
    51 <Nov/22 10:04 pm>[1136649.356068]  [<ffffffff812811cc>] xfs_buf_get+0x60/0x17e
    52 <Nov/22 10:04 pm>[1136649.361852]  [<ffffffff81277bc9>] xfs_trans_get_buf+0xe7/0x13e
    53 <Nov/22 10:04 pm>[1136649.368193]  [<ffffffff8124d7b6>] xfs_btree_get_buf_block+0x44/0x60
    54 <Nov/22 10:04 pm>[1136649.374935]  [<ffffffff8124ed73>] xfs_btree_split+0x1ef/0x5d0
    55 <Nov/22 10:04 pm>[1136649.381158]  [<ffffffff8124e223>] ? xfs_btree_lshift+0x69/0x545
    56 <Nov/22 10:04 pm>[1136649.387563]  [<ffffffff8128121a>] ? xfs_buf_get+0xae/0x17e
    57 <Nov/22 10:04 pm>[1136649.393508]  [<ffffffff8125001d>] xfs_btree_make_block_unfull+0x10f/0x14c
    58 <Nov/22 10:04 pm>[1136649.400774]  [<ffffffff81250222>] xfs_btree_insrec+0x1c8/0x4fc
    59 <Nov/22 10:04 pm>[1136649.407107]  [<ffffffff8124ca12>] ? xfs_btree_rec_addr+0xd/0x14
    60 <Nov/22 10:04 pm>[1136649.413497]  [<ffffffff8124f8e8>] ? xfs_btree_lookup+0x256/0x3dc
    61                  <Nov/22 10:04 pm>[1136649.419986]  [<ffffffff812505b4>] xfs_btree_insert+0x5e/0x14a
    62 <Nov/22 10:04 pm>[1136649.426228]  [<ffffffff8123acb4>] xfs_free_ag_extent+0x4a9/0x6a6
    63 <Nov/22 10:04 pm>[1136649.432747]  [<ffffffff8123c5bc>] xfs_alloc_fix_freelist+0x24c/0x3f2
    64 <Nov/22 10:04 pm>[1136649.439594]  [<ffffffff812734e7>] ? xfs_perag_get+0x59/0x99
    65 <Nov/22 10:04 pm>[1136649.445644]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
    66 <Nov/22 10:04 pm>[1136649.451954]  [<ffffffff812734b7>] ? xfs_perag_get+0x29/0x99
    67 <Nov/22 10:04 pm>[1136649.457988]  [<ffffffff8123cad0>] xfs_alloc_vextent+0x2cd/0x4ad
    68 <Nov/22 10:04 pm>[1136649.464369]  [<ffffffff812488c1>] xfs_bmap_btalloc+0x3c2/0x576
    69 <Nov/22 10:04 pm>[1136649.470704]  [<ffffffff81248a98>] xfs_bmap_alloc+0x23/0x25
    70 
    71 <Nov/22 10:04 pm>[1136649.476659]  [<ffffffff8124add5>] xfs_bmapi+0x7b5/0xe73
    72 <Nov/22 10:04 pm>[1136649.482329]  [<ffffffff8126cbeb>] ? xlog_grant_log_space+0x3e0/0x484
    73 <Nov/22 10:04 pm>[1136649.489144]  [<ffffffff8105861d>] ? down_write_nested+0x2a/0x4a
    74 <Nov/22 10:04 pm>[1136649.495556]  [<ffffffff81267363>] xfs_iomap_write_allocate+0x227/0x353
    75 <Nov/22 10:04 pm>[1136649.502553]  [<ffffffff81267eac>] xfs_iomap+0x318/0x39c
    76 <Nov/22 10:04 pm>[1136649.508219]  [<ffffffff8127d1e5>] xfs_map_blocks+0x27/0x2b
    77 <Nov/22 10:04 pm>[1136649.514166]  [<ffffffff8127e160>] xfs_page_state_convert+0x2d8/0x5b2
    78 <Nov/22 10:04 pm>[1136649.521007]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
    79 <Nov/22 10:04 pm>[1136649.527334]  [<ffffffff8127e71f>] xfs_vm_writepage+0xd9/0x113
    80 <Nov/22 10:04 pm>[1136649.533567]  [<ffffffff810b7612>] shrink_page_list+0x3c2/0x68f
    81 <Nov/22 10:04 pm>[1136649.539867]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
    82 <Nov/22 10:04 pm>[1136649.546197]  [<ffffffff810e5fb2>] ? mem_cgroup_del_lru_list+0x7b/0x8e
    83 <Nov/22 10:04 pm>[1136649.553149]  [<ffffffff810b8341>] shrink_zone+0x7c2/0xb86
    84 <Nov/22 10:04 pm>[1136649.558997]  [<ffffffff811069bf>] ? wakeup_flusher_threads+0x33/0xc6
    85 <Nov/22 10:04 pm>[1136649.565812]  [<ffffffff810b97c3>] do_try_to_free_pages+0x208/0x37a
    86 <Nov/22 10:04 pm>[1136649.572466]  [<ffffffff810b9b05>] try_to_free_pages+0x71/0x78
    87 <Nov/22 10:04 pm>[1136649.578699]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
    88 <Nov/22 10:04 pm>[1136649.585358]  [<ffffffff810b3026>] __alloc_pages_nodemask+0x407/0x671
    89 <Nov/22 10:04 pm>[1136649.592195]  [<ffffffff810da7be>] alloc_pages_current+0x96/0x9f
    90 <Nov/22 10:04 pm>[1136649.598573]  [<ffffffff810acb60>] __page_cache_alloc+0x59/0x5e
    91 <Nov/22 10:04 pm>[1136649.604899]  [<ffffffff810b4860>] __do_page_cache_readahead+0xe5/0x1fc
    92 <Nov/22 10:04 pm>[1136649.611888]  [<ffffffff810b47f6>] ? __do_page_cache_readahead+0x7b/0x1fc
    93 
    94 <Nov/22 10:04 pm>[1136649.619123]  [<ffffffff810b4993>] ra_submit+0x1c/0x20
    95 <Nov/22 10:04 pm>[1136649.624616]  [<ffffffff810b4cb7>] ondemand_readahead+0x243/0x256
    96 <Nov/22 10:04 pm>[1136649.631111]  [<ffffffff810b4ba0>] ? ondemand_readahead+0x12c/0x256
    97 <Nov/22 10:04 pm>[1136649.637810]  [<ffffffff810b4daa>] page_cache_sync_readahead+0x38/0x3a
    98 <Nov/22 10:04 pm>[1136649.644751]  [<ffffffff81109bf2>] __generic_file_splice_read+0xfa/0x41c
    99 <Nov/22 10:04 pm>[1136649.656339]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
    100 <Nov/22 10:04 pm>[1136649.662670]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
    101 <Nov/22 10:04 pm>[1136649.668972]  [<ffffffff8106433a>] ? __lock_acquire+0x995/0x9fc
    102 <Nov/22 10:04 pm>[1136649.675318]  [<ffffffff812615b1>] ? xfs_ilock+0x49/0xb2
    103 <Nov/22 10:04 pm>[1136649.681004]  [<ffffffff811086db>] ? spd_release_page+0x0/0x14
    104                  <Nov/22 10:04 pm>[1136649.687237]  [<ffffffff81109f57>] generic_file_splice_read+0x43/0x74
    105 <Nov/22 10:04 pm>[1136649.694060]  [<ffffffff8128191a>] xfs_file_splice_read+0xde/0x125
    106 <Nov/22 10:04 pm>[1136649.700639]  [<ffffffff8110875d>] vfs_splice_to+0x6e/0x81
    107 <Nov/22 10:04 pm>[1136649.706522]  [<ffffffff81108837>] splice_direct_to_actor+0xc7/0x19c
    108 <Nov/22 10:04 pm>[1136649.713286]  [<ffffffff811f562c>] ? nfsd_direct_splice_actor+0x0/0x12
    109 <Nov/22 10:04 pm>[1136649.720209]  [<ffffffff811f54ef>] nfsd_vfs_read+0x263/0x3a0
    110 <Nov/22 10:04 pm>[1136649.726284]  [<ffffffff81281760>] ? xfs_file_open+0x0/0x3f
    111 <Nov/22 10:04 pm>[1136649.732248]  [<ffffffff811f5c4e>] nfsd_read+0xa7/0xc3
    112 <Nov/22 10:04 pm>[1136649.737761]  [<ffffffff811fcd79>] nfsd3_proc_read+0xfe/0x142
    113 <Nov/22 10:04 pm>[1136649.743892]  [<ffffffff811f0854>] nfsd_dispatch+0xe9/0x1ca
    114 <Nov/22 10:04 pm>[1136649.749856]  [<ffffffff814f04c5>] svc_process+0x425/0x72a
    115 <Nov/22 10:04 pm>[1136649.755735]  [<ffffffff811f0ccf>] ? nfsd+0x0/0x139
    116 
    117 <Nov/22 10:04 pm>[1136649.760988]  [<ffffffff811f0dc2>] nfsd+0xf3/0x139
    118 <Nov/22 10:04 pm>[1136649.766158]  [<ffffffff810546e2>] kthread+0x7d/0x85
    119 <Nov/22 10:04 pm>[1136649.771492]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
    120 <Nov/22 10:04 pm>[1136649.777882]  [<ffffffff81054665>] ? kthread+0x0/0x85
    121 <Nov/22 10:04 pm>[1136649.783298]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
    122 <Nov/22 10:04 pm>[1136649.789871] Code: c7 a0 da a9 81 48 63 58 18 48 c7 04 24 21 fc 02 81 e8 52 47 03 00 49 8b 84 24 a0 0e 00 00 48 89 d9 48 8b 50 48 eb 16 48 8b 42 20 <48> 03 04 cd 80 37 b2 81 4c 01 28 48 8b 92 c8 00 00 00 48 85 d2
    123 <Nov/22 10:04 pm>[1136649.811185] RIP  [<ffffffff8102fc64>] cpuacct_charge+0x5f/0x96
    124 <Nov/22 10:04 pm>[1136649.817510]  RSP <ffff88061c81c110>
    125 <Nov/22 10:04 pm>[1136649.821450] CR2: 000000006aa54080
    126 <Nov/22 10:04 pm>[1136649.825456] ---[ end trace 43e8e7ba86136bef ]---
    127 <Nov/22 10:04 pm>[1136649.830562] Kernel panic - not syncing: Fatal exception in interrupt
    128 <Nov/22 10:04 pm>[1136649.837415] Pid: 8371, comm: nfsd Tainted: G      D    2.6.34.4 #3
    129 <Nov/22 10:04 pm>[1136649.844096] Call Trace:
    130 <Nov/22 10:05 pm>[1<1036>[711183.86751618.684]5 61B8UG7]:  sBUpGin:l socpkin lloocckku lpo ocnku pCP Uon#5 C, PUn#fs1d, /8ks3w18a,pd f0f/6ff088,8 00ff0f26f58830c00002
    131 <Nov/22 10:05 pm>653c00
    132 <Nov/22 10:05 pm>[113[61711836.785168.189700] 2P75i]d: P 6i0d:8 , 83c1om8,m : cokmswm:a pdnf0s Tda Tinaitentd:e d:G   G        D   D      2. 62..364.3.44<Nov/22 10:05 pm>#3
    133 
    134 <Nov/22 10:05 pm>[113[61711836.781568.189183] 0C94al]l C aTrlla ceT:ra
    135 <Nov/22 10:05 pm>ce:
    136 <Nov/22 10:05 pm>[1136[711138.68718880.8175]61 9 5]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
    137 <Nov/22 10:05 pm>[1136718.856199]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
    138 <Nov/22 10:05 pm>[1136718.856201]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
    139 
    140 <Nov/22 10:05 pm>[1136718.856203]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
    141 <Nov/22 10:05 pm>[1136718.856206]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
    142 <Nov/22 10:05 pm>[1136718.856208]  [<ffffffff810368b8>] wake_up_process+0x10/0x12
    143                  <Nov/22 10:05 pm>[1136718.856211]  [<ffffffff8127e964>] xfsbufd_wakeup+0x49/0x76
    144 <Nov/22 10:05 pm>[1136718.856213]  [<ffffffff810b8876>] shrink_slab+0x69/0x148
    145                  <Nov/22 10:05 pm>[1136718.856216] BUG: spinlock lockup on CPU#0, swapper/0, ffff880002653c00
    146 <Nov/22 10:05 pm>[1136718.856220]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
    147 <Nov/22 10:05 pm>[1136718.856222] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3
    148 <Nov/22 10:05 pm>[1136718.856223] Call Trace:
    149                  <Nov/22 10:05 pm>[1136718.856225]  [<ffffffff810b9340>] kswapd+0x640/0x8bb
    150 <Nov/22 10:05 pm>[1136718.856226]  <IRQ>  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
    151 <Nov/22 10:05 pm>[1136718.856231]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
    152 <Nov/22 10:05 pm>[1136718.856233]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
    153 <Nov/22 10:05 pm>[1136718.856237]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
    154 <Nov/22 10:05 pm>[1136718.856239]  [<ffffffff810b8d00>] ? kswapd+0x0/0x8bb
    155 <Nov/22 10:05 pm>[1136718.856241]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
    156 <Nov/22 10:05 pm>[1136718.856243]  [<ffffffff810546e2>] kthread+0x7d/0x85
    157 <Nov/22 10:05 pm>[1136718.856245]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
    158 <Nov/22 10:05 pm>[1136718.856248]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
    159 <Nov/22 10:05 pm>[1136718.856250]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
    160 <Nov/22 10:05 pm>[1136718.856252]  [<ffffffff81036899>] default_wake_function+0xd/0xf
    161 <Nov/22 10:05 pm>[1136718.856255]  [<ffffffff81054665>] ? kthread+0x0/0x85
    162 
    163 <Nov/22 10:05 pm>[1136718.856257]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
    164 <Nov/22 10:05 pm>[1136718.856260]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
    165 <Nov/22 10:05 pm>[1136718.856261]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
    166 <Nov/22 10:05 pm>[1136718.856262] sending NMI to all CPUs:
    167 <Nov/22 10:05 pm>[1136718.856266] NMI backtrace for cpu 0
    168 <Nov/22 10:05 pm>[1136718.890254] BUG: spinlock lockup on CPU#8, kswapd1/609, ffff880002633c00
    169 <Nov/22 10:05 pm>[1136718.890257] Pid: 609, comm: kswapd1 Tainted: G      D    2.6.34.4 #3
    170 <Nov/22 10:05 pm>[1136718.890258] Call Trace:
    171 <Nov/22 10:05 pm>[1136718.890260]  <IRQ>  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
    172 <Nov/22 10:05 pm>[1136718.890269]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
    173                  <Nov/22 10:05 pm>[1136718.890272]  [<ffffffff81032141>] ? sched_rt_period_timer+0xa4/0x1c1
    174 <Nov/22 10:05 pm>[1136718.890275]  [<ffffffff81032141>] sched_rt_period_timer+0xa4/0x1c1
    175 <Nov/22 10:05 pm>[1136718.890277]  [<ffffffff8103209d>] ? sched_rt_period_timer+0x0/0x1c1
    176 <Nov/22 10:05 pm>[1136718.890280]  [<ffffffff81057933>] __run_hrtimer+0xd0/0x12d
    177 <Nov/22 10:05 pm>[1136718.890282]  [<ffffffff81057b81>] hrtimer_interrupt+0xb5/0x19f
    178 <Nov/22 10:05 pm>[1136718.890287]  [<ffffffff810197a5>] smp_apic_timer_interrupt+0x82/0x95
    179 <Nov/22 10:05 pm>[1136718.890293]  [<ffffffff81003353>] apic_timer_interrupt+0x13/0x20
    180 <Nov/22 10:05 pm>[1136718.890294]  <EOI>  [<ffffffff811cae9b>] ? nfs_access_cache_shrinker+0x2c/0x229
    181 <Nov/22 10:05 pm>[1136718.890302]  [<ffffffff812c08b7>] ? delay_tsc+0x2d/0x59
    182 <Nov/22 10:05 pm>[1136718.890304]  [<ffffffff812c08fe>] __delay+0xa/0xc
    183 <Nov/22 10:05 pm>[1136718.890306]  [<ffffffff812c4e65>] do_raw_spin_lock+0xde/0x147
    184 <Nov/22 10:05 pm>[1136718.890308]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
    185 
    186 <Nov/22 10:05 pm>[1136718.890312]  [<ffffffff8127e930>] ? xfsbufd_wakeup+0x15/0x76
    187 <Nov/22 10:05 pm>[1136718.890314]  [<ffffffff8127e930>] xfsbufd_wakeup+0x15/0x76
    188 <Nov/22 10:05 pm>[1136718.890318]  [<ffffffff810b8876>] shrink_slab+0x69/0x148
    189 <Nov/22 10:05 pm>[1136718.890321]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
    190 <Nov/22 10:05 pm>[1136718.890323]  [<ffffffff810b9340>] kswapd+0x640/0x8bb
    191 <Nov/22 10:05 pm>[1136718.890325]  [<ffffffff810b6cc0>] ? isolate_pages_global+0x0/0x23f
    192 <Nov/22 10:05 pm>[1136718.890328]  [<ffffffff81054b6c>] ? autoremove_wake_function+0x0/0x38
    193 <Nov/22 10:05 pm>[1136718.890330]  [<ffffffff810b8d00>] ? kswapd+0x0/0x8bb
    194 <Nov/22 10:05 pm>[1136718.890332]  [<ffffffff810546e2>] kthread+0x7d/0x85
    195 <Nov/22 10:05 pm>[1136718.890335]  [<ffffffff81003794>] kernel_thread_helper+0x4/0x10
    196 <Nov/22 10:05 pm>[1136718.890337]  [<ffffffff81054665>] ? kthread+0x0/0x85
    197 <Nov/22 10:05 pm>[1136718.890339]  [<ffffffff81003790>] ? kernel_thread_helper+0x0/0x10
    198 <Nov/22 10:05 pm>[1136718.890340] sending NMI to all CPUs:
    199 <Nov/22 10:05 pm>[1136719.251114] CPU 0
    200 <Nov/22 10:05 pm>[1136719.253220] Modules linked in: w83793 hwmon_vid autofs4 ioatdma i2c_i801 i2c_core iTCO_wdt e1000e dca rtc_cmos tg3 libphy e1000 fuse dm_snapshot dm_mirror dm_region_hash dm_log scsi_wait_scan qla2xxx megaraid_sas megaraid_mbox megaraid_mm aacraid 3w_9xxx 3w_xxxx arcmsr aic7xxx aic79xx sr_mod sg ata_piix ahci sata_sis sata_nv sata_via pata_sl82c105 pata_sis
    201 <Nov/22 10:05 pm>[1136719.288964]
    202 <Nov/22 10:05 pm>[1136719.290864] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3 S7010/empty
    203 <Nov/22 10:05 pm>[1136719.298782] RIP: 0010:[<ffffffff812bfeb3>]  [<ffffffff812bfeb3>] vsnprintf+0xa9/0x41f
    204 <Nov/22 10:05 pm>[1136719.307405] RSP: 0018:ffff880002603500  EFLAGS: 00000093
    205 <Nov/22 10:05 pm>[1136719.313214] RAX: 0000000000000000 RBX: ffff880002603640 RCX: ffffffff8178117e
    206 <Nov/22 10:05 pm>[1136719.321033] RDX: ffffffff81781100 RSI: 0000000000000003 RDI: ffffffff8178117e
    207 <Nov/22 10:05 pm>[1136719.328874] RBP: ffff880002603560 R08: 0000000000000003 R09: ffffffff8178117b
    208 
    209 <Nov/22 10:05 pm>[1136719.336698] R10: ffffffff8103bc99 R11: ffff880002603550 R12: ffffffff81c9ab90
    210 <Nov/22 10:05 pm>[1136719.344548] R13: ffffffff8178117b R14: ffffffff81c9af90 R15: ffffffff81c9ab90
    211 <Nov/22 10:05 pm>[1136719.352403] FS:  0000000000000000(0000) GS:ffff880002600000(0000) knlGS:0000000000000000
    212 <Nov/22 10:05 pm>[1136719.361233] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
    213 <Nov/22 10:05 pm>[1136719.367450] CR2: 00007fcd38ff0228 CR3: 0000000386db3000 CR4: 00000000000006f0
    214 <Nov/22 10:05 pm>[1136719.375286] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
    215 <Nov/22 10:05 pm>[1136719.383151] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
    216 <Nov/22 10:05 pm>[1136719.390993] Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a92020)
    217 <Nov/22 10:05 pm>[1136719.399812] Stack:
    218 <Nov/22 10:05 pm>[1136719.402264]  0000000000000086 ffffffff81a99cd0 0000000000000400 ffffffff8178117e
    219 <Nov/22 10:05 pm>[1136719.410207] <0> 0000000000000000 0000000000000000 ffffffff81a99c58 0000000000000400
    220 <Nov/22 10:05 pm>[1136719.418834] <0> ffff880002603640 ffffffff8178117b 0000000000000004 ffff880002603fc0
    221 <Nov/22 10:05 pm>[1136719.427906] Call Trace:
    222 <Nov/22 10:05 pm>[1136719.430810]  <IRQ>
    223 <Nov/22 10:05 pm>[1136719.433379]  [<ffffffff812c0388>] vscnprintf+0x11/0x29
    224 <Nov/22 10:05 pm>[1136719.439010]  [<ffffffff8103bc99>] ? vprintk+0x152/0x3a0
    225 <Nov/22 10:05 pm>[1136719.444731]  [<ffffffff8103bcea>] vprintk+0x1a3/0x3a0
    226 <Nov/22 10:05 pm>[1136719.450280]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
    227 <Nov/22 10:05 pm>[1136719.456628]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
    228 <Nov/22 10:05 pm>[1136719.463137]  [<ffffffff8151c589>] printk+0x67/0x69
    229 <Nov/22 10:05 pm>[1136719.468406]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
    230 <Nov/22 10:05 pm>[1136719.474926]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
    231 
    232 <Nov/22 10:05 pm>[1136719.481428]  [<ffffffff8151c589>] ? printk+0x67/0x69
    233 <Nov/22 10:05 pm>[1136719.486881]  [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
    234 <Nov/22 10:05 pm>[1136719.493393]  [<ffffffff810060bb>] print_trace_address+0x4e/0x57
    235 <Nov/22 10:05 pm>[1136719.499822]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
    236 <Nov/22 10:05 pm>[1136719.506332]  [<ffffffff810061e9>] print_context_stack+0x62/0xbd
    237 <Nov/22 10:05 pm>[1136719.512790]  [<ffffffff810054e2>] dump_trace+0x23d/0x327
    238 <Nov/22 10:05 pm>[1136719.518614]  [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
    239 <Nov/22 10:05 pm>[1136719.524943]  [<ffffffff8100606b>] show_trace+0x10/0x12
    240 <Nov/22 10:05 pm>[1136719.530586]  [<ffffffff8151c401>] dump_stack+0x72/0x7b
    241 <Nov/22 10:05 pm>[1136719.536206]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
    242 <Nov/22 10:05 pm>[1136719.542525]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
    243 <Nov/22 10:05 pm>[1136719.548514]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
    244 <Nov/22 10:05 pm>[1136719.554497]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
    245 <Nov/22 10:05 pm>[1136719.560313]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
    246 <Nov/22 10:05 pm>[1136719.566400]  [<ffffffff81036899>] default_wake_function+0xd/0xf
    247 <Nov/22 10:05 pm>[1136719.572809]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
    248 <Nov/22 10:05 pm>[1136719.578970]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
    249 <Nov/22 10:05 pm>[1136719.584482]  [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
    250 <Nov/22 10:05 pm>[1136719.590840]  [<ffffffff814f1d64>] svc_udp_data_ready+0x4e/0x7b
    251 <Nov/22 10:05 pm>[1136719.597178]  [<ffffffff8144d67d>] sock_queue_rcv_skb+0x14e/0x179
    252                  <Nov/22 10:05 pm>[1136719.603686]  [<ffffffff81499a17>] __udp_queue_rcv_skb+0x12/0x6f
    253 <Nov/22 10:05 pm>[1136719.610101]  [<ffffffff8149ae9e>] udp_queue_rcv_skb+0x1aa/0x258
    254 
    255 <Nov/22 10:05 pm>[1136719.616531]  [<ffffffff8149b610>] __udp4_lib_rcv+0x31c/0x546
    256 <Nov/22 10:05 pm>[1136719.622705]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
    257 <Nov/22 10:05 pm>[1136719.629760]  [<ffffffff8149b84f>] udp_rcv+0x15/0x17
    258 <Nov/22 10:05 pm>[1136719.635097]  [<ffffffff8147a540>] ip_local_deliver_finish+0x111/0x1d0
    259 <Nov/22 10:05 pm>[1136719.642053]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
    260 <Nov/22 10:05 pm>[1136719.649115]  [<ffffffff8147a671>] ip_local_deliver+0x72/0x7a
    261 <Nov/22 10:05 pm>[1136719.655250]  [<ffffffff8147a170>] ip_rcv_finish+0x334/0x350
    262 <Nov/22 10:05 pm>[1136719.661325]  [<ffffffff8147a409>] ip_rcv+0x27d/0x2a3
    263 <Nov/22 10:05 pm>[1136719.666762]  [<ffffffff814570c8>] netif_receive_skb+0x3f4/0x431
    264 <Nov/22 10:05 pm>[1136719.673205]  [<ffffffff81456e72>] ? netif_receive_skb+0x19e/0x431
    265 <Nov/22 10:05 pm>[1136719.679793]  [<ffffffff8145792d>] napi_skb_finish+0x24/0x3c
    266 <Nov/22 10:05 pm>[1136719.685885]  [<ffffffff8145796f>] napi_gro_receive+0x2a/0x2f
    267 <Nov/22 10:05 pm>[1136719.696526]  [<ffffffffa01c28dd>] e1000_receive_skb+0x5c/0x65 [e1000e]
    268 <Nov/22 10:05 pm>[1136719.703567]  [<ffffffffa01c2b32>] e1000_clean_rx_irq+0x24c/0x301 [e1000e]
    269 <Nov/22 10:05 pm>[1136719.710891]  [<ffffffffa01c405a>] e1000_clean+0x75/0x22e [e1000e]
    270 <Nov/22 10:05 pm>[1136719.717524]  [<ffffffff814573ba>] net_rx_action+0xb2/0x1ff
    271 <Nov/22 10:05 pm>[1136719.723494]  [<ffffffff814574c5>] ? net_rx_action+0x1bd/0x1ff
    272 <Nov/22 10:05 pm>[1136719.729771]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
    273 <Nov/22 10:05 pm>[1136719.735692]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
    274 <Nov/22 10:05 pm>[1136719.741485]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
    275 <Nov/22 10:05 pm>[1136719.747128]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
    276 <Nov/22 10:05 pm>[1136719.752603]  [<ffffffff8100457c>] do_IRQ+0xaa/0xc1
    277 
    278 <Nov/22 10:05 pm>[1136719.757899]  [<ffffffff8151f913>] ret_from_intr+0x0/0xa
    279 <Nov/22 10:05 pm>[1136719.763622]  <EOI>
    280 <Nov/22 10:05 pm>[1136719.766199]  [<ffffffff8126aba0>] ? _xfs_log_force_lsn+0x73/0x29c
    281 <Nov/22 10:05 pm>[1136719.772797]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
    282 <Nov/22 10:05 pm>[1136719.779339]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
    283 <Nov/22 10:05 pm>[1136719.785840]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
    284 <Nov/22 10:05 pm>[1136719.792108]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
    285 <Nov/22 10:05 pm>[1136719.797552]  [<ffffffff815072da>] rest_init+0xbe/0xc2
    286 <Nov/22 10:05 pm>[1136719.803083]  [<ffffffff8150721c>] ? rest_init+0x0/0xc2
    287 <Nov/22 10:05 pm>[1136719.808713]  [<ffffffff81b3fd09>] start_kernel+0x389/0x395
    288 <Nov/22 10:05 pm>[1136719.814730]  [<ffffffff81b3f2a1>] x86_64_start_reservations+0xb1/0xb5
    289 <Nov/22 10:05 pm>[1136719.821708]  [<ffffffff81b3f386>] x86_64_start_kernel+0xe1/0xe8
    290 <Nov/22 10:05 pm>[1136719.828145] Code: 89 ef e8 5d e5 ff ff 4c 63 c0 89 c6 4b 8d 54 05 00 48 89 55 b8 8a 55 c0 80 fa 11 0f 87 1c 02 00 00 0f b6 c2 ff 24 c5 c0 40 64 81 <4d> 39 f4 73 20 4c 89 f0 48 63 ce 44 89 f2 4c 29 e0 44 29 e2 4c
    291 <Nov/22 10:05 pm>[1136719.851413] Call Trace:
    292 <Nov/22 10:05 pm>[1136719.854282]  <IRQ>  [<ffffffff812c0388>] vscnprintf+0x11/0x29
    293 <Nov/22 10:05 pm>[1136719.860606]  [<ffffffff8103bc99>] ? vprintk+0x152/0x3a0
    294 <Nov/22 10:05 pm>[1136719.866344]  [<ffffffff8103bcea>] vprintk+0x1a3/0x3a0
    295 <Nov/22 10:05 pm>[1136719.871919]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
    296 <Nov/22 10:05 pm>[1136719.878249]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
    297 <Nov/22 10:05 pm>[1136719.884768]  [<ffffffff8151c589>] printk+0x67/0x69
    298 <Nov/22 10:05 pm>[1136719.890044]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
    299 <Nov/22 10:05 pm>[1136719.896567]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
    300 
    301 <Nov/22 10:05 pm>[1136719.903120]  [<ffffffff8151c589>] ? printk+0x67/0x69
    302 <Nov/22 10:05 pm>[1136719.908561]  [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
    303 <Nov/22 10:05 pm>[1136719.915058]  [<ffffffff810060bb>] print_trace_address+0x4e/0x57
    304 <Nov/22 10:05 pm>[1136719.921489]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
    305 <Nov/22 10:05 pm>[1136719.927998]  [<ffffffff810061e9>] print_context_stack+0x62/0xbd
    306 <Nov/22 10:05 pm>[1136719.934430]  [<ffffffff810054e2>] dump_trace+0x23d/0x327
    307 <Nov/22 10:05 pm>[1136719.940246]  [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
    308 <Nov/22 10:05 pm>[1136719.946573]  [<ffffffff8100606b>] show_trace+0x10/0x12
    309 <Nov/22 10:05 pm>[1136719.952226]  [<ffffffff8151c401>] dump_stack+0x72/0x7b
    310 <Nov/22 10:05 pm>[1136719.957852]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
    311 <Nov/22 10:05 pm>[1136719.964201]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
    312 <Nov/22 10:05 pm>[1136719.970224]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
    313 <Nov/22 10:05 pm>[1136719.976224]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
    314 <Nov/22 10:05 pm>[1136719.982050]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
    315 <Nov/22 10:05 pm>[1136719.988127]  [<ffffffff81036899>] default_wake_function+0xd/0xf
    316 <Nov/22 10:05 pm>[1136719.994524]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
    317 <Nov/22 10:05 pm>[1136720.000680]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
    318 <Nov/22 10:05 pm>[1136720.006253]  [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
    319 <Nov/22 10:05 pm>[1136720.012583]  [<ffffffff814f1d64>] svc_udp_data_ready+0x4e/0x7b
    320                  <Nov/22 10:05 pm>[1136720.018903]  [<ffffffff8144d67d>] sock_queue_rcv_skb+0x14e/0x179
    321 <Nov/22 10:05 pm>[1136720.025429]  [<ffffffff81499a17>] __udp_queue_rcv_skb+0x12/0x6f
    322 <Nov/22 10:05 pm>[1136720.031869]  [<ffffffff8149ae9e>] udp_queue_rcv_skb+0x1aa/0x258
    323 
    324 <Nov/22 10:05 pm>[1136720.038318]  [<ffffffff8149b610>] __udp4_lib_rcv+0x31c/0x546
    325 <Nov/22 10:05 pm>[1136720.044480]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
    326 <Nov/22 10:05 pm>[1136720.051537]  [<ffffffff8149b84f>] udp_rcv+0x15/0x17
    327 <Nov/22 10:05 pm>[1136720.056881]  [<ffffffff8147a540>] ip_local_deliver_finish+0x111/0x1d0
    328 <Nov/22 10:05 pm>[1136720.063837]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
    329 <Nov/22 10:05 pm>[1136720.070892]  [<ffffffff8147a671>] ip_local_deliver+0x72/0x7a
    330 <Nov/22 10:05 pm>[1136720.077044]  [<ffffffff8147a170>] ip_rcv_finish+0x334/0x350
    331 <Nov/22 10:05 pm>[1136720.083162]  [<ffffffff8147a409>] ip_rcv+0x27d/0x2a3
    332 <Nov/22 10:05 pm>[1136720.088625]  [<ffffffff814570c8>] netif_receive_skb+0x3f4/0x431
    333 <Nov/22 10:05 pm>[1136720.095042]  [<ffffffff81456e72>] ? netif_receive_skb+0x19e/0x431
    334 <Nov/22 10:05 pm>[1136720.101682]  [<ffffffff8145792d>] napi_skb_finish+0x24/0x3c
    335 <Nov/22 10:05 pm>[1136720.107748]  [<ffffffff8145796f>] napi_gro_receive+0x2a/0x2f
    336 <Nov/22 10:05 pm>[1136720.113916]  [<ffffffffa01c28dd>] e1000_receive_skb+0x5c/0x65 [e1000e]
    337 <Nov/22 10:05 pm>[1136720.120998]  [<ffffffffa01c2b32>] e1000_clean_rx_irq+0x24c/0x301 [e1000e]
    338 <Nov/22 10:05 pm>[1136720.128354]  [<ffffffffa01c405a>] e1000_clean+0x75/0x22e [e1000e]
    339 <Nov/22 10:05 pm>[1136720.134971]  [<ffffffff814573ba>] net_rx_action+0xb2/0x1ff
    340 <Nov/22 10:05 pm>[1136720.140960]  [<ffffffff814574c5>] ? net_rx_action+0x1bd/0x1ff
    341 <Nov/22 10:05 pm>[1136720.147221]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
    342 <Nov/22 10:05 pm>[1136720.153139]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
    343 <Nov/22 10:05 pm>[1136720.158931]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
    344 <Nov/22 10:05 pm>[1136720.164532]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
    345 <Nov/22 10:05 pm>[1136720.169985]  [<ffffffff8100457c>] do_IRQ+0xaa/0xc1
    346 
    347 <Nov/22 10:05 pm>[1136720.175275]  [<ffffffff8151f913>] ret_from_intr+0x0/0xa
    348 <Nov/22 10:05 pm>[1136720.180965]  <EOI>  [<ffffffff8126aba0>] ? _xfs_log_force_lsn+0x73/0x29c
    349 <Nov/22 10:05 pm>[1136720.188277]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
    350 <Nov/22 10:05 pm>[1136720.194772]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
    351 <Nov/22 10:05 pm>[1136720.201283]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
    352 <Nov/22 10:05 pm>[1136720.207547]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
    353 <Nov/22 10:05 pm>[1136720.213018]  [<ffffffff815072da>] rest_init+0xbe/0xc2
    354 <Nov/22 10:05 pm>[1136720.218568]  [<ffffffff8150721c>] ? rest_init+0x0/0xc2
    355 <Nov/22 10:05 pm>[1136720.224178]  [<ffffffff81b3fd09>] start_kernel+0x389/0x395
    356 <Nov/22 10:05 pm>[1136720.230153]  [<ffffffff81b3f2a1>] x86_64_start_reservations+0xb1/0xb5
    357 <Nov/22 10:05 pm>[1136720.237133]  [<ffffffff81b3f386>] x86_64_start_kernel+0xe1/0xe8
    358 <Nov/22 10:05 pm>[1136720.243569] Pid: 0, comm: swapper Tainted: G      D    2.6.34.4 #3
    359 <Nov/22 10:05 pm>[1136720.250236] Call Trace:
    360 <Nov/22 10:05 pm>[1136720.253110]  <NMI>  [<ffffffff81009b2a>] ? show_regs+0x26/0x2a
    361 <Nov/22 10:05 pm>[1136720.259511]  [<ffffffff81520db8>] nmi_watchdog_tick+0xc3/0x1a5
    362 <Nov/22 10:05 pm>[1136720.265864]  [<ffffffff81520360>] do_nmi+0xb2/0x259
    363 <Nov/22 10:05 pm>[1136720.271220]  [<ffffffff8151fdfa>] nmi+0x1a/0x20
    364 <Nov/22 10:05 pm>[1136720.276251]  [<ffffffff8103bc99>] ? vprintk+0x152/0x3a0
    365                  <Nov/22 10:05 pm>[1136720.281959]  [<ffffffff812bfeb3>] ? vsnprintf+0xa9/0x41f
    366 <Nov/22 10:05 pm>[1136720.287757]  <<EOE>>  <IRQ>  [<ffffffff812c0388>] vscnprintf+0x11/0x29
    367 <Nov/22 10:05 pm>[1136720.294937]  [<ffffffff8103bc99>] ? vprintk+0x152/0x3a0
    368 <Nov/22 10:05 pm>[1136720.300663]  [<ffffffff8103bcea>] vprintk+0x1a3/0x3a0
    369 
    370 <Nov/22 10:05 pm>[1136720.306198]  [<ffffffff8151f67a>] ? _raw_spin_unlock+0x26/0x2b
    371 <Nov/22 10:05 pm>[1136720.312587]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
    372 <Nov/22 10:05 pm>[1136720.319106]  [<ffffffff8151c589>] printk+0x67/0x69
    373 <Nov/22 10:05 pm>[1136720.324391]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
    374 <Nov/22 10:05 pm>[1136720.330913]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
    375                  <Nov/22 10:05 pm>[1136720.337431]  [<ffffffff8151c589>] ? printk+0x67/0x69
    376 <Nov/22 10:05 pm>[1136720.342892]  [<ffffffff81019eb4>] ? touch_nmi_watchdog+0x5b/0x60
    377 <Nov/22 10:05 pm>[1136720.349413]  [<ffffffff810060bb>] print_trace_address+0x4e/0x57
    378 <Nov/22 10:05 pm>[1136720.355845]  [<ffffffff814fbb75>] ? svc_xprt_enqueue+0x1e2/0x23d
    379 <Nov/22 10:05 pm>[1136720.362361]  [<ffffffff810061e9>] print_context_stack+0x62/0xbd
    380 <Nov/22 10:05 pm>[1136720.368802]  [<ffffffff810054e2>] dump_trace+0x23d/0x327
    381 <Nov/22 10:05 pm>[1136720.374626]  [<ffffffff8100604f>] show_trace_log_lvl+0x4c/0x58
    382 <Nov/22 10:05 pm>[1136720.380965]  [<ffffffff8100606b>] show_trace+0x10/0x12
    383 <Nov/22 10:05 pm>[1136720.386589]  [<ffffffff8151c401>] dump_stack+0x72/0x7b
    384 <Nov/22 10:05 pm>[1136720.392208]  [<ffffffff812c4e9f>] do_raw_spin_lock+0x118/0x147
    385 <Nov/22 10:05 pm>[1136720.398522]  [<ffffffff8151f193>] _raw_spin_lock+0x34/0x39
    386 <Nov/22 10:05 pm>[1136720.404502]  [<ffffffff8102fb6b>] ? task_rq_lock+0x7e/0x9a
    387 <Nov/22 10:05 pm>[1136720.410492]  [<ffffffff8102fb6b>] task_rq_lock+0x7e/0x9a
    388 <Nov/22 10:05 pm>[1136720.416292]  [<ffffffff81036578>] try_to_wake_up+0x2e/0x342
    389 <Nov/22 10:05 pm>[1136720.422379]  [<ffffffff81036899>] default_wake_function+0xd/0xf
    390 <Nov/22 10:05 pm>[1136720.428811]  [<ffffffff8102c08d>] __wake_up_common+0x46/0x76
    391 <Nov/22 10:05 pm>[1136720.435000]  [<ffffffff8102faa6>] __wake_up+0x38/0x4f
    392 
    393 <Nov/22 10:05 pm>[1136720.440566]  [<ffffffff814fbb75>] svc_xprt_enqueue+0x1e2/0x23d
    394 <Nov/22 10:05 pm>[1136720.446912]  [<ffffffff814f1d64>] svc_udp_data_ready+0x4e/0x7b
    395 <Nov/22 10:05 pm>[1136720.453258]  [<ffffffff8144d67d>] sock_queue_rcv_skb+0x14e/0x179
    396 <Nov/22 10:05 pm>[1136720.459795]  [<ffffffff81499a17>] __udp_queue_rcv_skb+0x12/0x6f
    397 <Nov/22 10:05 pm>[1136720.466223]  [<ffffffff8149ae9e>] udp_queue_rcv_skb+0x1aa/0x258
    398 <Nov/22 10:05 pm>[1136720.472654]  [<ffffffff8149b610>] __udp4_lib_rcv+0x31c/0x546
    399 <Nov/22 10:05 pm>[1136720.478809]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
    400 <Nov/22 10:05 pm>[1136720.485840]  [<ffffffff8149b84f>] udp_rcv+0x15/0x17
    401 <Nov/22 10:05 pm>[1136720.491236]  [<ffffffff8147a540>] ip_local_deliver_finish+0x111/0x1d0
    402 <Nov/22 10:05 pm>[1136720.498193]  [<ffffffff8147a468>] ? ip_local_deliver_finish+0x39/0x1d0
    403 <Nov/22 10:05 pm>[1136720.505283]  [<ffffffff8147a671>] ip_local_deliver+0x72/0x7a
    404 <Nov/22 10:05 pm>[1136720.511442]  [<ffffffff8147a170>] ip_rcv_finish+0x334/0x350
    405 <Nov/22 10:05 pm>[1136720.517546]  [<ffffffff8147a409>] ip_rcv+0x27d/0x2a3
    406 <Nov/22 10:05 pm>[1136720.523016]  [<ffffffff814570c8>] netif_receive_skb+0x3f4/0x431
    407 <Nov/22 10:05 pm>[1136720.529449]  [<ffffffff81456e72>] ? netif_receive_skb+0x19e/0x431
    408 <Nov/22 10:05 pm>[1136720.536072]  [<ffffffff8145792d>] napi_skb_finish+0x24/0x3c
    409 <Nov/22 10:05 pm>[1136720.542155]  [<ffffffff8145796f>] napi_gro_receive+0x2a/0x2f
    410 <Nov/22 10:05 pm>[1136720.548304]  [<ffffffffa01c28dd>] e1000_receive_skb+0x5c/0x65 [e1000e]
    411 <Nov/22 10:05 pm>[1136720.555369]  [<ffffffffa01c2b32>] e1000_clean_rx_irq+0x24c/0x301 [e1000e]
    412 <Nov/22 10:05 pm>[1136720.562695]  [<ffffffffa01c405a>] e1000_clean+0x75/0x22e [e1000e]
    413 <Nov/22 10:05 pm>[1136720.569302]  [<ffffffff814573ba>] net_rx_action+0xb2/0x1ff
    414 <Nov/22 10:05 pm>[1136720.575298]  [<ffffffff814574c5>] ? net_rx_action+0x1bd/0x1ff
    415 
    416 <Nov/22 10:05 pm>[1136720.581566]  [<ffffffff8104103c>] __do_softirq+0xd8/0x194
    417 <Nov/22 10:05 pm>[1136720.587441]  [<ffffffff8100388c>] call_softirq+0x1c/0x28
    418 <Nov/22 10:05 pm>[1136720.593243]  [<ffffffff81004e7f>] do_softirq+0x33/0x68
    419 <Nov/22 10:05 pm>[1136720.598844]  [<ffffffff81040ec2>] irq_exit+0x36/0x75
    420 <Nov/22 10:05 pm>[1136720.604300]  [<ffffffff8100457c>] do_IRQ+0xaa/0xc1
    421 <Nov/22 10:05 pm>[1136720.609570]  [<ffffffff8151f913>] ret_from_intr+0x0/0xa
    422 <Nov/22 10:05 pm>[1136720.615302]  <EOI>  [<ffffffff8126aba0>] ? _xfs_log_force_lsn+0x73/0x29c
    423 <Nov/22 10:05 pm>[1136720.622589]  [<ffffffff81303870>] ? acpi_idle_enter_c1+0xdd/0xf8
    424                  <Nov/22 10:05 pm>[1136720.629110]  [<ffffffff81303851>] ? acpi_idle_enter_c1+0xbe/0xf8
    425 <Nov/22 10:05 pm>[1136720.635639]  [<ffffffff81430151>] cpuidle_idle_call+0x95/0xf0
    426 <Nov/22 10:05 pm>[1136720.641931]  [<ffffffff81001d4b>] cpu_idle+0x55/0x8d
    427 <Nov/22 10:05 pm>[1136720.647389]  [<ffffffff815072da>] rest_init+0xbe/0xc2
    428 <Nov/22 10:05 pm>[1136720.652906]  [<ffffffff8150721c>] ? rest_init+0x0/0xc2
    429 <Nov/22 10:05 pm>[1136720.658534]  [<ffffffff81b3fd09>] start_kernel+0x389/0x395
    430 <Nov/22 10:05 pm>[1136720.664534]  [<ffffffff81b3f2a1>] x86_64_start_reservations+0xb1/0xb5
    431 <Nov/22 10:05 pm>[1136720.671513]  [<ffffffff81b3f386>] x86_64_start_kernel+0xe1/0xe8
    432 <Nov/22 10:05 pm>[1136720.677924] NMI backtrace for cpu 8
    433 }}}
     1 * 20111122T2200 (about) ipp066 went down [wiki:ipp066-20111122-crash.log]
     2 * 20120802T1035 ipp066 crashed [wiki:ipp066-20120802-crash.log