IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Initial Version and Version 1 of Production_Cluster\ipp014panic


Ignore:
Timestamp:
Feb 20, 2009, 10:32:56 AM (17 years ago)
Author:
trac
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Production_Cluster\ipp014panic

    v1 v1  
     1<pre>
     2The following was on the console. The input was unresponsive
     3
     4<Oct/15 04:15 pm>This is ipp014.unknown_domain (Linux x86_64 2.6.27-rc8-00016-gd3a47e8) 17:36:58
     5
     6<Oct/15 05:38 pm>ipp014 login: [823902.827952] BUG: spinlock wrong owner on CPU#2, ppImage/27000
     7<Oct/15 05:38 pm>[823902.828447]  lock: ffffffff80f425b8, .magic: dead4ead, .owner: ppImage/27000, .owner_cpu: 2
     8<Oct/15 05:38 pm>[823902.828447] Pid: 27000, comm: ppImage Not tainted 2.6.27-rc8-00016-gd3a47e8 #1
     9<Oct/15 05:38 pm>[823902.828447]
     10<Oct/15 05:38 pm>[823902.828447] Call Trace:
     11<Oct/15 05:38 pm>[823902.828447]  [<ffffffff803bdf0e>] _raw_spin_unlock+0x51/0x80
     12<Oct/15 05:38 pm>[823902.828447]  [<ffffffff805c82f8>] _spin_unlock_irqrestore+0x27/0x2e
     13<Oct/15 05:38 pm>[823902.828447]  [<ffffffff802a27af>] __mem_cgroup_uncharge_common+0xcb/0x107
     14<Oct/15 05:38 pm>[823902.828447]  [<ffffffff8028d642>] page_remove_rmap+0x108/0x122
     15<Oct/15 05:38 pm>[823902.828447]  [<ffffffff80285e23>] unmap_vmas+0x47f/0x7ec
     16<Oct/15 05:38 pm>[823902.828447]  [<ffffffff80289afb>] unmap_region+0xb3/0x126
     17<Oct/15 05:38 pm>[823902.828447]  [<ffffffff8028a81b>] do_munmap+0x1f3/0x270
     18<Oct/15 05:38 pm>[823902.828447]  [<ffffffff805c7ae6>] __down_write_nested+0x34/0x9e
     19
     20<Oct/15 05:38 pm>[823902.828447]  [<ffffffff8028b715>] sys_munmap+0x40/0x5a
     21<Oct/15 05:38 pm>[823902.828447]  [<ffffffff8020be2b>] system_call_fastpath+0x16/0x1b
     22<Oct/15 05:38 pm>[823902.828447]
     23<Oct/15 05:47 pm>[824473.019627] BUG: unable to handle kernel NULL pointer dereference at 0000000000000158
     24<Oct/15 05:47 pm>[824473.021724] IP: [<ffffffff8022b04d>] pick_next_task_fair+0x7d/0xa9
     25<Oct/15 05:47 pm>[824473.021724] PGD 21fc8b067 PUD 22e41c067 PMD 0
     26<Oct/15 05:47 pm>[824473.021724] Oops: 0000 [1] SMP
     27<Oct/15 05:47 pm>[824473.021724] CPU 2
     28<Oct/15 05:47 pm>[824473.021724] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt k8temp forcedeth i2c_nforce2 i2c_core tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     29<Oct/15 05:47 pm>[824473.021724] Pid: 27703, comm: ppImage Not tainted 2.6.27-rc8-00016-gd3a47e8 #1
     30<Oct/15 05:47 pm>[824473.021724] RIP: 0010:[<ffffffff8022b04d>]  [<ffffffff8022b04d>] pick_next_task_fair+0x7d/0xa9
     31<Oct/15 05:47 pm>[824473.021724] RSP: 0018:ffff880223d69d18  EFLAGS: 00010046
     32<Oct/15 05:47 pm>[824473.021724] RAX: 000001e2b8508e7b RBX: ffff88021fc79e40 RCX: 0000000000000000
     33<Oct/15 05:47 pm>[824473.021724] RDX: ffff88013624a100 RSI: ffff88013624a1c8 RDI: ffff880223c09010
     34<Oct/15 05:47 pm>[824473.021724] RBP: ffff880223d69d48 R08: 0000000000000000 R09: ffff88013624a1c8
     35<Oct/15 05:47 pm>[824473.021724] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
     36<Oct/15 05:47 pm>[824473.021724] R13: 0002edda956529cd R14: ffff88013624a100 R15: 0000000000000000
     37<Oct/15 05:47 pm>[824473.021724] FS:  00007f83fbab56f0(0000) GS:ffff88022fa0d300(0000) knlGS:0000000000000000
     38<Oct/15 05:47 pm>[824473.021724] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
     39<Oct/15 05:47 pm>[824473.021724] CR2: 0000000000000158 CR3: 000000021e4ff000 CR4: 00000000000006e0
     40<Oct/15 05:47 pm>[824473.021724] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     41<Oct/15 05:47 pm>[824473.021724] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7<Oct/15 05:47 pm>0
     42
     43<Oct/15 05:47 pm>[824473.021724] Process ppImage (pid: 27703, threadinfo ffff880223d68000, task ffff8800a6bb8f10)
     44<Oct/15 05:47 pm>[824473.021724] Stack:  ffff88013624a100 ffff88013624a100 ffffffff805dc840 ffff880223d69d88
     45<Oct/15 05:47 pm>[824473.021724]  0000000000000000 000000010ebcc2ea ffff880223d69dd8 ffffffff805c5f2a
     46<Oct/15 05:47 pm>[824473.021724]  0000000000000000 ffffffff805567ed ffff8800a6bb8f10 0000000000000000
     47<Oct/15 05:47 pm>[824473.021724] Call Trace:
     48<Oct/15 05:47 pm>[824473.021724]  [<ffffffff805c5f2a>] schedule+0x434/0x7e1
     49<Oct/15 05:47 pm>[824473.021724]  [<ffffffff805567ed>] ? tcp_connect+0x319/0x322
     50<Oct/15 05:47 pm>[824473.021724]  [<ffffffff8055974b>] ? tcp_v4_connect+0x3c1/0x422
     51<Oct/15 05:47 pm>[824473.021724]  [<ffffffff805c64ab>] schedule_timeout+0x1e/0xad
     52<Oct/15 05:47 pm>[824473.021724]  [<ffffffff805201cd>] ? release_sock+0x2f/0xd5
     53<Oct/15 05:47 pm>[824473.021724]  [<ffffffff80564a0f>] ? inet_stream_connect+0x137/0x241
     54<Oct/15 05:47 pm>[824473.021724]  [<ffffffff80247b2a>] ? autoremove_wake_function+0x0/0x2e
     55<Oct/15 05:47 pm>[824473.021724]  [<ffffffff8051eb3b>] ? sys_connect+0x6c/0x9c
     56<Oct/15 05:47 pm>[824473.021724]  [<ffffffff802a6290>] ? fget+0xb6/0xbe
     57<Oct/15 05:47 pm>[824473.021724]  [<ffffffff805c7c2a>] ? lockdep_sys_exit_thunk+0x35/0x67
     58<Oct/15 05:47 pm>[824473.021724]  [<ffffffff8039c9ac>] ? cap_file_fcntl+0x0/0x3
     59<Oct/15 05:47 pm>[824473.021724]  [<ffffffff802a61da>] ? fget+0x0/0xbe
     60<Oct/15 05:47 pm>[824473.021724]  [<ffffffff8020be2b>] ? system_call_fastpath+0x16/0x1b
     61<Oct/15 05:47 pm>[824473.021724]
     62<Oct/15 05:47 pm>[824473.021724]
     63<Oct/15 05:47 pm>[824473.021724] Code: ec e8 52 f5 ff ff 49 39 c4 76 11 49 8b 86 30 08 00 00 4d 8d 67 f0 48 89 43 28 eb 04 4c 8b 63 60 4c 89 e6 48 89 df e8 6a fa ff ff <49> 8b 9c 24 58 01 00 00 48 85 db 75 9f 48 8b 7d d0 49 83 ec 38
     64<Oct/15 05:47 pm>[824473.021724] RIP  [<ffffffff8022b04d>] pick_next_task_fair+0x7d/0xa9
     65
     66<Oct/15 05:47 pm>[824473.021724]  RSP <ffff880223d69d18>
     67<Oct/15 05:47 pm>[824473.021724] CR2: 0000000000000158
     68<Oct/15 05:47 pm>[824473.021724] ---[ end trace 0dbe4de46ac70cd0 ]---
     69<Oct/15 05:50 pm>[824473.077977] BUG: spinlock lockup on CPU#1, swapper/0, ffff88013624a100
     70<Oct/15 05:50 pm>[824473.077977] Pid: 0, comm: swapper Tainted: G      D   2.6.27-rc8-00016-gd3a47e8 #1
     71<Oct/15 05:50 pm>[824473.077977]
     72                 <Oct/15 05:50 pm>[824473.077977] Call Trace:
     73<Oct/15 05:50 pm>[824473.077977]  <IRQ>  [<ffffffff803be0c4>] _raw_spin_lock+0xf9/0x120
     74<Oct/15 05:50 pm>[824473.077977]  [<ffffffff805c8248>] _spin_lock_irqsave+0x37/0x3f
     75<Oct/15 05:50 pm>[824473.077977]  [<ffffffff80231d47>] tg_shares_up+0xcf/0x195
     76<Oct/15 05:50 pm>[824473.077977]  [<ffffffff80231d47>] tg_shares_up+0xcf/0x195
     77<Oct/15 05:50 pm>[824473.077977]  [<ffffffff80231c78>] tg_shares_up+0x0/0x195
     78<Oct/15 05:50 pm>[824473.124229] BUG: spinlock lockup on CPU#0, ppImage/27409, ffff88013624a100
     79<Oct/15 05:50 pm>[824473.124229] Pid: 27409, comm: ppImage Tainted: G      D   2.6.27-rc8-00016-gd3a47e8 #1
     80<Oct/15 05:50 pm>[824473.124229]
     81<Oct/15 05:50 pm>[824473.124229] Call Trace:
     82<Oct/15 05:50 pm>[824473.124229]  <IRQ>  [<ffffffff803be0c4>] _raw_spin_lock+0xf9/0x120
     83<Oct/15 05:50 pm>[824473.124229]  [<ffffffff805c8248>] _spin_lock_irqsave+0x37/0x3f
     84<Oct/15 05:50 pm>[824473.124229]  [<ffffffff80231d47>] tg_shares_up+0xcf/0x195
     85<Oct/15 05:50 pm>[824473.124229]  [<ffffffff80231d47>] tg_shares_up+0xcf/0x195
     86<Oct/15 05:50 pm>[824473.124229]  [<ffffffff80231c78>] tg_shares_up+0x0/0x195
     87<Oct/15 05:50 pm>[824473.124229]  [<ffffffff802296d8>] tg_nop+0x0/0x6
     88
     89<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8022adb5>] walk_tg_tree+0x90/0xbc
     90<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8022ad25>] walk_tg_tree+0x0/0xbc
     91<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8022e3c2>] try_to_wake_up+0xb0/0x238
     92<Oct/15 05:50 pm>[824473.124229]  [<ffffffff80229d46>] __wake_up_common+0x41/0x74
     93<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8022a8d2>] __wake_up_sync+0x3a/0x56
     94<Oct/15 05:50 pm>[824473.124229]  [<ffffffff80521e4a>] sock_def_readable+0x3d/0x6a
     95<Oct/15 05:50 pm>[824473.124229]  [<ffffffff80553256>] tcp_rcv_established+0x60f/0x8e5
     96<Oct/15 05:50 pm>[824473.124229]  [<ffffffff805586a3>] tcp_v4_do_rcv+0x2c/0x1b7
     97<Oct/15 05:50 pm>[824473.124229]  [<ffffffff80559e93>] tcp_v4_rcv+0x263/0x654
     98<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8055a231>] tcp_v4_rcv+0x601/0x654
     99<Oct/15 05:50 pm>[824473.124229]  [<ffffffff80540d01>] ip_local_deliver+0xcb/0x164
     100<Oct/15 05:50 pm>[824473.124229]  [<ffffffff80540c91>] ip_local_deliver+0x5b/0x164
     101<Oct/15 05:50 pm>[824473.124229]  [<ffffffff80541294>] ip_rcv+0x4fa/0x552
     102<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8052904b>] netif_receive_skb+0x24f/0x298
     103<Oct/15 05:50 pm>[824473.124229]  [<ffffffff80528f03>] netif_receive_skb+0x107/0x298
     104<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8052b795>] process_backlog+0x71/0xbe
     105<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8052b07e>] net_rx_action+0xc5/0x1fb
     106<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8052b010>] net_rx_action+0x57/0x1fb
     107<Oct/15 05:50 pm>[824473.124229]  [<ffffffff80239d7a>] __do_softirq+0x5e/0xcd
     108<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8021d70b>] ack_apic_level+0x12/0xd3
     109<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8020d17c>] call_softirq+0x1c/0x28
     110<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8020e82a>] do_softirq+0x2c/0x68
     111
     112<Oct/15 05:50 pm>[824473.124229]  [<ffffffff80239cd6>] irq_exit+0x3f/0x85
     113<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8020e9ca>] do_IRQ+0x13c/0x15e
     114<Oct/15 05:50 pm>[824473.124229]  [<ffffffff8020c3f1>] ret_from_intr+0x0/0xa
     115<Oct/15 05:50 pm>[824473.124229]  <EOI>
     116<Oct/15 05:50 pm>[824473.329415] BUG: spinlock lockup on CPU#2, ppImage/27703, ffff88013624a100
     117<Oct/15 05:50 pm>[824473.329415] Pid: 27703, comm: ppImage Tainted: G      D   2<Oct/15 05:50 pm>d3a47e8 #1
     118                 <Oct/15 05:50 pm>[824473.329415]
     119<Oct/15 05:50 pm>[824473.329415] Call Trace:
     120<Oct/15 05:50 pm>[824473.329415]  <IRQ>  [<ffffffff803be0c4>] _raw_spin_lock+0xf9/0x120
     121<Oct/15 05:50 pm>[824473.329415]  [<ffffffff80232437>] scheduler_tick+0x47/0x1b8
     122<Oct/15 05:50 pm>[824473.329415]  [<ffffffff8023e0b1>] update_process_times+0x3f/0x4b
     123<Oct/15 05:50 pm>[824473.329415]  [<ffffffff802506fa>] tick_sched_timer+0x76/0xa7
     124<Oct/15 05:50 pm>[824473.329415]  [<ffffffff80250684>] tick_sched_timer+0x0/0xa7
     125<Oct/15 05:50 pm>[824473.329415]  [<ffffffff80249fce>] __run_hrtimer+0x5a/0x97
     126<Oct/15 05:50 pm>[824473.329415]  [<ffffffff8024a9fb>] hrtimer_interrupt+0xe3/0x15d
     127<Oct/15 05:50 pm>[824473.329415]  [<ffffffff8021c29c>] smp_apic_timer_interrupt+0x89/0xa9
     128<Oct/15 05:50 pm>[824473.329415]  [<ffffffff8020cbc6>] apic_timer_interrupt+0x66/0x70
     129<Oct/15 05:50 pm>[824473.329415]  <EOI>  [<ffffffff80241808>] exit_signals+0xf4/0x10c
     130<Oct/15 05:50 pm>[824473.329415]  [<ffffffff805c81ee>] _spin_unlock_irq+0x21/0x22
     131<Oct/15 05:50 pm>[824473.329415]  [<ffffffff802417c6>] exit_signals+0xb2/0x10c
     132<Oct/15 05:50 pm>[824473.329415]  [<ffffffff80238123>] do_exit+0xd6/0x778
     133<Oct/15 05:50 pm>[824473.329415]  [<ffffffff805c8851>] oops_begin+0x0/0x8c
     134
     135<Oct/15 05:50 pm>[824473.329415]  [<ffffffff805ca732>] do_page_fault+0x73c/0x7f5
     136<Oct/15 05:50 pm>[824473.329415]  [<ffffffff803b457d>] __next_cpu+0x19/0x26
     137<Oct/15 05:50 pm>[824473.329415]  [<ffffffff805c844d>] error_exit+0x0/0x9a
     138<Oct/15 05:50 pm>[824473.329415]  [<ffffffff8022b04d>] pick_next_task_fair+0x7d/0xa9
     139<Oct/15 05:50 pm>[824473.329415]  [<ffffffff8022b04d>] pick_next_task_fair+0x7d/0xa9
     140<Oct/15 05:50 pm>[824473.329415]  [<ffffffff805c5f2a>] schedule+0x434/0x7e1
     141<Oct/15 05:50 pm>[824473.329415]  [<ffffffff805567ed>] tcp_connect+0x319/0x322
     142<Oct/15 05:50 pm>[824473.329415]  [<ffffffff8055974b>] tcp_v4_connect+0x3c1/0x422
     143<Oct/15 05:50 pm>[824473.329415]  [<ffffffff805c64ab>] schedule_timeout+0x1e/0xad
     144<Oct/15 05:50 pm>[824473.329415]  [<ffffffff805201cd>] release_sock+0x2f/0xd5
     145<Oct/15 05:50 pm>[824473.329415]  [<ffffffff80564a0f>] inet_stream_connect+0x137/0x241
     146<Oct/15 05:50 pm>[824473.329415]  [<ffffffff80247b2a>] autoremove_wake_function+0x0/0x2e
     147<Oct/15 05:50 pm>[824473.329415]  [<ffffffff8051eb3b>] sys_connect+0x6c/0x9c
     148<Oct/15 05:50 pm>[824473.329415]  [<ffffffff802a6290>] fget+0xb6/0xbe
     149<Oct/15 05:50 pm>[824473.329415]  [<ffffffff805c7c2a>] lockdep_sys_exit_thunk+0x35/0x67
     150<Oct/15 05:50 pm>[824473.329415]  [<ffffffff8039c9ac>] cap_file_fcntl+0x0/0x3
     151<Oct/15 05:50 pm>[824473.329415]  [<ffffffff802a61da>] fget+0x0/0xbe
     152<Oct/15 05:50 pm>[824473.329415]  [<ffffffff8020be2b>] system_call_fastpath+0x16/0x1b
     153<Oct/15 05:50 pm>[824473.329415]
     154<Oct/15 05:50 pm>[824473.077977]  [<ffffffff802296d8>] tg_nop+0x0/0x6
     155<Oct/15 05:50 pm>[824473.077977]  [<ffffffff8022adb5>] walk_tg_tree+0x90/0xbc
     156<Oct/15 05:50 pm>[824473.077977]  [<ffffffff8022ad25>] walk_tg_tree+0x0/0xbc
     157
     158<Oct/15 05:50 pm>[824473.077977]  [<ffffffff8022e6b0>] rebalance_domains+0x13b/0x3f7
     159<Oct/15 05:50 pm>[824473.077977]  [<ffffffff80231990>] run_rebalance_domains+0x42/0xd8
     160<Oct/15 05:50 pm>[824473.077977]  [<ffffffff80239d7a>] __do_softirq+0x5e/0xcd
     161<Oct/15 05:50 pm>[824473.077977]  [<ffffffff8020d17c>] call_softirq+0x1c/0x28
     162<Oct/15 05:50 pm>[824473.077977]  [<ffffffff8020e82a>] do_softirq+0x2c/0x68
     163<Oct/15 05:50 pm>[824473.077977]  [<ffffffff80239cd6>] irq_exit+0x3f/0x85
     164<Oct/15 05:50 pm>[824473.077977]  [<ffffffff8021c2a1>] smp_apic_timer_interrupt+0x8e/0xa9
     165<Oct/15 05:50 pm>[824473.077977]  [<ffffffff8020cbc6>] apic_timer_interrupt+0x66/0x70
     166<Oct/15 05:50 pm>[824473.077977]  <EOI>  [<ffffffff802123da>] c1e_idle+0x0/0xe9
     167<Oct/15 05:50 pm>[824473.077977]  [<ffffffff805ca841>] __atomic_notifier_call_chain+0x0/0x83
     168<Oct/15 05:50 pm>[824473.077977]  [<ffffffff802122c5>] default_idle+0x27/0x3b
     169<Oct/15 05:50 pm>[824473.077977]  [<ffffffff802124bf>] c1e_idle+0xe5/0xe9
     170<Oct/15 05:50 pm>[824473.077977]  [<ffffffff8020abde>] cpu_idle+0x88/0xae
     171<Oct/15 05:50 pm>[824473.077977]
     172<Oct/15 05:51 pm>[824473.022141] BUG: spinlock lockup on CPU#3, ppImage/27440, ffff88013624a100
     173<Oct/15 05:51 pm>[824473.022141] Pid: 27440, comm: ppImage Tainted: G      D   2.6.27-rc8-00016-gd3a47e8 #1
     174<Oct/15 05:51 pm>[824473.022141]
     175                 <Oct/15 05:51 pm>[824473.022141] Call Trace:
     176<Oct/15 05:51 pm>[824473.022141]  <IRQ>  [<ffffffff803be0c4>] _raw_spin_lock+0xf9/0x120
     177<Oct/15 05:51 pm>[824473.022141]  [<ffffffff805c8248>] _spin_lock_irqsave+0x37/0x3f
     178<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80231d47>] tg_shares_up+0xcf/0x195
     179<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80231d47>] tg_shares_up+0xcf/0x195
     180
     181<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80231c78>] tg_shares_up+0x0/0x195
     182<Oct/15 05:51 pm>[824473.022141]  [<ffffffff802296d8>] tg_nop+0x0/0x6
     183<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8022adb5>] walk_tg_tree+0x90/0xbc
     184<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8022ad25>] walk_tg_tree+0x0/0xbc
     185<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8022e3c2>] try_to_wake_up+0xb0/0x238
     186<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80563ff6>] inet_sk_rebuild_header+0x19/0x31a
     187<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80247b33>] autoremove_wake_function+0x9/0x2e
     188<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80229d46>] __wake_up_common+0x41/0x74
     189<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8022a926>] __wake_up+0x38/0x4f
     190<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80520bb0>] sock_def_wakeup+0x3c/0x48
     191<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8055b670>] tcp_init_congestion_control+0x19/0xc5
     192<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8055250a>] tcp_rcv_state_process+0x2eb/0xa28
     193<Oct/15 05:51 pm>[824473.022141]  [<ffffffff805587d8>] tcp_v4_do_rcv+0x161/0x1b7
     194<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80559e93>] tcp_v4_rcv+0x263/0x654
     195<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8055a231>] tcp_v4_rcv+0x601/0x654
     196<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80540d01>] ip_local_deliver+0xcb/0x164
     197<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80540c91>] ip_local_deliver+0x5b/0x164
     198<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80541294>] ip_rcv+0x4fa/0x552
     199<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8052904b>] netif_receive_skb+0x24f/0x298
     200<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80528f03>] netif_receive_skb+0x107/0x298
     201<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8052b795>] process_backlog+0x71/0xbe
     202<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8052b07e>] net_rx_action+0xc5/0x1fb
     203
     204<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8052b010>] net_rx_action+0x57/0x1fb
     205<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80239d7a>] __do_softirq+0x5e/0xcd
     206<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8021d70b>] ack_apic_level+0x12/0xd3
     207<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8020d17c>] call_softirq+0x1c/0x28
     208<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8020e82a>] do_softirq+0x2c/0x68
     209<Oct/15 05:51 pm>[824473.022141]  [<ffffffff80239cd6>] irq_exit+0x3f/0x85
     210<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8020e9ca>] do_IRQ+0x13c/0x15e
     211<Oct/15 05:51 pm>[824473.022141]  [<ffffffff8020c3f1>] ret_from_intr+0x0/0xa
     212<Oct/15 05:51 pm>[824473.022141]  <EOI></pre>