<pre> <Jan/14 07:59 am>This is ipp016.unknown_domain (Linux x86_64 2.6.28-rc7-00105-gfeaf384) 09:42:23

<Jan/16 10:34 am>ipp016 login: [182195.397689] do_IRQ: 1.189 No irq handler for vector <Jan/20 03:06 pm>[544164.218969] divide error: 0000 #1 SMP <Jan/20 03:06 pm>[544164.222254] last sysfs file: /sys/devices/system/cpu/sched_mc_power_savings <Jan/20 03:06 pm>[544164.222254] CPU 0 <Jan/20 03:06 pm>[544164.222254] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt i2c_nforce2 i2c_core k8temp forcedeth tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan <Jan/20 03:06 pm>[544164.222254] Pid: 25009, comm: ppStack Not tainted 2.6.28-rc7-00105-gfeaf384 #4 <Jan/20 03:06 pm>[544164.222254] RIP: 0010:[<ffffffff80234fd4>] [<ffffffff80234fd4>] load_balance_fair+0xd9/0x221 <Jan/20 03:06 pm>[544164.222254] RSP: 0000:ffffffff80938d20 EFLAGS: 00010046 <Jan/20 03:06 pm>[544164.222254] RAX: 00000000000aa000 RBX: 0000000000000010 RCX: 0000000000000000 <Jan/20 03:06 pm>[544164.222254] RDX: 0000000000000000 RSI: 0000000000000027 RDI: ffff88022bda5b20 <Jan/20 03:06 pm>[544164.222254] RBP: ffffffff80938dc0 R08: 0000000000000002 R09: ffffffff8022f4b1 <Jan/20 03:06 pm>[544164.222254] R10: 0000000000000000 R11: ffffffff80231684 R12: ffff88002805e480 <Jan/20 03:06 pm>[544164.222254] R13: ffff88022b10e540 R14: 0000000000000001 R15: ffff88022bde5480 <Jan/20 03:06 pm>[544164.222254] FS: 0000000042816950(0063) GS:ffffffff80872d80(0000) knlGS:0000000000000000

<Jan/20 03:06 pm>[544164.222254] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <Jan/20 03:06 pm>[544164.222254] CR2: 00007f61500e5000 CR3: 00000001e4fc8000 CR4: 00000000000006e0 <Jan/20 03:06 pm>[544164.222254] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 <Jan/20 03:06 pm>[544164.222254] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 <Jan/20 03:06 pm>[544164.222254] Process ppStack (pid: 25009, threadinfo ffff88003fb66000, task ffff88022bda5b20) <Jan/20 03:06 pm>[544164.222254] Stack: <Jan/20 03:06 pm>[544164.222254] ffffffff80234efb 0000000100000400 0000000180938e88 ffff880028058a00 <Jan/20 03:06 pm>[544164.222254] 0000000000000154 ffff880135a4a480 000000002faeb680 ffff88002805e480 <Jan/20 03:06 pm>[544164.222254] 0000000000000010 0000000000000154 007fffffffffffff 0000000000000800 <Jan/20 03:06 pm>[544164.222254] Call Trace: <Jan/20 03:06 pm>[544164.222254] <IRQ> <0> [<ffffffff80234efb>] ? load_balance_fair+0x0/0x221 <Jan/20 03:06 pm>[544164.222254] [<ffffffff80257376>] ? lock_acquire+0x788/0x7f3 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8022df5a>] move_tasks+0x67/0xa3 <Jan/20 03:06 pm>[544164.222254] [<ffffffff802502b6>] ? sched_clock_cpu+0xe8/0x12b <Jan/20 03:06 pm>[544164.222254] [<ffffffff802329f6>] rebalance_domains+0x1e9/0x3f8 <Jan/20 03:06 pm>[544164.222254] [<ffffffff80257376>] ? lock_acquire+0x788/0x7f3 <Jan/20 03:06 pm>[544164.222254] [<ffffffff80236528>] run_rebalance_domains+0x42/0xd6 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8024ec57>] ? ktime_get+0xc/0x41 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8023e144>] do_softirq+0x7a/0x13d <Jan/20 03:06 pm>[544164.222254] [<ffffffff8020d07c>] ? call_softirq+0x1c/0x28 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8020e028>] ? do_softirq+0x2c/0x68 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8023e084>] ? irq_exit+0x3f/0x85

<Jan/20 03:06 pm>[544164.222254] [<ffffffff8021e66d>] ? smp_apic_timer_interrupt+0x93/0xac <Jan/20 03:06 pm>[544164.222254] [<ffffffff8020cacb>] ? apic_timer_interrupt+0x6b/0x70 <Jan/20 03:06 pm>[544164.222254] <EOI> <0>Code: 90 00 00 00 00 0f 84 16 01 00 00 48 8b 45 a8 c7 45 d4 00 00 00 00 c7 45 c0 00 00 00 00 48 0f af c2 8b 55 b0 ff c2 48 89 d1 31 d2 <48> f7 f1 48 85 c0 49 89 c6 49 89 c4 0f 84 9a 00 00 00 4c 89 ef <Jan/20 03:06 pm>[544164.222254] RIP [<ffffffff80234fd4>] load_balance_fair+0xd9/0x221 <Jan/20 03:06 pm>[544164.222254] RSP <ffffffff80938d20> <Jan/20 03:06 pm>[544164.222254] Kernel panic - not syncing: Fatal exception in <Jan/20 03:06 pm>

<Jan/20 03:06 pm>[544164.222254] ------------[ cut here ]------------

<Jan/20 03:06 pm>[544164.222254] WARNING: at kernel/smp.c:333 smp_call_function_mask+0x37/0x1d7() <Jan/20 03:06 pm>[544164.222254] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt i2c_nforce2 i2c_core k8temp forcedeth tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan <Jan/20 03:06 pm>[544164.222254] Pid: 25009, comm: ppStack Tainted: G D 2.6.28-rc7-00105-gfeaf384 #4 <Jan/20 03:06 pm>[544164.222254] Call Trace: <Jan/20 03:06 pm>[544164.222254] <IRQ> [<ffffffff802397db>] warn_on_slowpath+0x51/0x6d <Jan/20 03:06 pm>[544164.222254] [<ffffffff80239d00>] try_acquire_console_sem+0x12/0x31 <Jan/20 03:06 pm>[544164.222254] [<ffffffff80239ea9>] release_console_sem+0x18a/0x1a5 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8025bc31>] smp_call_function_mask+0x37/0x1d7 <Jan/20 03:06 pm>[544164.222254] [<ffffffff80264013>] crash_kexec+0x17/0xef <Jan/20 03:06 pm>[544164.222254] [<ffffffff802640e2>] crash_kexec+0xe6/0xef <Jan/20 03:06 pm>[544164.222254] [<ffffffff8021d53e>] native_smp_send_stop+0x1a/0x26 <Jan/20 03:06 pm>[544164.222254] [<ffffffff80239896>] panic+0x95/0x13f <Jan/20 03:06 pm>[544164.222254] [<ffffffff803af923>] vgacon_set_cursor_size+0x40/0x10b <Jan/20 03:06 pm>[544164.222254] [<ffffffff805b83ed>] oops_end+0x7b/0x88 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8020d3e8>] do_divide_error+0x7f/0x89

<Jan/20 03:06 pm>[544164.222254] [<ffffffff80234fd4>] load_balance_fair+0xd9/0x221 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8055bd5b>] bictcp_cong_avoid+0xf/0x220 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8022ecd2>] update_curr+0x93/0xc9 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8022ecd2>] update_curr+0x93/0xc9 <Jan/20 03:06 pm>[544164.222254] [<ffffffff805b795d>] error_exit+0x0/0x9a <Jan/20 03:06 pm>[544164.222254] [<ffffffff80231684>] double_rq_lock+0x4d/0x79 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8022f4b1>] walk_tg_tree+0x0/0xc7 <Jan/20 03:06 pm>[544164.222254] [<ffffffff80234fd4>] load_balance_fair+0xd9/0x221 <Jan/20 03:06 pm>[544164.222254] [<ffffffff80234efb>] load_balance_fair+0x0/0x221 <Jan/20 03:06 pm>[544164.222254] [<ffffffff80257376>] lock_acquire+0x788/0x7f3 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8022df5a>] move_tasks+0x67/0xa3 <Jan/20 03:06 pm>[544164.222254] [<ffffffff802502b6>] sched_clock_cpu+0xe8/0x12<Jan/20 03:06 pm>

<Jan/20 03:06 pm>[544164.222254] [<ffffffff802329f6>] rebalance_domains+0x1e9/0x3f8

<Jan/20 03:06 pm>[544164.222254] [<ffffffff80257376>] lock_acquire+0x788/0x7f3 <Jan/20 03:06 pm>[544164.222254] [<ffffffff80236528>] run_rebalance_domains+0x42/0xd6 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8024ec57>] ktime_get+0xc/0x41 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8023e144>] do_softirq+0x7a/0x13d

<Jan/20 03:06 pm>[544164.222254] [<ffffffff8020d07c>] call_softirq+0x1c/0x28

<Jan/20 03:06 pm>[544164.222254] [<ffffffff8020e028>] do_softirq+0x2c/0x68 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8023e084>] irq_exit+0x3f/0x85 <Jan/20 03:06 pm>[544164.222254] [<ffffffff8021e66d>] smp_apic_timer_interrupt+0x93/0xac <Jan/20 03:06 pm>[544164.222254] [<ffffffff8020cacb>] apic_timer_interrupt+0x6b/0x70

<Jan/20 03:06 pm>[544164.222254] <EOI> <4>---[ end trace 1ecc68bcae4cc46c ]--- <Jan/20 03:06 pm>[544164.226007] BUG: spinlock lockup on CPU#1, ppStack/25010, ffff88002805e480 <Jan/20 03:06 pm>[544164.226007] Pid: 25010, comm: ppStack Tainted: G D W 2.6.28-rc7-00105-gfeaf384 #4 <Jan/20 03:06 pm>[544164.226007] Call Trace: <Jan/20 03:06 pm>[544164.226007] <IRQ> [<ffffffff803a103f>] _raw_spin_lock+0xfb/0x122 <Jan/20 03:06 pm>[544164.226007] [<ffffffff805b7758>] _spin_lock_irqsave+0x37/0x3f <Jan/20 03:06 pm>[544164.226007] [<ffffffff80235bc3>] tg_shares_up+0x194/0x212 <Jan/20 03:06 pm>[544164.226007] [<ffffffff80235bc3>] tg_shares_up+0x194/0x212 <Jan/20 03:06 pm>[544164.226007] [<ffffffff80232bb6>] rebalance_domains+0x3a9/0x3f8 <Jan/20 03:06 pm>[544164.226007] [<ffffffff80235a2f>] tg_shares_up+0x0/0x212 <Jan/20 03:06 pm>[544164.226007] [<ffffffff8022d8c2>] tg_nop+0x0/0x8 <Jan/20 03:06 pm>[544164.226007] [<ffffffff8022f541>] walk_tg_tree+0x90/0xc7 <Jan/20 03:06 pm>[544164.226007] [<ffffffff8022f4b1>] walk_tg_tree+0x0/0xc7 <Jan/20 03:06 pm>[544164.226007] [<ffffffff80232657>] try_to_wake_up+0xb0/0x23b <Jan/20 03:06 pm>[544164.226007] [<ffffffff8024e48e>] hrtimer_wakeup+0x0/0x21 <Jan/20 03:06 pm>[544164.226007] [<ffffffff8024e4ab>] hrtimer_wakeup+0x1d/0x21 <Jan/20 03:06 pm>[544164.226007] [<ffffffff8024e508>] run_hrtimer+0x59/0x95 <Jan/20 03:06 pm>[544164.226007] [<ffffffff8024ed6d>] hrtimer_interrupt+0xe1/0x15b <Jan/20 03:06 pm>[544164.226007] [<ffffffff8021e668>] smp_apic_timer_interrupt+0x8e/0xac <Jan/20 03:06 pm>[544164.226007] [<ffffffff8020cacb>] apic_timer_interrupt+0x6b<Jan/20 03:06 pm>

<Jan/20 03:07 pm>[544164.226007] <EOI> <0>BUG: spinlock lockup on CPU#0, ppStack/25009, ffff88002805e480

<Jan/20 03:07 pm>[544164.222254] Pid: 25009, comm: ppStack Tainted: G D W 2.6.28-rc7-00105-gfeaf384 #4

<Jan/20 03:07 pm>[544164.222254] Call Trace: <Jan/20 03:07 pm>[544164.222254] <IRQ> [<ffffffff803a103f>] _raw_spin_lock+0xfb/0x122 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8022f1c9>] task_rq_lock+0x40/0x75 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8023268d>] try_to_wake_up+0xe6/0x23b <Jan/20 03:07 pm>[544164.222254] [<ffffffff8022dffe>] wake_up_common+0x41/0x74 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8022f03e>] wake_up+0x38/0x4e <Jan/20 03:07 pm>[544164.222254] [<ffffffff8024258a>] update_process_times+0x3f/0x50 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8025520a>] tick_sched_timer+0x76/0x9c <Jan/20 03:07 pm>[544164.222254] [<ffffffff80255194>] tick_sched_timer+0x0/0x9c <Jan/20 03:07 pm>[544164.222254] [<ffffffff8024e508>] run_hrtimer+0x59/0x95 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8024ed6d>] hrtimer_interrupt+0xe1/0x15b <Jan/20 03:07 pm>[544164.222254] [<ffffffff8021e668>] smp_apic_timer_interrupt+0x8e/0xac <Jan/20 03:07 pm>[544164.222254] [<ffffffff8020cacb>] apic_timer_interrupt+0x6b<Jan/20 03:07 pm>

<Jan/20 03:07 pm>[544164.222254] [<ffffffff8021e686>] native_apic_mem_write+0x0/0x9

<Jan/20 03:07 pm>[544164.222254] [<ffffffff8021dff7>] clear_local_APIC+0x187/0x19c <Jan/20 03:07 pm>[544164.222254] [<ffffffff804c3020>] i8042_panic_blink+0x0/0x145 <Jan/20 03:07 pm>[544164.222254] [<ffffffff805b9df2>] atomic_notifier_call_chain+0x0/0x83 <Jan/20 03:07 pm>[544164.222254] [<ffffffff805b9df2>] atomic_notifier_call_chain+0x0/0x83 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8023991e>] panic+0x11d/0x13f <Jan/20 03:07 pm>[544164.222254] [<ffffffff802398ab>] panic+0xaa/0x13f <Jan/20 03:07 pm>[544164.222254] [<ffffffff803af923>] vgacon_set_cursor_size+0x40/0x10b <Jan/20 03:07 pm>[544164.222254] [<ffffffff805b83ed>] oops_end+0x7b/0x88

<Jan/20 03:07 pm>[544164.222254] [<ffffffff8020d3e8>] do_divide_error+0x7f/0x89 <Jan/20 03:07 pm>[544164.222254] [<ffffffff80234fd4>] load_balance_fair+0xd9/0x221 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8055bd5b>] bictcp_cong_avoid+0xf/0x220 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8022ecd2>] update_curr+0x93/0xc9 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8022ecd2>] update_curr+0x93/0xc9 <Jan/20 03:07 pm>[544164.222254] [<ffffffff805b795d>] error_exit+0x0/0x9a <Jan/20 03:07 pm>[544164.222254] [<ffffffff80231684>] double_rq_lock+0x4d/0x79 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8022f4b1>] walk_tg_tree+0x0/0xc7 <Jan/20 03:07 pm>[544164.222254] [<ffffffff80234fd4>] load_balance_fair+0xd9/0x221 <Jan/20 03:07 pm>[544164.222254] [<ffffffff80234efb>] load_balance_fair+0x0/0x221 <Jan/20 03:07 pm>[544164.222254] [<ffffffff80257376>] lock_acquire+0x788/0x7f3 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8022df5a>] move_tasks+0x67/0xa3 <Jan/20 03:07 pm>[544164.222254] [<ffffffff802502b6>] sched_clock_cpu+0xe8/0x12b <Jan/20 03:07 pm>[544164.222254] [<ffffffff802329f6>] rebalance_domains+0x1e9/0x3f8 <Jan/20 03:07 pm>[544164.222254] [<ffffffff80257376>] lock_acquire+0x788/0x7f3 <Jan/20 03:07 pm>[544164.222254] [<ffffffff80236528>] run_rebalance_domains+0x42/0xd6 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8024ec57>] ktime_get+0xc/0x41 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8023e144>] do_softirq+0x7a/0x13d <Jan/20 03:07 pm>[544164.222254] [<ffffffff8020d07c>] call_softirq+0x1c/0x28 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8020e028>] do_softirq+0x2c/0x68 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8023e084>] irq_exit+0x3f/0x85 <Jan/20 03:07 pm>[544164.222254] [<ffffffff8021e66d>] smp_apic_timer_interrupt+<Jan/20 03:07 pm>

<Jan/20 03:07 pm>[544164.222254] [<ffffffff8020cacb>] apic_timer_interrupt+0x6b/0x70 <Jan/20 03:07 pm>[544164.222254] <EOI> <0>BUG: spinlock lockup on CPU#3, ppImage/24778, ffff880135a4a480 <Jan/20 03:07 pm>[544164.223507] Pid: 24778, comm: ppImage Tainted: G D W 2.6.28-rc7-00105-gfeaf384 #4 <Jan/20 03:07 pm>[544164.223507] Call Trace: <Jan/20 03:07 pm>[544164.223507] <IRQ> [<ffffffff803a103f>] _raw_spin_lock+0xfb/0x122 <Jan/20 03:07 pm>[544164.223507] [<ffffffff805b7758>] _spin_lock_irqsave+0x37/0x3f <Jan/20 03:07 pm>[544164.223507] [<ffffffff80235bc3>] tg_shares_up+0x194/0x212 <Jan/20 03:07 pm>[544164.223507] [<ffffffff80235bc3>] tg_shares_up+0x194/0x212 <Jan/20 03:07 pm>[544164.223507] [<ffffffff80235a2f>] tg_shares_up+0x0/0x212 <Jan/20 03:07 pm>[544164.223507] [<ffffffff8022d8c2>] tg_nop+0x0/0x8 <Jan/20 03:07 pm>[544164.223507] [<ffffffff8022f541>] walk_tg_tree+0x90/0xc7 <Jan/20 03:07 pm>[544164.223507] [<ffffffff8022f4b1>] walk_tg_tree+0x0/0xc7 <Jan/20 03:07 pm>[544164.223507] [<ffffffff80232657>] try_to_wake_up+0xb0/0x23b <Jan/20 03:07 pm>[544164.223507] [<ffffffff8024e48e>] hrtimer_wakeup+0x0/0x21 <Jan/20 03:07 pm>[544164.223507] [<ffffffff8024e4ab>] hrtimer_wakeup+0x1d/0x21 <Jan/20 03:07 pm>[544164.223507] [<ffffffff8024e508>] run_hrtimer+0x59/0x95 <Jan/20 03:07 pm>[544164.223507] [<ffffffff8024ed6d>] hrtimer_interrupt+0xe1/0x15b <Jan/20 03:07 pm>[544164.223507] [<ffffffff8021e668>] smp_apic_timer_interrupt+0x8e/0xac <Jan/20 03:07 pm>[544164.223507] [<ffffffff8020cacb>] apic_timer_interrupt+0x6b/0x70 <Jan/20 03:07 pm>[544164.223507] <EOI> [<ffffffff8028a8b6>] do_wp_page+0x5e/0x55b <Jan/20 03:07 pm>[544164.223507] [<ffffffff8028a88d>] do_wp_page+0x35/0x55b <Jan/20 03:07 pm>[544164.223507] [<ffffffff8028c4fa>] handle_mm_fault+0x6ca/0x728

<Jan/20 03:07 pm>[544164.223507] [<ffffffff805b99e1>] do_page_fault+0x424/0x7df <Jan/20 03:07 pm>[544164.223507] [<ffffffff805b7141>] lockdep_sys_exit_thunk+0x35/0x67 <Jan/20 03:07 pm>[544164.223507] [<ffffffff805b795d>] error_exit+0x0/0x9a <Jan/20 03:08 pm>[544164.223098] BUG: spinlock lockup on CPU#2, ppStack/25008, ffff880135a4a480 <Jan/20 03:08 pm>[544164.223098] Pid: 25008, comm: ppStack Tainted: G D W 2.6.28-rc7-00105-gfeaf384 #4 <Jan/20 03:08 pm>[544164.223098] Call Trace: <Jan/20 03:08 pm>[544164.223098] <IRQ> [<ffffffff803a103f>] _raw_spin_lock+0xfb/0x122 <Jan/20 03:08 pm>[544164.223098] [<ffffffff80236d8a>] scheduler_tick+0x47/0x1ba <Jan/20 03:08 pm>[544164.223098] [<ffffffff8024258f>] update_process_times+0x44/0x50 <Jan/20 03:08 pm>[544164.223098] [<ffffffff8025520a>] tick_sched_timer+0x76/0x9c <Jan/20 03:08 pm>[544164.223098] [<ffffffff80255194>] tick_sched_timer+0x0/0x9c <Jan/20 03:08 pm>[544164.223098] [<ffffffff8024e508>] run_hrtimer+0x59/0x95 <Jan/20 03:08 pm>[544164.223098] [<ffffffff8024ed6d>] hrtimer_interrupt+0xe1/0x15b <Jan/20 03:08 pm>[544164.223098] [<ffffffff8021e668>] smp_apic_timer_interrupt+0x8e/0xac <Jan/20 03:08 pm>[544164.223098] [<ffffffff8020cacb>] apic_timer_interrupt+0x6b/0x70 </pre>