IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Initial Version and Version 1 of Ippc28-crash-20121224


Ignore:
Timestamp:
Dec 24, 2014, 9:39:43 AM (12 years ago)
Author:
Mark Huber
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Ippc28-crash-20121224

    v1 v1  
     1from console -- power cycle stalled a while on starting local but seems to be back up
     2
     3{{{
     4<Dec/24 01:40 am>ippc28 login: [1507747.909968] general protection fault: 0000 [#1] SMP
     5<Dec/24 01:40 am>[1507747.912564] last sysfs file: /sys/class/i2c-adapter/i2c-0/0-002f/temp6_alarm
     6<Dec/24 01:40 am>[1507747.912564] CPU 0
     7<Dec/24 01:40 am>[1507747.912564] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     8<Dec/24 01:40 am>[1507747.912564] Pid: 1671, comm: rpciod/0 Not tainted 2.6.28-rc7-00105-gfeaf384 #4
     9<Dec/24 01:40 am>[1507747.912564] RIP: 0010:[<ffffffff802488fd>]  [<ffffffff802488fd>] run_workqueue+0x77/0x1ed
     10<Dec/24 01:40 am>[1507747.912564] RSP: 0018:ffff88063b293e80  EFLAGS: 00010087
     11<Dec/24 01:40 am>[1507747.912564] RAX: 1aa494a4d74ac649 RBX: ffff88013f277580 RCX: 0000000000000000
     12<Dec/24 01:40 am>[1507747.912564] RDX: ffff8809f4e1b4f8 RSI: ffff88013f2775b0 RDI: ffff88063b233000
     13<Dec/24 01:40 am>[1507747.912564] RBP: ffff88063b233000 R08: 0000000000000001 R09: 0000000000000001
     14<Dec/24 01:40 am>[1507747.912564] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88013f277578
     15<Dec/24 01:40 am>[1507747.912564] R13: cc4a7f6d487a4100 R14: ffffffff808b85a0 R15: 0000000000000000
     16<Dec/24 01:40 am>[1507747.912564] FS:  0000000000000000(0000) GS:ffffffff80872d80(0000) knlGS:0000000000000000
     17
     18<Dec/24 01:40 am>[1507747.912564] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     19<Dec/24 01:40 am>[1507747.912564] CR2: 0000000000a1c008 CR3: 0000000638454000 CR4: 00000000000006e0
     20<Dec/24 01:40 am>[1507747.912564] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     21<Dec/24 01:40 am>[1507747.912564] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     22<Dec/24 01:40 am>[1507747.912564] Process rpciod/0 (pid: 1671, threadinfo ffff88063b292000, task ffff88063b219e60)
     23<Dec/24 01:40 am>[1507747.912564] Stack:
     24<Dec/24 01:40 am>[1507747.912564]  0000000000000001 0000000000000046 5f185ad49f1d7d91 95c4cf635cb05640
     25<Dec/24 01:40 am>[1507747.912564]  a3d560adfe0c5e41 ffff88063b233000 ffff88063cc6dbf0 ffff88063b233000
     26<Dec/24 01:40 am>[1507748.069655]  ffff88063b233040 ffff88063cc6dbf0 ffffffff808b5d60 ffffffff80249511
     27<Dec/24 01:40 am>[1507748.069655] Call Trace:
     28<Dec/24 01:40 am>[1507748.069655]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
     29<Dec/24 01:40 am>[1507748.069655]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
     30<Dec/24 01:40 am>[1507748.069655]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
     31<Dec/24 01:40 am>[1507748.069655]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     32<Dec/24 01:40 am>[1507748.069655]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     33<Dec/24 01:40 am>[1507748.069655]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     34<Dec/24 01:40 am>[1507748.069655]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     35<Dec/24 01:40 am>[1507748.069655]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     36<Dec/24 01:40 am>[1507748.069655] Code: 48 8d 7c 24 10 48 8d 73 18 fc 4d 8b 6c 24 18 b9 06 00 00 00 f3 a5 4c 89 a5 80 00 00 00 48 89 ef 48 8b 43 08 48 8b 13 48 89 42 08 <48> 89 10 48 89 5b 08 48 89 1b e8 d1 ed 36 00 48 8b 43 f8 48 83
     37<Dec/24 01:40 am>[1507748.069655] RIP  [<ffffffff802488fd>] run_workqueue+0x77/0x1ed
     38<Dec/24 01:40 am>[1507748.069655]  RSP <ffff88063b293e80>
     39<Dec/24 01:40 am>[1507748.069655] ---[ end trace d2b44a22042f9805 ]---
     40
     41<Dec/24 01:40 am>[1507761.949892] nfs: server ippc18 not responding, still trying
     42<Dec/24 01:41 am>[1507794.116723] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
     43<Dec/24 01:41 am>[1507794.124527] ata1.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
     44<Dec/24 01:41 am>[1507794.124528]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
     45<Dec/24 01:41 am>[1507794.139389] ata1.00: status: { DRDY }
     46<Dec/24 01:41 am>[1507794.143535] ata1: hard resetting link
     47<Dec/24 01:41 am>[1507794.480033] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
     48<Dec/24 01:41 am>[1507799.493344] ata1.00: qc timeout (cmd 0xec)
     49<Dec/24 01:41 am>[1507799.497890] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
     50<Dec/24 01:41 am>[1507799.504459] ata1.00: revalidation failed (errno=-5)
     51<Dec/24 01:41 am>[1507799.509803] ata1: hard resetting link
     52<Dec/24 01:41 am>[1507799.876696] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
     53<Dec/24 01:41 am>[1507809.902766] ata2.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
     54<Dec/24 01:41 am>[1507809.903379] ata1.00: qc timeout (cmd 0xec)
     55<Dec/24 01:41 am>[1507809.903386] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
     56<Dec/24 01:41 am>[1507809.903388] ata1.00: revalidation failed (errno=-5)
     57<Dec/24 01:41 am>[1507809.903392] ata1: hard resetting link
     58<Dec/24 01:41 am>[1507809.931076] ata2.00: cmd 60/40:00:b1:05:03/00:00:00:00:00/40 tag 0 ncq 32768 in
     59<Dec/24 01:41 am>[1507809.931078]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
     60<Dec/24 01:41 am>[1507809.947109] ata2.00: status: { DRDY }
     61<Dec/24 01:41 am>[1507809.951224] ata2: hard resetting link
     62<Dec/24 01:41 am>[1507812.902775] ata2: SATA link up 3.0 Gbps (SStatus 123 SCont<Dec/24 01:41 am>
     63
     64<Dec/24 01:41 am>[1507812.903400] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
     65<Dec/24 01:41 am>[1507817.902768] ata2.00: qc timeout (cmd 0xec)
     66<Dec/24 01:41 am>[1507817.907368] ata2.00: failed to IDENTIFY (I/O error, err_mask=0x4)
     67<Dec/24 01:41 am>[1507817.914048] ata2.00: revalidation failed (errno=-5)
     68<Dec/24 01:41 am>[1507817.919434] ata2: hard resetting link
     69<Dec/24 01:41 am>[1507820.902776] ata2: SATA link up 3.0 Gbps (SStatus 123 SCont<Dec/24 01:41 am>
     70                 <Dec/24 01:41 am>[1507830.902766] ata2.00: qc timeout (cmd 0xec)
     71<Dec/24 01:41 am>[1507830.907323] ata2.00: failed to IDENTIFY (I/O error, err_mask=0x4)
     72<Dec/24 01:41 am>[1507830.913908] ata2.00: revalidation failed (errno=-5)
     73<Dec/24 01:41 am>[1507830.919250] ata2: hard resetting link
     74<Dec/24 01:41 am>[1507830.903013] BUG: spinlock lockup on CPU#0, rpciod/0/1671, <Dec/24 01:41 am>
     75                 <Dec/24 01:41 am>[1507830.903013] Pid: 1671, comm: rpciod/0 Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     76<Dec/24 01:41 am>[1507830.903013] Call Trace:
     77<Dec/24 01:41 am>[1507830.903013]  <IRQ>  [<ffffffff803a103f>] _raw_spin_lock+0xfb/0x122
     78<Dec/24 01:41 am>[1507830.903013]  [<ffffffff80589645>] xprt_init_autodisconnect+0x0/0x98
     79<Dec/24 01:41 am>[1507830.903013]  [<ffffffff805b7758>] _spin_lock_irqsave+0x37/0x3f
     80<Dec/24 01:41 am>[1507830.903013]  [<ffffffff802492ab>] __queue_work+0x10/0x32
     81<Dec/24 01:41 am>[1507830.903013]  [<ffffffff802492ab>] __queue_work+0x10/0x32
     82<Dec/24 01:41 am>[1507830.903013]  [<ffffffff80589645>] xprt_init_autodisconnect+0x0/0x98
     83<Dec/24 01:41 am>[1507830.903013]  [<ffffffff8024932c>] queue_work_on+0x37/0x40
     84<Dec/24 01:41 am>[1507830.903013]  [<ffffffff80241e07>] run_timer_softirq+0x169/0x1dd
     85<Dec/24 01:41 am>[1507830.903013]  [<ffffffff8023e144>] __do_softirq+0x7a/0x13d
     86
     87<Dec/24 01:41 am>[1507830.903013]  [<ffffffff8020d07c>] call_softirq+0x1c/0x28
     88<Dec/24 01:41 am>[1507830.903013]  [<ffffffff8020e028>] do_softirq+0x2c/0x68
     89<Dec/24 01:41 am>[1507830.903013]  [<ffffffff8023e084>] irq_exit+0x3f/0x85
     90<Dec/24 01:41 am>[1507830.903013]  [<ffffffff8021e66d>] smp_apic_timer_interrupt+0x93/0xac
     91<Dec/24 01:41 am>[1507830.903013]  [<ffffffff8020cacb>] apic_timer_interrupt+0x6b/0x70
     92<Dec/24 01:41 am>[1507830.903013]  <EOI>  [<ffffffff805b76fe>] _spin_unlock_irq+0x21/0x22
     93<Dec/24 01:41 am>[1507830.903013]  [<ffffffff803f3936>] tty_audit_exit+0x60/0x92
     94<Dec/24 01:41 am>[1507830.903013]  [<ffffffff8023c516>] do_exit+0x1b0/0x74d
     95<Dec/24 01:41 am>[1507830.903013]  [<ffffffff805b83fa>] oops_begin+0x0/0x8d
     96<Dec/24 01:41 am>[1507830.903013]  [<ffffffff805b795d>] error_exit+0x0/0x9a
     97<Dec/24 01:41 am>[1507830.903013]  [<ffffffff802488fd>] run_workqueue+0x77/0x1ed
     98<Dec/24 01:41 am>[1507830.903013]  [<ffffffff80249511>] worker_thread+0xd8/0xe3
     99<Dec/24 01:41 am>[1507830.903013]  [<ffffffff8024bfda>] autoremove_wake_function+0x0/0x2e
     100<Dec/24 01:41 am>[1507830.903013]  [<ffffffff80249439>] worker_thread+0x0/0xe3
     101<Dec/24 01:41 am>[1507830.903013]  [<ffffffff8024becc>] kthread+0x47/0x73
     102<Dec/24 01:41 am>[1507830.903013]  [<ffffffff80234852>] schedule_tail+0x27/0x5f
     103<Dec/24 01:41 am>[1507830.903013]  [<ffffffff8020cd19>] child_rip+0xa/0x11
     104<Dec/24 01:41 am>[1507830.903013]  [<ffffffff8024be85>] kthread+0x0/0x73
     105<Dec/24 01:41 am>[1507830.903013]  [<ffffffff8020cd0f>] child_rip+0x0/0x11
     106<Dec/24 01:41 am>[1507833.902778] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
     107<Dec/24 01:41 am>[1507842.903354] ata1.00: qc timeout (cmd 0xec)
     108<Dec/24 01:41 am>[1507842.907950] ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
     109
     110<Dec/24 01:41 am>[1507842.914597] ata1.00: revalidation failed (errno=-5)
     111<Dec/24 01:41 am>[1507842.920029] ata1.00: disabled
     112<Dec/24 01:41 am>[1507842.923489] ata1: hard resetting link
     113<Dec/24 01:41 am>[1507845.903359] ata1: SATA link up 3.0 Gbps (SStatus 123 SCont<Dec/24 01:41 am>
     114                 <Dec/24 01:41 am>[1507845.910057] ata1: EH complete
     115<Dec/24 01:41 am>[1507845.913479] sd 0:0:0:0: [sda] Result: hostbyte=DID_BAD_TAR<Dec/24 01:41 am>IVER_OK,SUGGEST_OK
     116                 <Dec/24 01:41 am>[1507845.922808] end_request: I/O error, dev sda, sector 96195185
     117<Dec/24 01:41 am>[1507845.928983] end_request: I/O error, dev sda, sector 96195185
     118<Dec/24 01:41 am>[1507845.932307] md: super_written gets error=-5, uptodate=0
     119<Dec/24 01:41 am>[1507845.932307] raid1: Disk failure on sda3, disabling device.
     120<Dec/24 01:41 am>[1507845.932307] raid1: Operation continuing on 1 devices.
     121<Dec/24 01:42 am>[1507863.902775] ata2.00: qc timeout (cmd 0xec)
     122<Dec/24 01:42 am>[1507863.907324] ata2.00: failed to IDENTIFY (I/O error, err_mask=0x4)
     123<Dec/24 01:42 am>[1507863.913897] ata2.00: revalidation failed (errno=-5)
     124<Dec/24 01:42 am>[1507863.919302] ata2.00: disabled
     125<Dec/24 01:42 am>[1507863.922742] ata2: hard resetting link
     126<Dec/24 01:42 am>[1507866.902771] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
     127<Dec/24 01:42 am>[1507866.909457] ata2: EH complete
     128<Dec/24 01:42 am>[1507866.912951] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     129<Dec/24 01:42 am>[1507866.922239] end_request: I/O error, dev sdb, sector 198065
     130<Dec/24 01:42 am>[1507866.928211] Read-error on swap-device (8:16:198073)
     131<Dec/24 01:42 am>[1507866.933557] Read-error on swap-device (8:16:198081)
     132
     133<Dec/24 01:42 am>[1507866.938886] Read-error on swap-device (8:16:198089)
     134<Dec/24 01:42 am>[1507866.944238] Read-error on swap-device (8:16:198097)
     135<Dec/24 01:42 am>[1507866.949586] Read-error on swap-device (8:16:198105)
     136<Dec/24 01:42 am>[1507866.954995] Read-error on swap-device (8:16:198113)
     137<Dec/24 01:42 am>[1507866.960403] Read-error on swap-device (8:16:198121)
     138<Dec/24 01:42 am>[1507866.965802] Read-error on swap-device (8:16:198129)
     139<Dec/24 01:42 am>[1507866.971227] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     140<Dec/24 01:42 am>[1507866.980626] end_request: I/O error, dev sdb, sector 96195185
     141<Dec/24 01:42 am>[1507866.986850] end_request: I/O error, dev sdb, sector 96195185
     142<Dec/24 01:42 am>[1507866.990174] md: super_written gets error=-5, uptodate=0
     143<Dec/24 01:42 am>[1507866.998796] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     144<Dec/24 01:42 am>[1507867.008133] end_request: I/O error, dev sdb, sector 96195185
     145<Dec/24 01:42 am>[1507867.014352] end_request: I/O error, dev sdb, sector 96195185
     146<Dec/24 01:42 am>[1507867.017673] md: super_written gets error=-5, uptodate=0
     147<Dec/24 01:42 am>[1507867.026397] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     148<Dec/24 01:42 am>[1507867.035706] end_request: I/O error, dev sdb, sector 73136425
     149<Dec/24 01:42 am>[1507867.041946] RAID1 conf printout:
     150<Dec/24 01:42 am>[1507867.042052] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     151<Dec/24 01:42 am>[1507867.042057] end_request: I/O error, dev sdb, sector 73136425
     152<Dec/24 01:42 am>[1507867.042085] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     153<Dec/24 01:42 am>[1507867.042089] end_request: I/O error, dev sdb, sector 73136425
     154<Dec/24 01:42 am>[1507867.042106] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     155
     156<Dec/24 01:42 am>[1507867.042109] end_request: I/O error, dev sdb, sector 73136425
     157<Dec/24 01:42 am>[1507867.042127] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     158<Dec/24 01:42 am>[1507867.042131] end_request: I/O error, dev sdb, sector 73136425
     159<Dec/24 01:42 am>[1507867.042147] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     160<Dec/24 01:42 am>[1507867.042150] end_request: I/O error, dev sdb, sector 73136425
     161<Dec/24 01:42 am>[1507867.042168] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     162<Dec/24 01:42 am>[1507867.042171] end_request: I/O error, dev sdb, sector 73136425
     163<Dec/24 01:42 am>[1507867.042187] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     164<Dec/24 01:42 am>[1507867.042191] end_request: I/O error, dev sdb, sector 73136425
     165<Dec/24 01:42 am>[1507867.042208] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     166<Dec/24 01:42 am>[1507867.042212] end_request: I/O error, dev sdb, sector 73136425
     167<Dec/24 01:42 am>[1507867.042228] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     168<Dec/24 01:42 am>[1507867.042231] end_request: I/O error, dev sdb, sector 73136425
     169<Dec/24 01:42 am>[1507867.042249] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     170<Dec/24 01:42 am>[1507867.042252] end_request: I/O error, dev sdb, sector 73136425
     171<Dec/24 01:42 am>[1507867.042268] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     172<Dec/24 01:42 am>[1507867.042271] end_request: I/O error, dev sdb, sector 73136425
     173<Dec/24 01:42 am>[1507867.042289] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     174<Dec/24 01:42 am>[1507867.042292] end_request: I/O error, dev sdb, sector 73136425
     175<Dec/24 01:42 am>[1507867.042308] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     176<Dec/24 01:42 am>[1507867.042311] end_request: I/O error, dev sdb, sector 73136425
     177<Dec/24 01:42 am>[1507867.042329] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     178
     179<Dec/24 01:42 am>[1507867.042332] end_request: I/O error, dev sdb, sector 73136425
     180<Dec/24 01:42 am>[1507867.042348] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     181<Dec/24 01:42 am>[1507867.042351] end_request: I/O error, dev sdb, sector 73136425
     182<Dec/24 01:42 am>[1507867.042369] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     183<Dec/24 01:42 am>[1507867.042372] end_request: I/O error, dev sdb, sector 73136425
     184<Dec/24 01:42 am>[1507867.042388] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     185<Dec/24 01:42 am>[1507867.042391] end_request: I/O error, dev sdb, sector 73136425
     186<Dec/24 01:42 am>[1507867.042409] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     187<Dec/24 01:42 am>[1507867.042412] end_request: I/O error, dev sdb, sector 73136425
     188<Dec/24 01:42 am>[1507867.042428] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     189<Dec/24 01:42 am>[1507867.042432] end_request: I/O error, dev sdb, sector 73136425
     190<Dec/24 01:42 am>[1507867.042449] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     191<Dec/24 01:42 am>[1507867.042452] end_request: I/O error, dev sdb, sector 73136425
     192<Dec/24 01:42 am>[1507867.042469] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     193<Dec/24 01:42 am>[1507867.042473] end_request: I/O error, dev sdb, sector 73136425
     194<Dec/24 01:42 am>[1507867.042491] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     195<Dec/24 01:42 am>[1507867.042494] end_request: I/O error, dev sdb, sector 73136425
     196<Dec/24 01:42 am>[1507867.042510] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     197<Dec/24 01:42 am>[1507867.042514] end_request: I/O error, dev sdb, sector 73136425
     198<Dec/24 01:42 am>[1507867.042531] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     199<Dec/24 01:42 am>[1507867.042535] end_request: I/O error, dev sdb, sector 73136425
     200<Dec/24 01:42 am>[1507867.042551] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     201
     202<Dec/24 01:42 am>[1507867.042554] end_request: I/O error, dev sdb, sector 73136425
     203<Dec/24 01:42 am>[1507867.042572] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     204<Dec/24 01:42 am>[1507867.042575] end_request: I/O error, dev sdb, sector 73136425
     205<Dec/24 01:42 am>[1507867.042591] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     206<Dec/24 01:42 am>[1507867.042595] end_request: I/O error, dev sdb, sector 73136425
     207<Dec/24 01:42 am>[1507867.042613] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     208<Dec/24 01:42 am>[1507867.042625] end_request: I/O error, dev sdb, sector 73136425
     209<Dec/24 01:42 am>[1507867.042643] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     210<Dec/24 01:42 am>[1507867.042646] end_request: I/O error, dev sdb, sector 73136425
     211<Dec/24 01:42 am>[1507867.042664] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     212<Dec/24 01:42 am>[1507867.042667] end_request: I/O error, dev sdb, sector 73136425
     213<Dec/24 01:42 am>[1507867.042684] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     214<Dec/24 01:42 am>[1507867.042687] end_request: I/O error, dev sdb, sector 73136425
     215<Dec/24 01:42 am>[1507867.042706] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     216<Dec/24 01:42 am>[1507867.042709] end_request: I/O error, dev sdb, sector 73136425
     217<Dec/24 01:42 am>[1507867.042726] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     218<Dec/24 01:42 am>[1507867.042729] end_request: I/O error, dev sdb, sector 73136425
     219<Dec/24 01:42 am>[1507867.042747] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     220<Dec/24 01:42 am>[1507867.042750] end_request: I/O error, dev sdb, sector 73136425
     221<Dec/24 01:42 am>[1507867.042766] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     222<Dec/24 01:42 am>[1507867.042769] end_request: I/O error, dev sdb, sector 73136425
     223<Dec/24 01:42 am>[1507867.042787] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     224
     225<Dec/24 01:42 am>[1507867.042790] end_request: I/O error, dev sdb, sector 73136425
     226<Dec/24 01:42 am>[1507867.042807] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     227<Dec/24 01:42 am>[1507867.042810] end_request: I/O error, dev sdb, sector 73136425
     228<Dec/24 01:42 am>[1507867.042828] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     229<Dec/24 01:42 am>[1507867.042831] end_request: I/O error, dev sdb, sector 73136425
     230<Dec/24 01:42 am>[1507867.042847] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     231<Dec/24 01:42 am>[1507867.042850] end_request: I/O error, dev sdb, sector 73136425
     232<Dec/24 01:42 am>[1507867.042868] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     233<Dec/24 01:42 am>[1507867.042871] end_request: I/O error, dev sdb, sector 73136425
     234<Dec/24 01:42 am>[1507867.042888] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     235<Dec/24 01:42 am>[1507867.042891] end_request: I/O error, dev sdb, sector 73136425
     236<Dec/24 01:42 am>[1507867.042909] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     237<Dec/24 01:42 am>[1507867.042912] end_request: I/O error, dev sdb, sector 73136425
     238<Dec/24 01:42 am>[1507867.042928] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     239<Dec/24 01:42 am>[1507867.042931] end_request: I/O error, dev sdb, sector 73136425
     240<Dec/24 01:42 am>[1507867.042949] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     241<Dec/24 01:42 am>[1507867.042952] end_request: I/O error, dev sdb, sector 73136425
     242<Dec/24 01:42 am>[1507867.042969] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     243<Dec/24 01:42 am>[1507867.042972] end_request: I/O error, dev sdb, sector 73136425
     244<Dec/24 01:42 am>[1507867.042990] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     245<Dec/24 01:42 am>[1507867.042993] end_request: I/O error, dev sdb, sector 73136425
     246<Dec/24 01:42 am>[1507867.043010] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     247
     248<Dec/24 01:42 am>[1507867.043013] end_request: I/O error, dev sdb, sector 73136425
     249<Dec/24 01:42 am>[1507867.043038] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     250<Dec/24 01:42 am>[1507867.043041] end_request: I/O error, dev sdb, sector 73136425
     251<Dec/24 01:42 am>[1507867.043058] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     252<Dec/24 01:42 am>[1507867.043061] end_request: I/O error, dev sdb, sector 73136425
     253<Dec/24 01:42 am>[1507867.043079] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     254<Dec/24 01:42 am>[1507867.043082] end_request: I/O error, dev sdb, sector 73136425
     255<Dec/24 01:42 am>[1507867.043098] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     256<Dec/24 01:42 am>[1507867.043102] end_request: I/O error, dev sdb, sector 73136425
     257<Dec/24 01:42 am>[1507867.043119] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     258<Dec/24 01:42 am>[1507867.043123] end_request: I/O error, dev sdb, sector 73136425
     259<Dec/24 01:42 am>[1507867.043139] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     260<Dec/24 01:42 am>[1507867.043142] end_request: I/O error, dev sdb, sector 73136425
     261<Dec/24 01:42 am>[1507867.043160] sd 1:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK
     262<Dec/24 01:42 am>[1507867.043163] end_request: I/O error, dev sdb, sector 73136425
     263<Dec/24 01:42 am>[1507867.886971]  --- wd:1 rd:2
     264<Dec/24 01:42 am>[1507867.890137]  disk 0, wo:1, o:0, dev:sda3
     265<Dec/24 01:42 am>[1507867.894539]  disk 1, wo:0, o:1, dev:sdb3
     266<Dec/24 01:42 am>[1507869.603310] BUG: soft lockup - CPU#11 stuck for 61s! [events/11:50]
     267<Dec/24 01:42 am>[1507869.603316] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     268<Dec/24 01:42 am>[1507869.603316] CPU 11:
     269<Dec/24 01:42 am>[1507869.603316] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     270
     271<Dec/24 01:42 am>[1507869.603316] Pid: 50, comm: events/11 Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     272<Dec/24 01:42 am>[1507869.603316] RIP: 0010:[<ffffffff8025bd5d>]  [<ffffffff8025bd5d>] smp_call_function_mask+0x163/0x1d7
     273<Dec/24 01:42 am>[1507869.603316] RSP: 0018:ffff88063ceb7d30  EFLAGS: 00000202
     274<Dec/24 01:42 am>[1507869.603316] RAX: 0000000000000020 RBX: 000000000000000b RCX: 0000000000000020
     275<Dec/24 01:42 am>[1507869.603316] RDX: ffffffff8081a300 RSI: 0000000000000020 RDI: 0000000000000000
     276<Dec/24 01:42 am>[1507869.603316] RBP: 000000000000000b R08: 0000000000000020 R09: ffff88063ceb7cf0
     277<Dec/24 01:42 am>[1507869.603316] R10: 0000000000000000 R11: ffffffff802230fe R12: 0000000000000001
     278<Dec/24 01:42 am>[1507869.603316] R13: ffffffff80257376 R14: 0000000000000069 R15: ffff88063ceac388
     279<Dec/24 01:42 am>[1507869.603316] FS:  0000000000000000(0000) GS:ffff880c3cc52900(0000) knlGS:0000000000000000
     280<Dec/24 01:42 am>[1507869.603316] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     281<Dec/24 01:42 am>[1507869.603316] CR2: 00007fff666100c0 CR3: 0000000000201000 CR4: 00000000000006e0
     282<Dec/24 01:42 am>[1507869.603316] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     283<Dec/24 01:42 am>[1507869.603316] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     284<Dec/24 01:42 am>[1507869.603316] Call Trace:
     285<Dec/24 01:42 am>[1507869.603316]  [<ffffffff80257376>] ? __lock_acquire+0x788/0x7f3
     286<Dec/24 01:42 am>[1507869.603316]  [<ffffffff8022ecd2>] ? update_curr+0x93/0xc9
     287<Dec/24 01:42 am>[1507869.603316]  [<ffffffff80218079>] ? mcheck_check_cpu+0x0/0x2f
     288<Dec/24 01:42 am>[1507869.603316]  [<ffffffff8023dcbb>] ? on_each_cpu+0x10/0x22
     289<Dec/24 01:42 am>[1507869.603316]  [<ffffffff80217997>] ? mcheck_timer+0x0/0x76
     290<Dec/24 01:42 am>[1507869.603316]  [<ffffffff802179ab>] ? mcheck_timer+0x14/0x76
     291<Dec/24 01:42 am>[1507869.603316]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
     292<Dec/24 01:42 am>[1507869.603316]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
     293
     294<Dec/24 01:42 am>[1507869.603316]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
     295<Dec/24 01:42 am>[1507869.603316]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
     296<Dec/24 01:42 am>[1507869.603316]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
     297<Dec/24 01:42 am>[1507869.603316]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     298<Dec/24 01:42 am>[1507869.603316]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     299<Dec/24 01:42 am>[1507869.603316]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     300<Dec/24 01:42 am>[1507869.603316]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     301<Dec/24 01:42 am>[1507869.603316]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     302<Dec/24 01:43 am>[1507935.103303] BUG: soft lockup - CPU#11 stuck for 61s! [events/11:50]
     303<Dec/24 01:43 am>[1507935.103310] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     304<Dec/24 01:43 am>[1507935.103310] CPU 11:
     305<Dec/24 01:43 am>[1507935.103310] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     306<Dec/24 01:43 am>[1507935.103310] Pid: 50, comm: events/11 Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     307<Dec/24 01:43 am>[1507935.103310] RIP: 0010:[<ffffffff8025bd5d>]  [<ffffffff8025bd5d>] smp_call_function_mask+0x163/0x1d7
     308<Dec/24 01:43 am>[1507935.103310] RSP: 0018:ffff88063ceb7d30  EFLAGS: 00000202
     309<Dec/24 01:43 am>[1507935.103310] RAX: 0000000000000020 RBX: 000000000000000b RCX: 0000000000000020
     310<Dec/24 01:43 am>[1507935.103310] RDX: ffffffff8081a300 RSI: 0000000000000020 RDI: 0000000000000000
     311<Dec/24 01:43 am>[1507935.103310] RBP: 000000000000000b R08: 0000000000000020 R09: ffff88063ceb7cf0
     312<Dec/24 01:43 am>[1507935.103310] R10: 0000000000000000 R11: ffffffff802230fe R12: 0000000000000001
     313<Dec/24 01:43 am>[1507935.103310] R13: ffffffff80257376 R14: 0000000000000069 R15: ffff88063ceac388
     314<Dec/24 01:43 am>[1507935.103310] FS:  0000000000000000(0000) GS:ffff880c3cc52900(0000) knlGS:0000000000000000
     315<Dec/24 01:43 am>[1507935.103310] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     316
     317<Dec/24 01:43 am>[1507935.103310] CR2: 00007fff666100c0 CR3: 0000000000201000 CR4: 00000000000006e0
     318<Dec/24 01:43 am>[1507935.103310] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     319<Dec/24 01:43 am>[1507935.103310] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     320<Dec/24 01:43 am>[1507935.103310] Call Trace:
     321<Dec/24 01:43 am>[1507935.103310]  [<ffffffff80257376>] ? __lock_acquire+0x788/0x7f3
     322<Dec/24 01:43 am>[1507935.103310]  [<ffffffff8022ecd2>] ? update_curr+0x93/0xc9
     323<Dec/24 01:43 am>[1507935.103310]  [<ffffffff80218079>] ? mcheck_check_cpu+0x0/0x2f
     324<Dec/24 01:43 am>[1507935.103310]  [<ffffffff8023dcbb>] ? on_each_cpu+0x10/0x22
     325<Dec/24 01:43 am>[1507935.103310]  [<ffffffff80217997>] ? mcheck_timer+0x0/0x76
     326<Dec/24 01:43 am>[1507935.103310]  [<ffffffff802179ab>] ? mcheck_timer+0x14/0x76
     327<Dec/24 01:43 am>[1507935.103310]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
     328<Dec/24 01:43 am>[1507935.103310]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
     329<Dec/24 01:43 am>[1507935.103310]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
     330<Dec/24 01:43 am>[1507935.103310]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
     331<Dec/24 01:43 am>[1507935.103310]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
     332<Dec/24 01:43 am>[1507935.103310]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     333<Dec/24 01:43 am>[1507935.103310]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     334<Dec/24 01:43 am>[1507935.103310]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     335<Dec/24 01:43 am>[1507935.103310]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     336<Dec/24 01:43 am>[1507935.103310]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     337<Dec/24 01:44 am>[1508000.599962] BUG: soft lockup - CPU#11 stuck for 61s! [events/11:50]
     338<Dec/24 01:44 am>[1508000.599971] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     339
     340<Dec/24 01:44 am>[1508000.599971] CPU 11:
     341<Dec/24 01:44 am>[1508000.599971] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     342<Dec/24 01:44 am>[1508000.599971] Pid: 50, comm: events/11 Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     343<Dec/24 01:44 am>[1508000.599971] RIP: 0010:[<ffffffff8025bd59>]  [<ffffffff8025bd59>] smp_call_function_mask+0x15f/0x1d7
     344<Dec/24 01:44 am>[1508000.599971] RSP: 0018:ffff88063ceb7d30  EFLAGS: 00000202
     345<Dec/24 01:44 am>[1508000.599971] RAX: 0000000000000020 RBX: 000000000000000b RCX: 0000000000000020
     346<Dec/24 01:44 am>[1508000.599971] RDX: ffffffff8081a300 RSI: 0000000000000020 RDI: 0000000000000000
     347<Dec/24 01:44 am>[1508000.599971] RBP: 000000000000000b R08: 0000000000000020 R09: ffff88063ceb7cf0
     348<Dec/24 01:44 am>[1508000.599971] R10: 0000000000000000 R11: ffffffff802230fe R12: 0000000000000001
     349<Dec/24 01:44 am>[1508000.599971] R13: ffffffff80257376 R14: 0000000000000069 R15: ffff88063ceac388
     350<Dec/24 01:44 am>[1508000.599971] FS:  0000000000000000(0000) GS:ffff880c3cc52900(0000) knlGS:0000000000000000
     351<Dec/24 01:44 am>[1508000.599971] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     352<Dec/24 01:44 am>[1508000.599971] CR2: 00007fff666100c0 CR3: 0000000000201000 CR4: 00000000000006e0
     353<Dec/24 01:44 am>[1508000.599971] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     354<Dec/24 01:44 am>[1508000.599971] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     355<Dec/24 01:44 am>[1508000.599971] Call Trace:
     356<Dec/24 01:44 am>[1508000.599971]  [<ffffffff80257376>] ? __lock_acquire+0x788/0x7f3
     357<Dec/24 01:44 am>[1508000.599971]  [<ffffffff8022ecd2>] ? update_curr+0x93/0xc9
     358<Dec/24 01:44 am>[1508000.599971]  [<ffffffff80218079>] ? mcheck_check_cpu+0x0/0x2f
     359<Dec/24 01:44 am>[1508000.599971]  [<ffffffff8023dcbb>] ? on_each_cpu+0x10/0x22
     360<Dec/24 01:44 am>[1508000.599971]  [<ffffffff80217997>] ? mcheck_timer+0x0/0x76
     361<Dec/24 01:44 am>[1508000.599971]  [<ffffffff802179ab>] ? mcheck_timer+0x14/0x76
     362
     363<Dec/24 01:44 am>[1508000.599971]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1<Dec/24 01:44 am>
     364                 <Dec/24 01:44 am>[1508000.599971]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
     365<Dec/24 01:44 am>[1508000.599971]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
     366<Dec/24 01:44 am>[1508000.599971]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
     367<Dec/24 01:44 am>[1508000.599971]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
     368<Dec/24 01:44 am>[1508000.599971]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     369<Dec/24 01:44 am>[1508000.599971]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     370<Dec/24 01:44 am>[1508000.599971]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     371<Dec/24 01:44 am>[1508000.599971]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     372<Dec/24 01:44 am>[1508000.599971]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     373<Dec/24 01:45 am>[1508066.099956] BUG: soft lockup - CPU#11 stuck for 61s! [events/11:50]
     374<Dec/24 01:45 am>[1508066.099964] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     375<Dec/24 01:45 am>[1508066.099964] CPU 11:
     376<Dec/24 01:45 am>[1508066.099964] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     377<Dec/24 01:45 am>[1508066.099964] Pid: 50, comm: events/11 Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     378<Dec/24 01:45 am>[1508066.099964] RIP: 0010:[<ffffffff8025bd5d>]  [<ffffffff8025bd5d>] smp_call_function_mask+0x163/0x1d7
     379<Dec/24 01:45 am>[1508066.099964] RSP: 0018:ffff88063ceb7d30  EFLAGS: 00000202
     380<Dec/24 01:45 am>[1508066.099964] RAX: 0000000000000020 RBX: 000000000000000b RCX: 0000000000000020
     381<Dec/24 01:45 am>[1508066.099964] RDX: ffffffff8081a300 RSI: 0000000000000020 RDI: 0000000000000000
     382<Dec/24 01:45 am>[1508066.099964] RBP: 000000000000000b R08: 0000000000000020 R09: ffff88063ceb7cf0
     383<Dec/24 01:45 am>[1508066.099964] R10: 0000000000000000 R11: ffffffff802230fe R12: 0000000000000001
     384<Dec/24 01:45 am>[1508066.099964] R13: ffffffff80257376 R14: 0000000000000069 R15: ffff88063ceac388
     385
     386<Dec/24 01:45 am>[1508066.099964] FS:  0000000000000000(0000) GS:ffff880c3cc52900(0000) knlGS:0000000000000000
     387<Dec/24 01:45 am>[1508066.099964] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     388<Dec/24 01:45 am>[1508066.099964] CR2: 00007fff666100c0 CR3: 0000000000201000 CR4: 00000000000006e0
     389<Dec/24 01:45 am>[1508066.099964] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     390<Dec/24 01:45 am>[1508066.099964] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     391<Dec/24 01:45 am>[1508066.099964] Call Trace:
     392<Dec/24 01:45 am>[1508066.099964]  [<ffffffff80257376>] ? __lock_acquire+0x788/0x7f3
     393<Dec/24 01:45 am>[1508066.099964]  [<ffffffff8022ecd2>] ? update_curr+0x93/0xc9
     394<Dec/24 01:45 am>[1508066.099964]  [<ffffffff80218079>] ? mcheck_check_cpu+0x0/0x2f
     395<Dec/24 01:45 am>[1508066.099964]  [<ffffffff8023dcbb>] ? on_each_cpu+0x10/0x22
     396<Dec/24 01:45 am>[1508066.099964]  [<ffffffff80217997>] ? mcheck_timer+0x0/0x76
     397<Dec/24 01:45 am>[1508066.099964]  [<ffffffff802179ab>] ? mcheck_timer+0x14/0x76
     398<Dec/24 01:45 am>[1508066.099964]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
     399<Dec/24 01:45 am>[1508066.099964]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
     400<Dec/24 01:45 am>[1508066.099964]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
     401<Dec/24 01:45 am>[1508066.099964]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
     402<Dec/24 01:45 am>[1508066.099964]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
     403<Dec/24 01:45 am>[1508066.099964]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     404<Dec/24 01:45 am>[1508066.099964]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     405<Dec/24 01:45 am>[1508066.099964]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     406<Dec/24 01:45 am>[1508066.099964]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     407<Dec/24 01:45 am>[1508066.099964]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     408
     409<Dec/24 01:46 am>[1508131.596616] BUG: soft lockup - CPU#11 stuck for 61s! [events/11:50]
     410<Dec/24 01:46 am>[1508131.596624] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     411<Dec/24 01:46 am>[1508131.596624] CPU 11:
     412<Dec/24 01:46 am>[1508131.596624] Modules linked in: w83793 hwmon_vid autofs4 e100 i2c_i801 i2c_core iTCO_wdt e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     413<Dec/24 01:46 am>[1508131.596624] Pid: 50, comm: events/11 Tainted: G      D    2.6.28-rc7-00105-gfeaf384 #4
     414<Dec/24 01:46 am>[1508131.596624] RIP: 0010:[<ffffffff8025bd5d>]  [<ffffffff8025bd5d>] smp_call_function_mask+0x163/0x1d7
     415<Dec/24 01:46 am>[1508131.596624] RSP: 0018:ffff88063ceb7d30  EFLAGS: 00000202
     416<Dec/24 01:46 am>[1508131.596624] RAX: 0000000000000020 RBX: 000000000000000b RCX: 0000000000000020
     417<Dec/24 01:46 am>[1508131.596624] RDX: ffffffff8081a300 RSI: 0000000000000020 RDI: 0000000000000000
     418<Dec/24 01:46 am>[1508131.596624] RBP: 000000000000000b R08: 0000000000000020 R09: ffff88063ceb7cf0
     419<Dec/24 01:46 am>[1508131.596624] R10: 0000000000000000 R11: ffffffff802230fe R12: 0000000000000001
     420<Dec/24 01:46 am>[1508131.596624] R13: ffffffff80257376 R14: 0000000000000069 R15: ffff88063ceac388
     421<Dec/24 01:46 am>[1508131.596624] FS:  0000000000000000(0000) GS:ffff880c3cc52900(0000) knlGS:0000000000000000
     422<Dec/24 01:46 am>[1508131.596624] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     423<Dec/24 01:46 am>[1508131.596624] CR2: 00007fff666100c0 CR3: 0000000000201000 CR4: 00000000000006e0
     424<Dec/24 01:46 am>[1508131.596624] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     425<Dec/24 01:46 am>[1508131.596624] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     426<Dec/24 01:46 am>[1508131.596624] Call Trace:
     427<Dec/24 01:46 am>[1508131.596624]  [<ffffffff80257376>] ? __lock_acquire+0x788/0x7f3
     428<Dec/24 01:46 am>[1508131.596624]  [<ffffffff8022ecd2>] ? update_curr+0x93/0xc9
     429<Dec/24 01:46 am>[1508131.596624]  [<ffffffff80218079>] ? mcheck_check_cpu+0x0/0x2f
     430<Dec/24 01:46 am>[1508131.596624]  [<ffffffff8023dcbb>] ? on_each_cpu+0x10/0x22
     431
     432<Dec/24 01:46 am>[1508131.596624]  [<ffffffff80217997>] ? mcheck_timer+0x0/0x76
     433<Dec/24 01:46 am>[1508131.596624]  [<ffffffff802179ab>] ? mcheck_timer+0x14/0x76
     434<Dec/24 01:46 am>[1508131.596624]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
     435<Dec/24 01:46 am>[1508131.596624]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
     436<Dec/24 01:46 am>[1508131.596624]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
     437<Dec/24 01:46 am>[1508131.596624]  [<ffffffff8024bfda>] ? autoremove_wake_functi<Dec/24 01:46 am>
     438                 <Dec/24 01:46 am>[1508131.596624]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
     439<Dec/24 01:46 am>[1508131.596624]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     440<Dec/24 01:46 am>[1508131.596624]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     441<Dec/24 01:46 am>[1508131.596624]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     442<Dec/24 01:46 am>[1508131.596624]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     443<Dec/24 01:46 am>[1508131.596624]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     444
     445File erased !
     446
     447}}}