IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changes between Initial Version and Version 1 of ipp005.20101030


Ignore:
Timestamp:
Oct 30, 2010, 6:22:07 AM (16 years ago)
Author:
bills
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • ipp005.20101030

    v1 v1  
     1Checked ganglia this morning. Lots of failed jobs. ipp005 high load. Ganglia shows that started around 11pm last night. Here are the contents of the console.
     2{{{
     3
     4<Oct/20 08:42 am>This is ipp005.ifa.hawaii.edu (Linux x86_64 2.6.28-rc7-00105-gfeaf384) 11:02:14
     5
     6<Oct/29 06:08 am>ipp005 login: [768588.516988]
     7<Oct/29 06:08 am>[768588.516991] =========================
     8<Oct/29 06:08 am>[768588.520874] [ BUG: held lock freed! ]
     9<Oct/29 06:08 am>[768588.520874] -------------------------
     10<Oct/29 06:08 am>[768588.520874] nfsd/7639 is freeing memory ffff8801a5d9c000-ffff8801a5d9c2b7, with a lock still held there!
     11<Oct/29 06:08 am>[768588.520874]  (&(&ip->i_lock)->mr_lock){....}, at: [<ffffffffa00c1bbb>] xfs_iget_core+0x3de/0x5ae [xfs]
     12
     13<Oct/29 06:08 am>[768588.520874] 2 locks held by nfsd/7639:
     14<Oct/29 06:08 am>[768588.520874]  #0:  (hash_sem){....}, at: [<ffffffff8034f2fd>] nfsd+0x1b7/0x296
     15<Oct/29 06:08 am>[768588.520874]  #1:  (&(&ip->i_lock)->mr_lock){....}, at: [<ffffffffa00c1bbb>] xfs_iget_core+0x3de/0x5ae [xfs]
     16<Oct/29 06:08 am>[768588.520874]
     17<Oct/29 06:08 am>[768588.520874] stack backtrace:
     18<Oct/29 06:08 am>[768588.520874] Pid: 7639, comm: nfsd Not tainted 2.6.28-rc7-00105-gfeaf384 #4
     19<Oct/29 06:08 am>[768588.520874] Call Trace:
     20                 <Oct/29 06:08 am>[768588.520874]  [<ffffffff80257acf>] debug_check_no_locks_freed+0xda/0x100
     21<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00c1d70>] xfs_iget_core+0x593/0x5ae [xfs]
     22<Oct/29 06:08 am>[768588.520874]  [<ffffffff802a5d85>] kmem_cache_free+0x6a/0xab
     23<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00c1d70>] xfs_iget_core+0x593/0x5ae [xfs]
     24<Oct/29 06:08 am>[768588.520874]  [<ffffffff805b7388>] _spin_unlock+0x17/0x20
     25<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00c1e6d>] xfs_iget+0xe2/0x189 [xfs]
     26<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00e0b78>] xfs_nfs_get_inode+0x39/0x88 [xfs]
     27<Oct/29 06:08 am>[768588.520874]  [<ffffffffa00e0c62>] xfs_fs_fh_to_dentry+0x69/0x76 [xfs]
     28<Oct/29 06:08 am>[768588.520874]  [<ffffffff8034e8ff>] exportfs_decode_fh+0x30/0x1cd
     29<Oct/29 06:08 am>[768588.520874]  [<ffffffff80351f21>] nfsd_acceptable+0x0/0xd3
     30<Oct/29 06:08 am>[768588.520874]  [<ffffffff803562ca>] exp_find_key+0x96/0xa9
     31<Oct/29 06:08 am>[768588.520874]  [<ffffffff8050ac9a>] sock_recvmsg+0xd5/0xed
     32<Oct/29 06:08 am>[768588.520874]  [<ffffffff8022eda5>] place_entity+0x52/0x75
     33<Oct/29 06:08 am>[768588.520874]  [<ffffffff80356346>] exp_find+0x69/0x74
     34<Oct/29 06:08 am>[768588.520874]  [<ffffffff803517a4>] fh_verify+0x269/0x54d
     35
     36<Oct/29 06:08 am>[768588.681209]  [<ffffffff80353053>] nfsd_open+0x1f/0x17b
     37<Oct/29 06:08 am>[768588.681209]  [<ffffffff80353530>] nfsd_read+0x7f/0xc4
     38                 <Oct/29 06:08 am>[768588.681209]  [<ffffffff8035a34b>] nfsd3_proc_read+0xfe/0x141
     39<Oct/29 06:08 am>[768588.681209]  [<ffffffff8034ecd8>] nfsd_dispatch+0xde/0x1c2
     40<Oct/29 06:08 am>[768588.681209]  [<ffffffff8058febb>] svc_process+0x408/0x6e9
     41<Oct/29 06:08 am>[768588.681209]  [<ffffffff805b70a2>] __down_read+0x34/0x9e
     42<Oct/29 06:08 am>[768588.681209]  [<ffffffff8034f305>] nfsd+0x1bf/0x296
     43<Oct/29 06:08 am>[768588.681209]  [<ffffffff8034f146>] nfsd+0x0/0x296
     44<Oct/29 06:08 am>[768588.681209]  [<ffffffff8024becc>] kthread+0x47/0x73
     45<Oct/29 06:08 am>[768588.681209]  [<ffffffff80234852>] schedule_tail+0x27/0x5f
     46<Oct/29 06:08 am>[768588.681209]  [<ffffffff8020cd19>] child_rip+0xa/0x11
     47<Oct/29 06:08 am>[768588.681209]  [<ffffffff8024be85>] kthread+0x0/0x73
     48<Oct/29 06:08 am>[768588.681209]  [<ffffffff8020cd0f>] child_rip+0x0/0x11
     49<Oct/29 08:42 am>[777846.860683] eth0: too many iterations (61) in nv_nic_irq.
     50<Oct/29 08:49 am>[778280.004590] eth0: too many iterations (61) in nv_nic_irq.
     51<Oct/29 10:18 am>[783590.637909] eth0: too many iterations (61) in nv_nic_irq.
     52<Oct/29 10:50 am>[785500.050813] eth0: too many iterations (61) in nv_nic_irq.
     53<Oct/29 11:53 am>[789307.308652] eth0: too many iterations (61) in nv_nic_irq.
     54<Oct/29 12:04 pm>[789975.432215] eth0: too many iterations (61) in nv_nic_irq.
     55<Oct/29 12:18 pm>[790831.449968] eth0: too many iterations (61) in nv_nic_irq.
     56<Oct/29 12:18 pm>[790835.238038] eth0: too many iterations (61) in nv_nic_irq.
     57<Oct/29 02:20 pm>[798133.388035] eth0: too many iterations (61) in nv_nic_irq.
     58
     59<Oct/29 03:42 pm>[803028.043815] eth0: too many iterations (61) in nv_nic_irq.
     60<Oct/29 04:21 pm>[805413.358445] eth0: too many iterations (61) in nv_nic_irq.
     61<Oct/29 04:25 pm>[805631.004822] eth0: too many iterations (61) in nv_nic_irq.
     62<Oct/29 04:45 pm>[806813.000336] eth0: too many iterations (61) in nv_nic_irq.
     63<Oct/29 05:12 pm>[808422.702304] eth0: too many iterations (61) in nv_nic_irq.
     64<Oct/29 08:31 pm>[820402.099219] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
     65<Oct/29 08:31 pm>[820402.099219] IP: [<ffffffff8022dfe5>] __wake_up_common+0x28/0x74
     66<Oct/29 08:31 pm>[820402.099219] PGD 1c7d97067 PUD 1c7d50067 PMD 0
     67<Oct/29 08:31 pm>[820402.099219] Oops: 0000 [#1] SMP
     68<Oct/29 08:31 pm>[820402.099219] last sysfs file: /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
     69<Oct/29 08:31 pm>[820402.099219] CPU 3
     70<Oct/29 08:31 pm>[820402.099219] Modules linked in: w83627hf hwmon_vid autofs4 smsc37b787_wdt i2c_nforce2 k8temp i2c_core forcedeth tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
     71<Oct/29 08:31 pm>[820402.099219] Pid: 3168, comm: xfslogd/3 Not tainted 2.6.28-rc7-00105-gfeaf384 #4
     72<Oct/29 08:31 pm>[820402.099219] RIP: 0010:[<ffffffff8022dfe5>]  [<ffffffff8022dfe5>] __wake_up_common+0x28/0x74
     73<Oct/29 08:31 pm>[820402.099219] RSP: 0018:ffff88012e06bce0  EFLAGS: 00010097
     74<Oct/29 08:31 pm>[820402.099219] RAX: 0000000000000286 RBX: ffff880161aa7858 RCX: 0000000000000000
     75<Oct/29 08:31 pm>[820402.099219] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880161aa7858
     76<Oct/29 08:31 pm>[820402.099219] RBP: ffff88012e06bd10 R08: ffffffffffffffe8 R09: 0000000000000001
     77<Oct/29 08:31 pm>[820402.099219] R10: 0000000000000000 R11: ffffffff8039d94e R12: 0000000000000000
     78<Oct/29 08:31 pm>[820402.099219] R13: 0000000000000000 R14: ffff880161aa7858 R15: 0000000000000000
     79<Oct/29 08:31 pm>[820402.099219] FS:  0000000040eeb950(0000) GS:ffff88022fa0d780(0000) knlGS:0000000000000000
     80<Oct/29 08:31 pm>[820402.099219] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
     81
     82<Oct/29 08:31 pm>[820402.099219] CR2: 0000000000000000 CR3: 00000001c7d5e000 CR4: 00000000000006e0
     83<Oct/29 08:31 pm>[820402.099219] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
     84<Oct/29 08:31 pm>[820402.099219] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
     85<Oct/29 08:31 pm>[820402.099219] Process xfslogd/3 (pid: 3168, threadinfo ffff88012e06a000, task ffff88012e1b0f30)
     86<Oct/29 08:31 pm>[820402.099219] Stack:
     87<Oct/29 08:31 pm>[820402.099219]  0000000300000000 ffff880161aa7858 0000000000000000 0000000000000000
     88<Oct/29 08:31 pm>[820402.099219]  0000000000000286 0000000000000003 ffff88012e06bd50 ffffffff8022f03e
     89<Oct/29 08:31 pm>[820402.099219]  ffffffffffffff6e ffff880016af5001 0000000000000000 ffff880161aa7800
     90<Oct/29 08:31 pm>[820402.099219] Call Trace:
     91<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8022f03e>] __wake_up+0x38/0x4e
     92<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00b042f>] xfs_buf_item_unpin+0x25/0xa4 [xfs]
     93<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00d50d3>] ? xfs_trans_chunk_committed+0xba/0xf0 [xfs]
     94<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00d514d>] ? xfs_trans_committed+0x44/0xd2 [xfs]
     95<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00ca407>] ? xlog_state_do_callback+0x169/0x281 [xfs]
     96<Oct/29 08:31 pm>[820402.099219]  [<ffffffffa00e079c>] ? xfs_buf_iodone_work+0x0/0x74 [xfs]
     97<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
     98<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
     99<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
     100<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
     101<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
     102<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8024becc>] ? kthread+0x47/0x73
     103<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
     104
     105<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
     106<Oct/29 08:31 pm>[820402.099219]  [<ffffffff80212496>] ? c1e_idle+0x0/0xe9
     107<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8024be85>] ? kthread+0x0/0x73
     108<Oct/29 08:31 pm>[820402.099219]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
     109<Oct/29 08:31 pm>[820402.099219] Code: e5 c9 c3 55 48 89 e5 41 57 4d 89 c7 41 56 49 89 fe 41 55 41 54 41 89 d4 53 48 83 ec 08 89 75 d4 89 4d d0 4c 8b 47 30 49 83 e8 18 <49> 8b 58 18 eb 29 45 8b 28 4c 89 f9 8b 55 d0 8b 75 d4 4c 89 c7
     110<Oct/29 08:31 pm>[820402.099219] RIP  [<ffffffff8022dfe5>] __wake_up_common+0x28/0x74
     111<Oct/29 08:31 pm>[820402.099219]  RSP <ffff88012e06bce0>
     112<Oct/29 08:31 pm>[820402.099219] CR2: 0000000000000000
     113<Oct/29 08:31 pm>[820402.099219] ---[ end trace 06a2b6b5cc51d31c ]---
     114
     115File erased !
     116
     117
     118}}}