{{{ Sep 27 06:27:03 ipp012 [329882.488381] INFO: task nfsd:7633 blocked for more than 120 seconds. Sep 27 06:27:03 ipp012 [329882.488389] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 27 06:27:03 ipp012 [329882.488396] nfsd D ffff880129dc8000 0 7633 2 0x00000000 Sep 27 06:27:03 ipp012 [329882.488409] ffff88012941b930 0000000000000046 00000000000001cb ffffffff81a891f0 Sep 27 06:27:03 ipp012 [329882.488421] ffff88012941b8d0 0000000000004000 0000000000013f00 000000000000e078 Sep 27 06:27:03 ipp012 [329882.488431] 0000000000000000 0000000000000000 ffff880129dc8000 ffff88012a603090 Sep 27 06:27:03 ipp012 [329882.488442] Call Trace: Sep 27 06:27:03 ipp012 [329882.488465] [] ? generic_file_aio_write+0x52/0xc5 Sep 27 06:27:03 ipp012 [329882.488478] [] mutex_lock_nested+0x14e/0x253 Sep 27 06:27:03 ipp012 [329882.488487] [] ? generic_file_aio_write+0x52/0xc5 Sep 27 06:27:26 ipp012 [329882.488497] [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:26 ipp012 [329882.488509] [] ext3_file_write+0x1e/0x9e Sep 27 06:27:26 ipp012 [329882.488517] [] ? ext3_file_write+0x0/0x9e Sep 27 06:27:26 ipp012 [329882.488529] [] do_sync_readv_writev+0xe3/0x12b Sep 27 06:27:26 ipp012 [329882.488542] [] ? autoremove_wake_function+0x0/0x38 Sep 27 06:27:26 ipp012 [329882.488554] [] ? exportfs_decode_fh+0xca/0x20f Sep 27 06:27:26 ipp012 [329882.488562] [] ? rw_copy_check_uvector+0x4d/0xe5 Sep 27 06:27:26 ipp012 [329882.488574] [] ? security_file_permission+0x11/0x13 Sep 27 06:27:26 ipp012 [329882.488583] [] do_readv_writev+0xb8/0x1a0 Sep 27 06:27:26 ipp012 [329882.488592] [] ? file_move+0x4b/0x50 Sep 27 06:27:26 ipp012 [329882.488602] [] vfs_writev+0x3e/0x49 Sep 27 06:27:26 ipp012 [329882.488611] [] nfsd_vfs_write+0x14b/0x384 Sep 27 06:27:26 ipp012 [329882.488620] [] ? dentry_open+0x83/0x8c Sep 27 06:27:26 ipp012 [329882.488628] [] ? nfsd_open+0x195/0x1b8 Sep 27 06:27:26 ipp012 [329882.488636] [] nfsd_write+0xbd/0xd9 Sep 27 06:27:26 ipp012 [329882.488646] [] nfsd3_proc_write+0xf3/0x110 Sep 27 06:27:26 ipp012 [329882.488655] [] nfsd_dispatch+0xfd/0x1de Sep 27 06:27:26 ipp012 [329882.488665] [] svc_process+0x425/0x72a Sep 27 06:27:26 ipp012 [329882.488673] [] ? nfsd+0x0/0x14b Sep 27 06:27:26 ipp012 [329882.488680] [] nfsd+0x105/0x14b Sep 27 06:27:26 ipp012 [329882.488689] [] kthread+0x8f/0x97 Sep 27 06:27:26 ipp012 [329882.488698] [] child_rip+0xa/0x20 Sep 27 06:27:26 ipp012 [329882.488709] [] ? __atomic_notifier_call_chain+0x0/0x87 Sep 27 06:27:26 ipp012 [329882.488718] [] ? kthread+0x0/0x97 Sep 27 06:27:26 ipp012 [329882.488725] [] ? child_rip+0x0/0x20 Sep 27 06:27:26 ipp012 [329882.488732] 2 locks held by nfsd/7633: Sep 27 06:27:26 ipp012 [329882.488736] #0: (hash_sem){......}, at: [] exp_readlock+0x10/0x12 Sep 27 06:27:26 ipp012 [329882.488761] #1: (&sb->s_type->i_mutex_key#6){......}, at: [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:26 ipp012 [329882.488780] INFO: task nfsd:7636 blocked for more than 120 seconds. Sep 27 06:27:26 ipp012 [329882.488785] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 27 06:27:26 ipp012 [329882.488790] nfsd D 0000000000000000 0 7636 2 0x00000000 Sep 27 06:27:26 ipp012 [329882.488800] ffff88012a8ad930 0000000000000046 ffff88012a8ad8f8 ffff88012a8ad8f4 Sep 27 06:27:26 ipp012 [329882.488811] ffff88012a8ad8d0 0000000000004000 0000000000013f00 000000000000e078 Sep 27 06:27:26 ipp012 [329882.488821] 0000000000000000 0000000000000000 ffff880129dce120 ffff88012babd0f0 Sep 27 06:27:26 ipp012 [329882.488831] Call Trace: Sep 27 06:27:26 ipp012 [329882.488839] [] mutex_lock_nested+0x14e/0x253 Sep 27 06:27:26 ipp012 [329882.488848] [] ? generic_file_aio_write+0x52/0xc5 Sep 27 06:27:26 ipp012 [329882.488858] [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:26 ipp012 [329882.488867] [] ext3_file_write+0x1e/0x9e Sep 27 06:27:26 ipp012 [329882.488875] [] ? ext3_file_write+0x0/0x9e Sep 27 06:27:26 ipp012 [329882.488884] [] do_sync_readv_writev+0xe3/0x12b Sep 27 06:27:26 ipp012 [329882.488893] [] ? autoremove_wake_function+0x0/0x38 Sep 27 06:27:26 ipp012 [329882.488902] [] ? exportfs_decode_fh+0xca/0x20f Sep 27 06:27:26 ipp012 [329882.488911] [] ? rw_copy_check_uvector+0x4d/0xe5 Sep 27 06:27:26 ipp012 [329882.488920] [] ? security_file_permission+0x11/0x13 Sep 27 06:27:26 ipp012 [329882.488929] [] do_readv_writev+0xb8/0x1a0 Sep 27 06:27:26 ipp012 [329882.488938] [] ? file_move+0x4b/0x50 Sep 27 06:27:26 ipp012 [329882.488947] [] vfs_writev+0x3e/0x49 Sep 27 06:27:26 ipp012 [329882.488955] [] nfsd_vfs_write+0x14b/0x384 Sep 27 06:27:26 ipp012 [329882.488963] [] ? dentry_open+0x83/0x8c Sep 27 06:27:26 ipp012 [329882.488972] [] ? nfsd_open+0x195/0x1b8 Sep 27 06:27:26 ipp012 [329882.488980] [] nfsd_write+0xbd/0xd9 Sep 27 06:27:26 ipp012 [329882.488988] [] nfsd3_proc_write+0xf3/0x110 Sep 27 06:27:26 ipp012 [329882.488997] [] nfsd_dispatch+0xfd/0x1de Sep 27 06:27:26 ipp012 [329882.489004] [] svc_process+0x425/0x72a Sep 27 06:27:26 ipp012 [329882.489012] [] ? nfsd+0x0/0x14b Sep 27 06:27:26 ipp012 [329882.489020] [] nfsd+0x105/0x14b Sep 27 06:27:26 ipp012 [329882.489028] [] kthread+0x8f/0x97 Sep 27 06:27:26 ipp012 [329882.489036] [] child_rip+0xa/0x20 Sep 27 06:27:26 ipp012 [329882.489045] [] ? __atomic_notifier_call_chain+0x0/0x87 Sep 27 06:27:26 ipp012 [329882.489054] [] ? kthread+0x0/0x97 Sep 27 06:27:26 ipp012 [329882.489061] [] ? child_rip+0x0/0x20 Sep 27 06:27:26 ipp012 [329882.489066] 2 locks held by nfsd/7636: Sep 27 06:27:26 ipp012 [329882.489070] #0: (hash_sem){......}, at: [] exp_readlock+0x10/0x12 Sep 27 06:27:26 ipp012 [329882.489082] #1: (&sb->s_type->i_mutex_key#6){......}, at: [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:26 ipp012 [329882.489099] INFO: task nfsd:7637 blocked for more than 120 seconds. Sep 27 06:27:26 ipp012 [329882.489104] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 27 06:27:26 ipp012 [329882.489109] nfsd D ffff880129dc9030 0 7637 2 0x00000000 Sep 27 06:27:26 ipp012 [329882.489118] ffff88012a8afcb0 0000000000000046 00000000000001cb ffffffff81a891f0 Sep 27 06:27:26 ipp012 [329882.489129] ffff88012a8afc50 0000000000004000 0000000000013f00 000000000000e078 Sep 27 06:27:26 ipp012 [329882.489138] 0000000000000000 0000000000000000 ffff880129dc9030 ffff88012a601030 Sep 27 06:27:26 ipp012 [329882.489148] Call Trace: Sep 27 06:27:26 ipp012 [329882.489157] [] ? nfsd_sync+0x4c/0xb0 Sep 27 06:27:26 ipp012 [329882.489166] [] ? file_move+0x1e/0x50 Sep 27 06:27:26 ipp012 [329882.489174] [] mutex_lock_nested+0x14e/0x253 Sep 27 06:27:26 ipp012 [329882.489182] [] ? nfsd_sync+0x4c/0xb0 Sep 27 06:27:26 ipp012 [329882.489191] [] nfsd_sync+0x4c/0xb0 Sep 27 06:27:26 ipp012 [329882.489199] [] nfsd_commit+0x5b/0x7e Sep 27 06:27:26 ipp012 [329882.489207] [] nfsd3_proc_commit+0xb5/0xc4 Sep 27 06:27:26 ipp012 [329882.489215] [] nfsd_dispatch+0xfd/0x1de Sep 27 06:27:26 ipp012 [329882.489223] [] svc_process+0x425/0x72a Sep 27 06:27:26 ipp012 [329882.489231] [] ? nfsd+0x0/0x14b Sep 27 06:27:26 ipp012 [329882.489239] [] nfsd+0x105/0x14b Sep 27 06:27:26 ipp012 [329882.489247] [] kthread+0x8f/0x97 Sep 27 06:27:26 ipp012 [329882.489255] [] child_rip+0xa/0x20 Sep 27 06:27:26 ipp012 [329882.489264] [] ? __atomic_notifier_call_chain+0x0/0x87 Sep 27 06:27:26 ipp012 [329882.489273] [] ? kthread+0x0/0x97 Sep 27 06:27:26 ipp012 [329882.489280] [] ? child_rip+0x0/0x20 Sep 27 06:27:26 ipp012 [329882.489285] 2 locks held by nfsd/7637: Sep 27 06:27:26 ipp012 [329882.489289] #0: (hash_sem){......}, at: [] exp_readlock+0x10/0x12 Sep 27 06:27:26 ipp012 [329882.489301] #1: (&sb->s_type->i_mutex_key#6){......}, at: [] nfsd_sync+0x4c/0xb0 Sep 27 06:27:26 ipp012 [329882.489321] INFO: task nfsd:7645 blocked for more than 120 seconds. Sep 27 06:27:26 ipp012 [329882.489326] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 27 06:27:26 ipp012 [329882.489331] nfsd D 0000000000000000 0 7645 2 0x00000000 Sep 27 06:27:26 ipp012 [329882.489340] ffff880129c43930 0000000000000046 ffff880129c438f8 ffff880129c438f4 Sep 27 06:27:26 ipp012 [329882.489350] ffff880129c438d0 0000000000004000 0000000000013f00 000000000000e078 Sep 27 06:27:26 ipp012 [329882.489360] 0000000000000000 0000000000000000 ffff88012a6050f0 ffff8801296bb090 Sep 27 06:27:26 ipp012 [329882.489370] Call Trace: Sep 27 06:27:26 ipp012 [329882.489378] [] mutex_lock_nested+0x14e/0x253 Sep 27 06:27:26 ipp012 [329882.489387] [] ? generic_file_aio_write+0x52/0xc5 Sep 27 06:27:26 ipp012 [329882.489397] [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:26 ipp012 [329882.489406] [] ext3_file_write+0x1e/0x9e Sep 27 06:27:26 ipp012 [329882.489414] [] ? ext3_file_write+0x0/0x9e Sep 27 06:27:26 ipp012 [329882.489422] [] do_sync_readv_writev+0xe3/0x12b Sep 27 06:27:26 ipp012 [329882.489434] [] ? get_partial_node+0x1b/0x8a Sep 27 06:27:26 ipp012 [329882.489444] [] ? autoremove_wake_function+0x0/0x38 Sep 27 06:27:26 ipp012 [329882.489453] [] ? rw_copy_check_uvector+0x4d/0xe5 Sep 27 06:27:26 ipp012 [329882.489462] [] ? security_file_permission+0x11/0x13 Sep 27 06:27:26 ipp012 [329882.489471] [] do_readv_writev+0xb8/0x1a0 Sep 27 06:27:26 ipp012 [329882.489479] [] ? file_move+0x4b/0x50 Sep 27 06:27:26 ipp012 [329882.489488] [] vfs_writev+0x3e/0x49 Sep 27 06:27:26 ipp012 [329882.489497] [] nfsd_vfs_write+0x14b/0x384 Sep 27 06:27:26 ipp012 [329882.489505] [] ? dentry_open+0x83/0x8c Sep 27 06:27:26 ipp012 [329882.489514] [] ? nfsd_open+0x195/0x1b8 Sep 27 06:27:26 ipp012 [329882.489522] [] nfsd_write+0xbd/0xd9 Sep 27 06:27:26 ipp012 [329882.489530] [] nfsd3_proc_write+0xf3/0x110 Sep 27 06:27:26 ipp012 [329882.489539] [] nfsd_dispatch+0xfd/0x1de Sep 27 06:27:26 ipp012 [329882.489546] [] svc_process+0x425/0x72a Sep 27 06:27:26 ipp012 [329882.489555] [] ? nfsd+0x0/0x14b Sep 27 06:27:26 ipp012 [329882.489562] [] nfsd+0x105/0x14b Sep 27 06:27:26 ipp012 [329882.489570] [] kthread+0x8f/0x97 Sep 27 06:27:26 ipp012 [329882.489578] [] child_rip+0xa/0x20 Sep 27 06:27:26 ipp012 [329882.489587] [] ? __atomic_notifier_call_chain+0x0/0x87 Sep 27 06:27:26 ipp012 [329882.489596] [] ? kthread+0x0/0x97 Sep 27 06:27:26 ipp012 [329882.489603] [] ? child_rip+0x0/0x20 Sep 27 06:27:26 ipp012 [329882.489608] 2 locks held by nfsd/7645: Sep 27 06:27:26 ipp012 [329882.489612] #0: (hash_sem){......}, at: [] exp_readlock+0x10/0x12 Sep 27 06:27:26 ipp012 [329882.489624] #1: (&sb->s_type->i_mutex_key#6){......}, at: [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:26 ipp012 [329882.489641] INFO: task nfsd:7646 blocked for more than 120 seconds. Sep 27 06:27:26 ipp012 [329882.489645] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 27 06:27:26 ipp012 [329882.489651] nfsd D ffff88012a601030 0 7646 2 0x00000000 Sep 27 06:27:26 ipp012 [329882.489660] ffff88012a8bd930 0000000000000046 0000000000000000 ffffffff81a891f0 Sep 27 06:27:26 ipp012 [329882.489670] ffff88012a8bd8d0 0000000000004000 0000000000013f00 000000000000e078 Sep 27 06:27:26 ipp012 [329882.489680] 0000000000000000 0000000000000000 ffff88012a601030 ffff88012bad2060 Sep 27 06:27:26 ipp012 [329882.489690] Call Trace: Sep 27 06:27:26 ipp012 [329882.489698] [] mutex_lock_nested+0x14e/0x253 Sep 27 06:27:26 ipp012 [329882.489707] [] ? generic_file_aio_write+0x52/0xc5 Sep 27 06:27:26 ipp012 [329882.489717] [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:26 ipp012 [329882.489725] [] ext3_file_write+0x1e/0x9e Sep 27 06:27:26 ipp012 [329882.489733] [] ? ext3_file_write+0x0/0x9e Sep 27 06:27:26 ipp012 [329882.489742] [] do_sync_readv_writev+0xe3/0x12b Sep 27 06:27:26 ipp012 [329882.489752] [] ? autoremove_wake_function+0x0/0x38 Sep 27 06:27:26 ipp012 [329882.489761] [] ? rw_copy_check_uvector+0x4d/0xe5 Sep 27 06:27:26 ipp012 [329882.489770] [] ? security_file_permission+0x11/0x13 Sep 27 06:27:26 ipp012 [329882.489779] [] do_readv_writev+0xb8/0x1a0 Sep 27 06:27:26 ipp012 [329882.489787] [] ? file_move+0x4b/0x50 Sep 27 06:27:26 ipp012 [329882.489796] [] vfs_writev+0x3e/0x49 Sep 27 06:27:26 ipp012 [329882.489805] [] nfsd_vfs_write+0x14b/0x384 Sep 27 06:27:26 ipp012 [329882.489813] [] ? dentry_open+0x83/0x8c Sep 27 06:27:26 ipp012 [329882.489821] [] ? nfsd_open+0x195/0x1b8 Sep 27 06:27:26 ipp012 [329882.489830] [] nfsd_write+0xbd/0xd9 Sep 27 06:27:26 ipp012 [329882.489838] [] nfsd3_proc_write+0xf3/0x110 Sep 27 06:27:26 ipp012 [329882.489846] [] nfsd_dispatch+0xfd/0x1de Sep 27 06:27:26 ipp012 [329882.489854] [] svc_process+0x425/0x72a Sep 27 06:27:26 ipp012 [329882.489862] [] ? nfsd+0x0/0x14b Sep 27 06:27:26 ipp012 [329882.489870] [] nfsd+0x105/0x14b Sep 27 06:27:26 ipp012 [329882.489878] [] kthread+0x8f/0x97 Sep 27 06:27:26 ipp012 [329882.489886] [] child_rip+0xa/0x20 Sep 27 06:27:26 ipp012 [329882.489895] [] ? __atomic_notifier_call_chain+0x0/0x87 Sep 27 06:27:26 ipp012 [329882.489903] [] ? kthread+0x0/0x97 Sep 27 06:27:26 ipp012 [329882.489910] [] ? child_rip+0x0/0x20 Sep 27 06:27:26 ipp012 [329882.489916] 2 locks held by nfsd/7646: Sep 27 06:27:26 ipp012 [329882.489920] #0: (hash_sem){......}, at: [] exp_readlock+0x10/0x12 Sep 27 06:27:26 ipp012 [329882.489931] #1: (&sb->s_type->i_mutex_key#6){......}, at: [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:26 ipp012 [329882.489953] INFO: task nfsd:7659 blocked for more than 120 seconds. Sep 27 06:27:26 ipp012 [329882.489958] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 27 06:27:27 ipp012 [329882.489963] nfsd D ffff8801294250f0 0 7659 2 0x00000000 Sep 27 06:27:27 ipp012 [329882.489973] ffff880129c83930 0000000000000046 0000000000000000 ffffffff81a891f0 Sep 27 06:27:27 ipp012 [329882.489983] ffff880129c838d0 0000000000004000 0000000000013f00 000000000000e078 Sep 27 06:27:27 ipp012 [329882.489992] 0000000000000000 0000000000000000 ffff8801294250f0 ffff88012bab9030 Sep 27 06:27:27 ipp012 [329882.490002] Call Trace: Sep 27 06:27:27 ipp012 [329882.490011] [] mutex_lock_nested+0x14e/0x253 Sep 27 06:27:27 ipp012 [329882.490020] [] ? generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.490029] [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.490038] [] ext3_file_write+0x1e/0x9e Sep 27 06:27:27 ipp012 [329882.490046] [] ? ext3_file_write+0x0/0x9e Sep 27 06:27:27 ipp012 [329882.490055] [] do_sync_readv_writev+0xe3/0x12b Sep 27 06:27:27 ipp012 [329882.490064] [] ? get_partial_node+0x1b/0x8a Sep 27 06:27:27 ipp012 [329882.490073] [] ? autoremove_wake_function+0x0/0x38 Sep 27 06:27:27 ipp012 [329882.490082] [] ? rw_copy_check_uvector+0x4d/0xe5 Sep 27 06:27:27 ipp012 [329882.490091] [] ? security_file_permission+0x11/0x13 Sep 27 06:27:27 ipp012 [329882.490100] [] do_readv_writev+0xb8/0x1a0 Sep 27 06:27:27 ipp012 [329882.490109] [] ? file_move+0x4b/0x50 Sep 27 06:27:27 ipp012 [329882.490118] [] vfs_writev+0x3e/0x49 Sep 27 06:27:27 ipp012 [329882.490126] [] nfsd_vfs_write+0x14b/0x384 Sep 27 06:27:27 ipp012 [329882.490135] [] ? dentry_open+0x83/0x8c Sep 27 06:27:27 ipp012 [329882.490143] [] ? nfsd_open+0x195/0x1b8 Sep 27 06:27:27 ipp012 [329882.490152] [] nfsd_write+0xbd/0xd9 Sep 27 06:27:27 ipp012 [329882.490160] [] nfsd3_proc_write+0xf3/0x110 Sep 27 06:27:27 ipp012 [329882.490168] [] nfsd_dispatch+0xfd/0x1de Sep 27 06:27:27 ipp012 [329882.490176] [] svc_process+0x425/0x72a Sep 27 06:27:27 ipp012 [329882.490184] [] ? nfsd+0x0/0x14b Sep 27 06:27:27 ipp012 [329882.490191] [] nfsd+0x105/0x14b Sep 27 06:27:27 ipp012 [329882.490199] [] kthread+0x8f/0x97 Sep 27 06:27:27 ipp012 [329882.490207] [] child_rip+0xa/0x20 Sep 27 06:27:27 ipp012 [329882.490216] [] ? kthread+0x0/0x97 Sep 27 06:27:27 ipp012 [329882.490223] [] ? child_rip+0x0/0x20 Sep 27 06:27:27 ipp012 [329882.490229] 2 locks held by nfsd/7659: Sep 27 06:27:27 ipp012 [329882.490232] #0: (hash_sem){......}, at: [] exp_readlock+0x10/0x12 Sep 27 06:27:27 ipp012 [329882.490244] #1: (&sb->s_type->i_mutex_key#6){......}, at: [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.490262] INFO: task nfsd:7661 blocked for more than 120 seconds. Sep 27 06:27:27 ipp012 [329882.490266] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 27 06:27:27 ipp012 [329882.490272] nfsd D ffff88012a440000 0 7661 2 0x00000000 Sep 27 06:27:27 ipp012 [329882.490281] ffff88012a58f930 0000000000000046 0000000000000000 ffffffff81a891f0 Sep 27 06:27:27 ipp012 [329882.490291] ffff88012a58f8d0 0000000000004000 0000000000013f00 000000000000e078 Sep 27 06:27:27 ipp012 [329882.490301] 0000000000000000 0000000000000000 ffff88012a440000 ffff88012babd0f0 Sep 27 06:27:27 ipp012 [329882.490311] Call Trace: Sep 27 06:27:27 ipp012 [329882.490319] [] mutex_lock_nested+0x14e/0x253 Sep 27 06:27:27 ipp012 [329882.490328] [] ? generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.490337] [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.490346] [] ext3_file_write+0x1e/0x9e Sep 27 06:27:27 ipp012 [329882.490354] [] ? ext3_file_write+0x0/0x9e Sep 27 06:27:27 ipp012 [329882.490363] [] do_sync_readv_writev+0xe3/0x12b Sep 27 06:27:27 ipp012 [329882.490372] [] ? autoremove_wake_function+0x0/0x38 Sep 27 06:27:27 ipp012 [329882.490381] [] ? exportfs_decode_fh+0xca/0x20f Sep 27 06:27:27 ipp012 [329882.490390] [] ? rw_copy_check_uvector+0x4d/0xe5 Sep 27 06:27:27 ipp012 [329882.490399] [] ? security_file_permission+0x11/0x13 Sep 27 06:27:27 ipp012 [329882.490408] [] do_readv_writev+0xb8/0x1a0 Sep 27 06:27:27 ipp012 [329882.490416] [] ? file_move+0x4b/0x50 Sep 27 06:27:27 ipp012 [329882.490425] [] vfs_writev+0x3e/0x49 Sep 27 06:27:27 ipp012 [329882.490434] [] nfsd_vfs_write+0x14b/0x384 Sep 27 06:27:27 ipp012 [329882.490442] [] ? dentry_open+0x83/0x8c Sep 27 06:27:27 ipp012 [329882.490450] [] ? nfsd_open+0x195/0x1b8 Sep 27 06:27:27 ipp012 [329882.490459] [] nfsd_write+0xbd/0xd9 Sep 27 06:27:27 ipp012 [329882.490467] [] nfsd3_proc_write+0xf3/0x110 Sep 27 06:27:27 ipp012 [329882.490475] [] nfsd_dispatch+0xfd/0x1de Sep 27 06:27:27 ipp012 [329882.490483] [] svc_process+0x425/0x72a Sep 27 06:27:27 ipp012 [329882.490491] [] ? nfsd+0x0/0x14b Sep 27 06:27:27 ipp012 [329882.490499] [] nfsd+0x105/0x14b Sep 27 06:27:27 ipp012 [329882.490507] [] kthread+0x8f/0x97 Sep 27 06:27:27 ipp012 [329882.490515] [] child_rip+0xa/0x20 Sep 27 06:27:27 ipp012 [329882.490524] [] ? __atomic_notifier_call_chain+0x0/0x87 Sep 27 06:27:27 ipp012 [329882.490532] [] ? kthread+0x0/0x97 Sep 27 06:27:27 ipp012 [329882.490539] [] ? child_rip+0x0/0x20 Sep 27 06:27:27 ipp012 [329882.490545] 2 locks held by nfsd/7661: Sep 27 06:27:27 ipp012 [329882.490549] #0: (hash_sem){......}, at: [] exp_readlock+0x10/0x12 Sep 27 06:27:27 ipp012 [329882.490560] #1: (&sb->s_type->i_mutex_key#6){......}, at: [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.490578] INFO: task nfsd:7665 blocked for more than 120 seconds. Sep 27 06:27:27 ipp012 [329882.490583] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 27 06:27:27 ipp012 [329882.490588] nfsd D ffff88012a4440c0 0 7665 2 0x00000000 Sep 27 06:27:27 ipp012 [329882.490598] ffff880129417930 0000000000000046 00000000000001cb ffffffff81a891f0 Sep 27 06:27:27 ipp012 [329882.490608] ffff8801294178d0 0000000000004000 0000000000013f00 000000000000e078 Sep 27 06:27:27 ipp012 [329882.490618] 0000000000000000 0000000000000000 ffff88012a4440c0 ffff880129552060 Sep 27 06:27:27 ipp012 [329882.490628] Call Trace: Sep 27 06:27:27 ipp012 [329882.490637] [] ? generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.490645] [] mutex_lock_nested+0x14e/0x253 Sep 27 06:27:27 ipp012 [329882.490654] [] ? generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.490663] [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.490672] [] ext3_file_write+0x1e/0x9e Sep 27 06:27:27 ipp012 [329882.490680] [] ? ext3_file_write+0x0/0x9e Sep 27 06:27:27 ipp012 [329882.490689] [] do_sync_readv_writev+0xe3/0x12b Sep 27 06:27:27 ipp012 [329882.490698] [] ? get_partial_node+0x1b/0x8a Sep 27 06:27:27 ipp012 [329882.490707] [] ? autoremove_wake_function+0x0/0x38 Sep 27 06:27:27 ipp012 [329882.490716] [] ? rw_copy_check_uvector+0x4d/0xe5 Sep 27 06:27:27 ipp012 [329882.490725] [] ? security_file_permission+0x11/0x13 Sep 27 06:27:27 ipp012 [329882.490734] [] do_readv_writev+0xb8/0x1a0 Sep 27 06:27:27 ipp012 [329882.490743] [] ? file_move+0x4b/0x50 Sep 27 06:27:27 ipp012 [329882.490752] [] vfs_writev+0x3e/0x49 Sep 27 06:27:27 ipp012 [329882.490760] [] nfsd_vfs_write+0x14b/0x384 Sep 27 06:27:27 ipp012 [329882.490769] [] ? dentry_open+0x83/0x8c Sep 27 06:27:27 ipp012 [329882.490777] [] ? nfsd_open+0x195/0x1b8 Sep 27 06:27:27 ipp012 [329882.490786] [] nfsd_write+0xbd/0xd9 Sep 27 06:27:27 ipp012 [329882.490794] [] nfsd3_proc_write+0xf3/0x110 Sep 27 06:27:27 ipp012 [329882.490802] [] nfsd_dispatch+0xfd/0x1de Sep 27 06:27:27 ipp012 [329882.490810] [] svc_process+0x425/0x72a Sep 27 06:27:27 ipp012 [329882.490818] [] ? nfsd+0x0/0x14b Sep 27 06:27:27 ipp012 [329882.490826] [] nfsd+0x105/0x14b Sep 27 06:27:27 ipp012 [329882.490834] [] kthread+0x8f/0x97 Sep 27 06:27:27 ipp012 [329882.490842] [] child_rip+0xa/0x20 Sep 27 06:27:27 ipp012 [329882.490850] [] ? __atomic_notifier_call_chain+0x0/0x87 Sep 27 06:27:27 ipp012 [329882.490859] [] ? kthread+0x0/0x97 Sep 27 06:27:27 ipp012 [329882.490866] [] ? child_rip+0x0/0x20 Sep 27 06:27:27 ipp012 [329882.490872] 2 locks held by nfsd/7665: Sep 27 06:27:27 ipp012 [329882.490876] #0: (hash_sem){......}, at: [] exp_readlock+0x10/0x12 Sep 27 06:27:27 ipp012 [329882.490887] #1: (&sb->s_type->i_mutex_key#6){......}, at: [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.490904] INFO: task nfsd:7666 blocked for more than 120 seconds. Sep 27 06:27:27 ipp012 [329882.490909] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 27 06:27:27 ipp012 [329882.490914] nfsd D ffff88012a4450f0 0 7666 2 0x00000000 Sep 27 06:27:27 ipp012 [329882.490925] ffff880129455930 0000000000000046 00000000000001cb ffffffff81a891f0 Sep 27 06:27:27 ipp012 [329882.490935] ffff8801294558d0 0000000000004000 0000000000013f00 000000000000e078 Sep 27 06:27:27 ipp012 [329882.490944] 0000000000000000 0000000000000000 ffff88012a4450f0 ffff88012970c0c0 Sep 27 06:27:27 ipp012 [329882.490954] Call Trace: Sep 27 06:27:27 ipp012 [329882.490963] [] ? generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.490971] [] mutex_lock_nested+0x14e/0x253 Sep 27 06:27:27 ipp012 [329882.490980] [] ? generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.490990] [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.490998] [] ext3_file_write+0x1e/0x9e Sep 27 06:27:27 ipp012 [329882.491007] [] ? ext3_file_write+0x0/0x9e Sep 27 06:27:27 ipp012 [329882.491015] [] do_sync_readv_writev+0xe3/0x12b Sep 27 06:27:27 ipp012 [329882.491025] [] ? autoremove_wake_function+0x0/0x38 Sep 27 06:27:27 ipp012 [329882.491033] [] ? exportfs_decode_fh+0xca/0x20f Sep 27 06:27:27 ipp012 [329882.491042] [] ? rw_copy_check_uvector+0x4d/0xe5 Sep 27 06:27:27 ipp012 [329882.491051] [] ? security_file_permission+0x11/0x13 Sep 27 06:27:27 ipp012 [329882.491060] [] do_readv_writev+0xb8/0x1a0 Sep 27 06:27:27 ipp012 [329882.491069] [] ? file_move+0x4b/0x50 Sep 27 06:27:27 ipp012 [329882.491078] [] vfs_writev+0x3e/0x49 Sep 27 06:27:27 ipp012 [329882.491087] [] nfsd_vfs_write+0x14b/0x384 Sep 27 06:27:27 ipp012 [329882.491095] [] ? dentry_open+0x83/0x8c Sep 27 06:27:27 ipp012 [329882.491103] [] ? nfsd_open+0x195/0x1b8 Sep 27 06:27:27 ipp012 [329882.491112] [] nfsd_write+0xbd/0xd9 Sep 27 06:27:27 ipp012 [329882.491120] [] nfsd3_proc_write+0xf3/0x110 Sep 27 06:27:27 ipp012 [329882.491128] [] nfsd_dispatch+0xfd/0x1de Sep 27 06:27:27 ipp012 [329882.491136] [] svc_process+0x425/0x72a Sep 27 06:27:27 ipp012 [329882.491144] [] ? nfsd+0x0/0x14b Sep 27 06:27:27 ipp012 [329882.491152] [] nfsd+0x105/0x14b Sep 27 06:27:27 ipp012 [329882.491160] [] kthread+0x8f/0x97 Sep 27 06:27:27 ipp012 [329882.491168] [] child_rip+0xa/0x20 Sep 27 06:27:27 ipp012 [329882.491177] [] ? __atomic_notifier_call_chain+0x0/0x87 Sep 27 06:27:27 ipp012 [329882.491186] [] ? kthread+0x0/0x97 Sep 27 06:27:27 ipp012 [329882.491193] [] ? child_rip+0x0/0x20 Sep 27 06:27:27 ipp012 [329882.491198] 2 locks held by nfsd/7666: Sep 27 06:27:27 ipp012 [329882.491202] #0: (hash_sem){......}, at: [] exp_readlock+0x10/0x12 Sep 27 06:27:27 ipp012 [329882.491214] #1: (&sb->s_type->i_mutex_key#6){......}, at: [] generic_file_aio_write+0x52/0xc5 Sep 27 06:27:27 ipp012 [329882.491231] INFO: task nfsd:7667 blocked for more than 120 seconds. Sep 27 06:27:27 ipp012 [329882.491235] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 27 06:27:27 ipp012 [329882.491240] nfsd D ffff88012a446120 0 7667 2 0x00000000 Sep 27 06:27:27 ipp012 [329882.491251] ffff880129457cb0 0000000000000046 00000000000001cb ffffffff81a891f0 Sep 27 06:27:27 ipp012 [329882.491261] ffff880129457c50 0000000000004000 0000000000013f00 000000000000e078 Sep 27 06:27:27 ipp012 [329882.491270] 0000000000000000 0000000000000000 ffff88012a446120 ffff8801296bd0f0 Sep 27 06:27:27 ipp012 [329882.491280] Call Trace: Sep 27 06:27:27 ipp012 [329882.491289] [] ? nfsd_sync+0x4c/0xb0 Sep 27 06:27:27 ipp012 [329882.491298] [] ? file_move+0x1e/0x50 Sep 27 06:27:27 ipp012 [329882.491305] [] mutex_lock_nested+0x14e/0x253 Sep 27 06:27:27 ipp012 [329882.491314] [] ? nfsd_sync+0x4c/0xb0 Sep 27 06:27:27 ipp012 [329882.491322] [] nfsd_sync+0x4c/0xb0 Sep 27 06:27:27 ipp012 [329882.491331] [] nfsd_commit+0x5b/0x7e Sep 27 06:27:27 ipp012 [329882.491339] [] nfsd3_proc_commit+0xb5/0xc4 Sep 27 06:27:27 ipp012 [329882.491347] [] nfsd_dispatch+0xfd/0x1de Sep 27 06:27:27 ipp012 [329882.491355] [] svc_process+0x425/0x72a Sep 27 06:27:27 ipp012 [329882.491363] [] ? nfsd+0x0/0x14b Sep 27 06:27:27 ipp012 [329882.491370] [] nfsd+0x105/0x14b Sep 27 06:27:27 ipp012 [329882.491379] [] kthread+0x8f/0x97 Sep 27 06:27:27 ipp012 [329882.491387] [] child_rip+0xa/0x20 Sep 27 06:27:27 ipp012 [329882.491395] [] ? __atomic_notifier_call_chain+0x0/0x87 Sep 27 06:27:27 ipp012 [329882.491404] [] ? kthread+0x0/0x97 Sep 27 06:27:27 ipp012 [329882.491411] [] ? child_rip+0x0/0x20 Sep 27 06:27:27 ipp012 [329882.491417] 2 locks held by nfsd/7667: Sep 27 06:27:27 ipp012 [329882.491421] #0: (hash_sem){......}, at: [] exp_readlock+0x10/0x12 Sep 27 06:27:27 ipp012 [329882.491432] #1: (&sb->s_type->i_mutex_key#6){......}, at: [] nfsd_sync+0x4c/0xb0 . Sep 27 06:37:39 ipp012 [330518.101607] arcmsr6: scsi id = 0 lun = 0 ccb = '0xffff8800cf8c57c0' poll command abort successfully Sep 27 06:37:39 ipp012 [330518.101622] arcmsr6: scsi id = 0 lun = 0 ccb = '0xffff8800cf8d9080' poll command abort successfully Sep 27 06:37:39 ipp012 [330518.101632] arcmsr6: scsi id = 0 lun = 0 ccb = '0xffff8800cf8e2980' poll command abort successfully Sep 27 06:37:39 ipp012 [330518.101641] arcmsr6: scsi id = 0 lun = 0 ccb = '0xffff8800cf8e3b80' poll command abort successfully Sep 27 06:37:39 ipp012 [330518.101650] arcmsr6: scsi id = 0 lun = 0 ccb = '0xffff8800cf8d5cc0' poll command abort successfully Sep 27 06:37:39 ipp012 [330518.101659] arcmsr6: scsi id = 0 lun = 0 ccb = '0xffff8800cf8e8380' poll command abort successfully Sep 27 06:37:52 ipp012 [330518.130328] arcmsr6: scsi id = 0 lun = 0 ccb = '0xffff8800cf8e3dc0' poll command abort successfully Sep 27 06:37:52 ipp012 [330518.130360] arcmsr6: scsi id = 0 lun = 0 ccb = '0xffff8800cf8d92c0' poll command abort successfully Sep 27 06:37:52 ipp012 [330518.130371] arcmsr6: scsi id = 0 lun = 0 ccb = '0xffff8800cf8c1b00' poll command abort successfully Sep 27 06:37:52 ipp012 [330518.160344] arcmsr6: scsi id = 0 lun = 0 ccb = '0xffff8800cf8dbb40' poll command abort successfully Sep 27 06:37:52 ipp012 [330518.160364] arcmsr6: scsi id = 0 lun = 0 ccb = '0xffff8800cf8dcf80' poll command abort successfully Sep 27 06:37:52 ipp012 [330520.651615] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.651647] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.651675] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.651703] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.651730] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.651757] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.651785] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.651812] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.651839] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.651867] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.651895] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.651922] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.651959] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.651987] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.652015] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.652042] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.652069] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.652097] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.652124] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.652151] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.652178] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.652206] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.652233] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.652261] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.652287] arcmsr6: abort device command of scsi id = 0 lun = 0 Sep 27 06:37:52 ipp012 [330520.652314] arcmsr6: abort device command of scsi id = 0 lun = 0 ...[[BR]] Sep 27 06:58:32 ipp012 nrpe[9878]: Error: Could not complete SSL handshake. 5[[BR]] ... Sep 27 07:00:27 ipp012 [331886.290206] nfs: server ipp053 not responding, timed out[[BR]] ...[[BR]] Sep 27 07:00:28 ipp012 [331886.881583] nfs: server ippc18 not responding, timed out[[BR]] ...[[BR]] Sep 27 07:16:42 ipp012 sshd[10202]: Did not receive identification string from 166.122.172.70 (ipp052)[[BR]]