PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

Tuesday : 2018.01.02

• MEH: ippc111,c118 fixed 20171213 but still out of processing, uncommenting in ~ippitc/ippconfig/pantasks_hosts.input now for adding in during normal restart

Wednesday : YYYY.MM.DD

Thursday : 2018.01.04

• MEH: apparently ipp123-126 was never setup for processing with proper /local/ipp link and files... setting up now and testing with QUB file updates..
  • ippc102, ippx029 seems to also need to be fixed as well -- ippx029 doesn't have ippx029.0 mounted (doesn't exist) --
• MEH: apparently ippc127 has been using the ippc19 disk for user homedir since the switch to ippc18, take out of all processing -- Haydn fixed now

Friday : 2018.01.05

• MEH: ipp100.0,102.0,103.0,104.0,118.0,119.0,122.0 all nearly full -- put neb-host repair to keep space available for dvo/ipptopsps work
• MEH: using nodes to update warps to fix the incorrect processing on ippc127 with wrong homedir through weekend
• MEH: Haydn recovered the ippx029.0 raid, looks like old data and wants to reset the raid so uses normal mount device --

Saturday : 2018.01.06

• MEH: unfortunately been successful in repeat lockup of older computer nodes ippc23-c62 and even ippx052 with GPF or BUG: unable to handle kernel NULL pointer dereference while just running chip+warp updates half loaded (temperatures seem to be fine in nodes around the failed nodes at the time, but haven't see at time of issue since blocks logging into) and when only a subset of nodes are being used in a cabinet -- example console output below

ippc59 login: [ 7437.737569] general protection fault: 0000 [#1] SMP 
[ 7437.739606] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:01:05.0/class

[ 7437.739606] CPU 5 
[ 7437.739606] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 iTCO_wdt i2c_core e1000e tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
[ 7437.739606] Pid: 1690, comm: rpciod/5 Not tainted 2.6.28-rc7-00105-gfeaf384 #4
[ 7437.739606] RIP: 0010:[<ffffffff8059399a>]  [<ffffffff8059399a>] rpcb_getport_done+0x25/0xa4
[ 7437.739606] RSP: 0018:ffff88063ad21e20  EFLAGS: 00010246
[ 7437.739606] RAX: ffffffff80593975 RBX: 2e30313d72646461 RCX: ffff88063ac8f0f0
[ 7437.739606] RDX: 0000000000000201 RSI: ffff88063a45b080 RDI: ffff8804e4ecb040
[ 7437.739606] RBP: 00000000ffffffa3 R08: 0000000000000001 R09: ffff880a5c1e81a8
[ 7437.739606] R10: 0000000000000000 R11: ffffffff8050a1ce R12: ffff88063a45b080
[ 7437.739606] R13: ffff8804e4ecb040 R14: ffffffff808b85a0 R15: 0000000000000000
[ 7437.739606] FS:  0000000000000000(0000) GS:ffff88063cc63200(0000) knlGS:0000000000000000
[ 7437.739606] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[ 7437.849392] CR2: 00007f83a85ad7f0 CR3: 000000045ddf7000 CR4: 00000000000006e0
[ 7437.849392] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7437.849392] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 7437.849392] Process rpciod/5 (pid: 1690, threadinfo ffff88063ad20000, task ffff88063ac8ea50)
[ 7437.849392] Stack:
[ 7437.849392]  ffff880a5c1e8610 ffff8804e4ecb040 ffff8804e4ecb040 ffff8804e4ecb0d0
[ 7437.849392]  ffffffff8058d274 ffffffff8058cba0 0000000000000000 ffffffff8058d0bf
[ 7437.849392]  ffff88063ad21e90 ffff88063acfc700 ffff8804e4ecb0e8 ffffffff80248973
[ 7437.849392] Call Trace:
[ 7437.849392]  [<ffffffff8058d274>] ? rpc_async_schedule+0x0/0xc

[ 7437.849392]  [<ffffffff8058cba0>] ? rpc_exit_task+0x1f/0x4b
[ 7437.919753]  [<ffffffff8058d0bf>] ? __rpc_execute+0x73/0x228
[ 7437.919753]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
[ 7437.919753]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
[ 7437.919753]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
[ 7437.942798]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
[ 7437.942798]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
[ 7437.942798]  [<ffffffff8024becc>] ? kthread+0x47/0x73
[ 7437.942798]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
[ 7437.942798]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
[ 7437.942798]  [<ffffffff804fa23c>] ? menu_reflect+0x0/0x6d
[ 7437.942798]  [<ffffffff8024be85>] ? kthread+0x0/0x73
[ 7437.942798]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
[ 7437.942798] Code: fb ff ff ff 5a c3 41 55 49 89 fd 41 54 49 89 f4 55 53 48 83 ec 08 8b 6f 30 48 8b 1e 83 fd fb 75 05 40 b5 a3 eb 05 83 fd a3 75 08 <ff> 83 54 05 00 00 eb 04 85 ed 79 0e 48 8b 43 08 31 f6 48 89 df 
[ 7437.942798] RIP  [<ffffffff8059399a>] rpcb_getport_done+0x25/0xa4
[ 7437.942798]  RSP <ffff88063ad21e20>
[ 7438.023981] ---[ end trace e518b8b9f4fcd4d9 ]---
[ 7781.039304] lockd: server ipp124 not responding, still trying
[ 8064.415955] lockd: server ipp124 not responding, still trying
[12424.142226] nfs: server ippc18 not responding, still trying



This is ippc60.ifa.hawaii.edu (Linux x86_64 2.6.28-rc7-00105-gfeaf384) 16:04:52

ippc60 login: [21388106.103992] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
[21388106.104527] IP: [<ffffffff805939af>] rpcb_getport_done+0x3a/0xa4
[21388106.104527] PGD 1a8b2f067 PUD 5da123067 PMD 0 
[21388106.104527] Oops: 0000 [#1] SMP 

[21388106.104527] last sysfs file: /sys/class/i2c-adapter/i2c-0/0-002f/temp6_alarm
[21388106.104527] CPU 0 
[21388106.104527] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 e1000e i2c_core iTCO_wdt tg3 libphy e1000 xfs dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
[21388106.104527] Pid: 1685, comm: rpciod/0 Tainted: G   M       2.6.28-rc7-00105-gfeaf384 #4
[21388106.104527] RIP: 0010:[<ffffffff805939af>]  [<ffffffff805939af>] rpcb_getport_done+0x3a/0xa4
[21388106.104527] RSP: 0018:ffff88063acdbe20  EFLAGS: 00010246
[21388106.104527] RAX: 0000000000000000 RBX: ffff880c3b485fc0 RCX: ffff88038b5351a8
[21388106.104527] RDX: 0000000000000201 RSI: 0000000000000000 RDI: ffff880c3b485fc0
[21388106.104527] RBP: 00000000ffffffa3 R08: 0000000000000001 R09: 0000000000000000
[21388106.104527] R10: 0000000000000000 R11: ffffffff8050a1ce R12: ffff880c3b485380
[21388106.104527] R13: ffff8804fa79b180 R14: ffffffff808b85a0 R15: 0000000000000000
[21388106.104527] FS:  0000000000000000(0000) GS:ffffffff80872d80(0000) knlGS:0000000000000000
[21388106.104527] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[21388106.104527] CR2: 0000000000000020 CR3: 000000063a4bd000 CR4: 00000000000006e0
[21388106.104527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[21388106.104527] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[21388106.104527] Process rpciod/0 (pid: 1685, threadinfo ffff88063acda000, task ffff88063ac69e60)
[21388106.104527] Stack:
[21388106.104527]  ffff88038b535610 ffff8804fa79b180 ffff8804fa79b180 ffff8804fa79b210
[21388106.104527]  ffffffff8058d274 ffffffff8058cba0 0000000000000000 ffffffff8058d0bf
[21388106.104527]  ffff88063acdbe90 ffff88063acd8100 ffff8804fa79b228 ffffffff80248973
[21388106.104527] Call Trace:

[21388106.104527]  [<ffffffff8058d274>] ? rpc_async_schedule+0x0/0xc
[21388106.104527]  [<ffffffff8058cba0>] ? rpc_exit_task+0x1f/0x4b
[21388106.104527]  [<ffffffff8058d0bf>] ? __rpc_execute+0x73/0x228
[21388106.104527]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
[21388106.104527]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
[21388106.104527]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
[21388106.104527]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
[21388106.104527]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
[21388106.104527]  [<ffffffff8024becc>] ? kthread+0x47/0x73
[21388106.104527]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
[21388106.104527]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
[21388106.104527]  [<ffffffff804fa23c>] ? menu_reflect+0x0/0x6d
[21388106.104527]  [<ffffffff8024be85>] ? kthread+0x0/0x73
[21388106.104527]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
[21388106.104527] Code: 08 8b 6f 30 48 8b 1e 83 fd fb 75 05 40 b5 a3 eb 05 83 fd a3 75 08 ff 83 54 05 00 00 eb 04 85 ed 79 0e 48 8b 43 08 31 f6 48 89 df <ff> 50 20 eb 31 8b 46 14 48 8b 53 08 66 85 c0 75 0f 31 f6 48 89 
[21388106.104527] RIP  [<ffffffff805939af>] rpcb_getport_done+0x3a/0xa4
[21388106.104527]  RSP <ffff88063acdbe20>
[21388106.104527] CR2: 0000000000000020
[21388106.418953] ---[ end trace 17f9195be0d96afa ]---
[21388166.098002] lockd: server ipp101 not responding, still trying
[21388166.197981] lockd: server ipp123 not responding, still trying
[21388166.204401] lockd: server ipp123 not responding, still trying

Sunday : YYYY.MM.DD