(Up to PS1 IPP Czar Logs)

Monday : 2012-11-03

Serge is czar

• 06:30 (Serge): No observation last night.
• 10:00 (Serge): Stopping replication on ippdb02 and starting backup to ippc62 (/export/ippc62.1/backup_nebulous/20121203)
• 10:30 (Serge): Power cycled ipp012. The not-so-interesting apparent fault is the following:

  <Dec/03 01:40 am>ipp012 login: [2201848.347410] BUG: unable to handle kernel paging request at ffffffff800073d4
  <Dec/03 01:40 am>[2201848.349821] IP: [<ffffffff80589612>] xprt_autoclose+0x19/0x4c
  <Dec/03 01:40 am>[2201848.349821] PGD 203067 PUD 207063 PMD 0 
  <Dec/03 01:40 am>[2201848.349821] Oops: 0000 [#1] SMP 
  <Dec/03 01:40 am>[2201848.349821] last sysfs file: /sys/class/i2c-adapter/i2c-0/0-002f/temp6_alarm
  <Dec/03 01:40 am>[2201848.349821] CPU 0 
  <Dec/03 01:40 am>[2201848.349821] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 iTCO_wdt i2c_core e1000e tg3 libphy e1000 xfs 
  dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
  <Dec/03 01:40 am>[2201848.349821] Pid: 1522, comm: rpciod/0 Tainted: G        W  2.6.28-rc7-00105-gfeaf384 #4
  <Dec/03 01:40 am>[2201848.349821] RIP: 0010:[<ffffffff80589612>]  [<ffffffff80589612>] xprt_autoclose+0x19/0x4c
  <Dec/03 01:40 am>[2201848.349821] RSP: 0018:ffff88063c8ede60  EFLAGS: 00010246
  <Dec/03 01:40 am>[2201848.349821] RAX: ffffffff80007374 RBX: ffff8800697d0578 RCX: 0000000000000000
  <Dec/03 01:40 am>[2201848.349821] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8800697d0000
  <Dec/03 01:40 am>[2201848.349821] RBP: ffff88033cc09a00 R08: 0000000000000001 R09: 0000000000000000
  <Dec/03 01:40 am>[2201848.349821] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800697d0578
  <Dec/03 01:40 am>[2201848.349821] R13: ffff8800697d0000 R14: ffffffff808b85a0 R15: 0000000000000000
  
  <Dec/03 01:40 am>[2201848.349821] FS:  0000000000000000(0000) GS:ffffffff80872d80(0000) knlGS:0000000000000000
  <Dec/03 01:40 am>[2201848.349821] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
  <Dec/03 01:40 am>[2201848.349821] CR2: ffffffff800073d4 CR3: 0000000043d72000 CR4: 00000000000006e0
  <Dec/03 01:40 am>[2201848.349821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  <Dec/03 01:40 am>[2201848.349821] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
  <Dec/03 01:40 am>[2201848.349821] Process rpciod/0 (pid: 1522, threadinfo ffff88063c8ec000, task ffff88063ccc5b20)
  <Dec/03 01:40 am>[2201848.349821] Stack:
  <Dec/03 01:40 am>[2201848.349821]  ffff88063c8ede90 ffff8800697d0578 ffffffff805895f9 ffffffff80248973
  <Dec/03 01:40 am>[2201848.349821]  ffffffff8024891d 0000000000000046 0000000000000000 0000000000000000
  <Dec/03 01:40 am>[2201848.349821]  0000000000000000 ffff88033cc09a00 ffff88063e43bbf0 ffff88033cc09a00
  <Dec/03 01:40 am>[2201848.349821] Call Trace:
  <Dec/03 01:40 am>[2201848.349821]  [<ffffffff805895f9>] ? xprt_autoclose+0x0/0x4c
  <Dec/03 01:40 am>[2201848.349821]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
  <Dec/03 01:40 am>[2201848.349821]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
  <Dec/03 01:40 am>[2201848.349821]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
  <Dec/03 01:40 am>[2201848.349821]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
  <Dec/03 01:40 am>[2201848.349821]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
  <Dec/03 01:40 am>[2201848.349821]  [<ffffffff8024becc>] ? kthread+0x47/0x73
  <Dec/03 01:40 am>[2201848.349821]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
  <Dec/03 01:40 am>[2201848.349821]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
  <Dec/03 01:40 am>[2201848.349821]  [<ffffffff8024be85>] ? kthread+0x0/0x73
  <Dec/03 01:40 am>[2201848.349821]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11
  
  <Dec/03 01:40 am>[2201848.349821] Code: 89 f7 e8 59 dd 02 00 89 d8 5b 5d 41 5c 41 5d 41 5e c3 41 55 4c 8d af 88 fa ff ff 41 54 53 48 
  8b 87 90 fa ff ff 48 89 fb 4c 89 ef <ff> 50 60 f0 80 63 d0 f7 4c 8d a3 98 00 00 00 4c 89 e7 e8 43 de 
  <Dec/03 01:40 am>[2201848.349821] RIP  [<ffffffff80589612>] xprt_autoclose+0x19/0x4c
  <Dec/03 01:40 am>[2201848.349821]  RSP <ffff88063c8ede60>
  <Dec/03 01:40 am>[2201848.349821] CR2: ffffffff800073d4
  <Dec/03 01:40 am>[2201848.349821] ---[ end trace 5dc435962a52df42 ]---
  

  • MEH: stopping lm_sensors on ipp010 and ipp011 (~5 and 1 week since last related crash) to see if this problem can be mitigated
• 10:45 bill set deepstack to stop. Going to restart with 8 threads per host.
• 11:15 bill set sts label to active. We'll need to keep an eye on the psastro processes pointing 2 contains a globular cluster and we had some memory explosions yesterday. Camera poll limit is 25
• 11:45 bill skycal.pro added to stdscience
• 12:34 bill restarted deepstack with 8 threads per job
• 13:00 (Serge) Bill set chip.off
• 14:19 bill {{set chip.on}}}, deepstack to off. Setting 8 threads per job resulted in apparently single threaded operation for some reason.

Tuesday : 2012.12.04

• 05:00 bill: cluster is wedged due to ipp013 same signature as usual power cycled it. Took a very long time to do its memory check

   <Dec/04 00:27 am>ipp013 login: [4365681.451101] BUG: unable to handle kernel paging request at ffffffff800073d4
  

• 05:12 staticsky has finished 5 filter runs for 267 < RA < 303 DEC > 10. turned off deepstack for now and added 3 x compute3 to stdscience.
• 06:40 since rebooting ipp013 the system has made very little progress. Stopping processing.
• 07:12 setting 10, 11, 12, 13, and 19 to repair mode got things going again. Running with this state for now. We can revisit this during the shutdown later today. Retarted rpc.statd on ipp012 since its glockfiles weren't finishing
• 07:30 stdscience is now making progress. poll limit set to 250. There are plenty of jobs in the books, but pantasks is falling trouble getting

them to the queue. pantasks needs its daily restart

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD