IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links
wiki:PS1_IPP_Czarlog_20121203

Version 42 (modified by bills, 14 years ago) ( diff )

--

      1. Monday : 2012-11-03
      2. Tuesday : 2012.12.04
      3. Wednesday : YYYY.MM.DD
      4. Thursday : YYYY.MM.DD
      5. Friday : YYYY.MM.DD
      6. Saturday : YYYY.MM.DD
      7. Sunday : YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2012-11-03

Serge is czar

  • 06:30 (Serge): No observation last night.
  • 10:00 (Serge): Stopping replication on ippdb02 and starting backup to ippc62 (/export/ippc62.1/backup_nebulous/20121203)
  • 10:30 (Serge): Power cycled ipp012. The not-so-interesting apparent fault is the following: 
    <Dec/03 01:40 am>ipp012 login: [2201848.347410] BUG: unable to handle kernel paging request at ffffffff800073d4
<Dec/03 01:40 am>[2201848.349821] IP: [<ffffffff80589612>] xprt_autoclose+0x19/0x4c
<Dec/03 01:40 am>[2201848.349821] PGD 203067 PUD 207063 PMD 0 
<Dec/03 01:40 am>[2201848.349821] Oops: 0000 [#1] SMP 
<Dec/03 01:40 am>[2201848.349821] last sysfs file: /sys/class/i2c-adapter/i2c-0/0-002f/temp6_alarm
<Dec/03 01:40 am>[2201848.349821] CPU 0 
<Dec/03 01:40 am>[2201848.349821] Modules linked in: w83793 hwmon_vid autofs4 i2c_i801 iTCO_wdt i2c_core e1000e tg3 libphy e1000 xfs 
dm_snapshot dm_mirror dm_region_hash dm_log aacraid 3w_9xxx 3w_xxxx atp870u arcmsr aic7xxx scsi_wait_scan
<Dec/03 01:40 am>[2201848.349821] Pid: 1522, comm: rpciod/0 Tainted: G        W  2.6.28-rc7-00105-gfeaf384 #4
<Dec/03 01:40 am>[2201848.349821] RIP: 0010:[<ffffffff80589612>]  [<ffffffff80589612>] xprt_autoclose+0x19/0x4c
<Dec/03 01:40 am>[2201848.349821] RSP: 0018:ffff88063c8ede60  EFLAGS: 00010246
<Dec/03 01:40 am>[2201848.349821] RAX: ffffffff80007374 RBX: ffff8800697d0578 RCX: 0000000000000000
<Dec/03 01:40 am>[2201848.349821] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8800697d0000
<Dec/03 01:40 am>[2201848.349821] RBP: ffff88033cc09a00 R08: 0000000000000001 R09: 0000000000000000
<Dec/03 01:40 am>[2201848.349821] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800697d0578
<Dec/03 01:40 am>[2201848.349821] R13: ffff8800697d0000 R14: ffffffff808b85a0 R15: 0000000000000000

<Dec/03 01:40 am>[2201848.349821] FS:  0000000000000000(0000) GS:ffffffff80872d80(0000) knlGS:0000000000000000
<Dec/03 01:40 am>[2201848.349821] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<Dec/03 01:40 am>[2201848.349821] CR2: ffffffff800073d4 CR3: 0000000043d72000 CR4: 00000000000006e0
<Dec/03 01:40 am>[2201848.349821] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<Dec/03 01:40 am>[2201848.349821] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<Dec/03 01:40 am>[2201848.349821] Process rpciod/0 (pid: 1522, threadinfo ffff88063c8ec000, task ffff88063ccc5b20)
<Dec/03 01:40 am>[2201848.349821] Stack:
<Dec/03 01:40 am>[2201848.349821]  ffff88063c8ede90 ffff8800697d0578 ffffffff805895f9 ffffffff80248973
<Dec/03 01:40 am>[2201848.349821]  ffffffff8024891d 0000000000000046 0000000000000000 0000000000000000
<Dec/03 01:40 am>[2201848.349821]  0000000000000000 ffff88033cc09a00 ffff88063e43bbf0 ffff88033cc09a00
<Dec/03 01:40 am>[2201848.349821] Call Trace:
<Dec/03 01:40 am>[2201848.349821]  [<ffffffff805895f9>] ? xprt_autoclose+0x0/0x4c
<Dec/03 01:40 am>[2201848.349821]  [<ffffffff80248973>] ? run_workqueue+0xed/0x1ed
<Dec/03 01:40 am>[2201848.349821]  [<ffffffff8024891d>] ? run_workqueue+0x97/0x1ed
<Dec/03 01:40 am>[2201848.349821]  [<ffffffff80249511>] ? worker_thread+0xd8/0xe3
<Dec/03 01:40 am>[2201848.349821]  [<ffffffff8024bfda>] ? autoremove_wake_function+0x0/0x2e
<Dec/03 01:40 am>[2201848.349821]  [<ffffffff80249439>] ? worker_thread+0x0/0xe3
<Dec/03 01:40 am>[2201848.349821]  [<ffffffff8024becc>] ? kthread+0x47/0x73
<Dec/03 01:40 am>[2201848.349821]  [<ffffffff80234852>] ? schedule_tail+0x27/0x5f
<Dec/03 01:40 am>[2201848.349821]  [<ffffffff8020cd19>] ? child_rip+0xa/0x11
<Dec/03 01:40 am>[2201848.349821]  [<ffffffff8024be85>] ? kthread+0x0/0x73
<Dec/03 01:40 am>[2201848.349821]  [<ffffffff8020cd0f>] ? child_rip+0x0/0x11

<Dec/03 01:40 am>[2201848.349821] Code: 89 f7 e8 59 dd 02 00 89 d8 5b 5d 41 5c 41 5d 41 5e c3 41 55 4c 8d af 88 fa ff ff 41 54 53 48 
8b 87 90 fa ff ff 48 89 fb 4c 89 ef <ff> 50 60 f0 80 63 d0 f7 4c 8d a3 98 00 00 00 4c 89 e7 e8 43 de 
<Dec/03 01:40 am>[2201848.349821] RIP  [<ffffffff80589612>] xprt_autoclose+0x19/0x4c
<Dec/03 01:40 am>[2201848.349821]  RSP <ffff88063c8ede60>
<Dec/03 01:40 am>[2201848.349821] CR2: ffffffff800073d4
<Dec/03 01:40 am>[2201848.349821] ---[ end trace 5dc435962a52df42 ]---
    • MEH: stopping lm_sensors on ipp010 and ipp011 (~5 and 1 week since last related crash) to see if this problem can be mitigated
  • 10:45 bill set deepstack to stop. Going to restart with 8 threads per host.
  • 11:15 bill set sts label to active. We'll need to keep an eye on the psastro processes pointing 2 contains a globular cluster and we had some memory explosions yesterday. Camera poll limit is 25
  • 11:45 bill skycal.pro added to stdscience
  • 12:34 bill restarted deepstack with 8 threads per job
  • 13:00 (Serge) Bill set chip.off
  • 14:19 bill {{set chip.on}}}, deepstack to off. Setting 8 threads per job resulted in apparently single threaded operation for some reason.

Tuesday : 2012.12.04

  • 05:00 bill: cluster is wedged due to ipp013 same signature as usual power cycled it. Took a very long time to do its memory check 
     <Dec/04 00:27 am>ipp013 login: [4365681.451101] BUG: unable to handle kernel paging request at ffffffff800073d4
    • remember to add these to the ipp013_log respectively
  • 05:12 staticsky has finished 5 filter runs for 267 < RA < 303 DEC > 10. turned off deepstack for now and added 3 x compute3 to stdscience.
  • 06:40 since rebooting ipp013 the system has made very little progress. Stopping processing.
  • 07:12 setting 10, 11, 12, 13, and 19 to repair mode got things going again. Running with this state for now. We can revisit this during the shutdown later today. Retarted rpc.statd on ipp012 since its glockfiles weren't finishing
  • 07:30 stdscience is now making progress. poll limit set to 250. There are plenty of jobs in the books, but pantasks is falling trouble getting them to the queue. pantasks needs its daily restart
  • 09:27 CZW: stopping all pantasks_servers to allow for NFS switchover at 10:00.
    • MEH: will shut down ghost running refstacks @09:50
  • 09:27:01: Serge: Restarted replication slave ippdb02
  • 10:00 CZW: all processing stopped for NFS switchover.
  • 11:00 MEH: on the problem machines ipp010,011,012,013,014,016,018,019,020
    • restarted rpc.statd twice to clear old process
    • shut down lm_sensors
    • once processing up stable for a while, will slowly put back into neb-host up from repair
  • 11:30 CZW: processing back up.
  • 12:50 CZW: set ipp02[3-6] to nebulous up.
  • 13:30 MEH: stack gets 2x wave4 back in, adding ipp060 which is normally out. MD05.refstack can run with stdscience on compute3 so have started there unless LAP stacks start lagging warps.
  • 13:50 MEH: ipp020 has given up on some mounts, rebooting and turning off lm_sensors on startup.
  • 14:25 MEH: ipp010 neb-host up, will do others in sequence to see if rpc.statd restart and lm_sensors off have cleared. stdscience also has 7x wave1, seems possibly excessive. turning one of those off for - ~8 nodes.
  • 14:55 ipp011 neb-host up (neb-host has date stamp now for tracking)
  • 15:00 Bill: Two staticsky runs have y band inputs that generate unreasonable numbers of detections (> 200,000). Since they are y band these cause forced photometry detections for the other bands. This causes memory explosion when processing. If the runs had completed, they would generate ~2,000,000 false detections for dvo and psps and ultimately the user. I have updated the stack tables to mark the offending stacks as bad quality and removed them from the corresponding staticskyInput. The stack_ids are 1094024 and 1494695.
  • 15:25 MEH: ipp012 neb-host up
  • 15:55 ipp013 neb-host up
  • 18:25 ipp019 neb-host up
  • 18:55 ipp016 neb-host up
  • 16:55 Bill turned on skycal.revert we're getting catalog lock failures from getstar. It uses the same mechanism as glockfile.

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD

Note: See TracWiki for help on using the wiki.