PS1 IPP Czar Logs for the week 2017-07-31 - 2017-08-06

(Up to PS1 IPP Czar Logs)

Monday : 2017-07-31

• 16:00 CZW: Restarted ippitc pantasks. The check_system.sh stop.block.restart seems to be working correctly.

Tuesday : 2017-08-01

• 15:15 CZW: Power cycling stare01, which I appear to have killed with ppStack.
• 15:54 CZW: And again. Leaving processing off on that host, as it seems to have an issue. From the console:

  [ 1411.290072] Pid: 14737, comm: addstar Tainted: G   M       2.6.28-rc7-00105-gfeaf384 #4
  [ 1411.290072] Call Trace:
  [ 1411.290072]  <#MC>  [<ffffffff802397db>] warn_on_slowpath+0x51/0x6d
  [ 1411.290072]  [<ffffffff803eec86>] notify_update+0x2b/0x30
  [ 1411.290072]  [<ffffffff8025bc31>] smp_call_function_mask+0x37/0x1d7
  [ 1411.290072]  [<ffffffff80264013>] crash_kexec+0x17/0xef
  [ 1411.290072]  [<ffffffff802640e2>] crash_kexec+0xe6/0xef
  [ 1411.290072]  [<ffffffff8021d53e>] native_smp_send_stop+0x1a/0x26
  [ 1411.290072]  [<ffffffff80239896>] panic+0x95/0x13f
  [ 1411.290072]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5
  [ 1411.290072]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5
  [ 1411.290072]  [<ffffffff80239d5d>] release_console_sem+0x3e/0x1a5
  [ 1411.290072]  [<ffffffff805b9e66>] __atomic_notifier_call_chain+0x74/0x83
  
  [ 1411.290072]  [<ffffffff805b9df2>] __atomic_notifier_call_chain+0x0/0x83
  [ 1411.290072]  [<ffffffff8023969e>] oops_enter+0x9/0x10
  [ 1411.290072]  [<ffffffff80217b55>] mce_log+0x0/0x7f
  [ 1411.290072]  [<ffffffff80217eef>] do_machine_check+0x2d5/0x378
  [ 1411.290072]  [<ffffffff8020d04f>] machine_check+0x7f/0x90
  [ 1411.290072]  <<EOE>> <4>---[ end trace 9eaaeb060d55ea58 ]---
  

Wednesday : 2017-08-02

• 15:00 CZW: Adding x3 to the /data/stare04.1/watersc1/hsc/stdsci2/ pantasks, in an attempt to get HSC stacks finished at a reasonable rate. With stare03/04 now responding properly to NFS, this should not cause mount issues.

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD