PS1 IPP Czar Logs for the week 2017.07.10 - 2017.07.16

(Up to PS1 IPP Czar Logs)

Monday : 2017.07.10

• MEH: as discussed at the meeting last week and this week, priority w-band refstacks need to be running on as many nodes as possible, but are removed from data nodes during nightly processing (~ippqub/stack/ptolemy.rc, PSNSC.wref.20170707)
  • ipp058-066 repair->up during day for refstack processing (until h02-con power management setup)

Tuesday : 2017.07.11

• 13:00 EAM : I have updated czarpoll for ippMonitor in a couple of useful ways:
  • I have fixed a bug preventing it from reading the labels from the pantasks
  • I have modified the code to merge the stdscience and distribution pantasks labels together for display
    • this latter should obviate the need to manually hack the CzarDb.pm code to add in labels.
  • I have added the gnuplot binary code path to the .xml config file, and removed the hard-coded GNUPLOT value from Plotter.pm
  • I have moved the config file up one level (to ippMonitor/czartool/czarconfig.xml from ippMonitor/czartool/czartool/czarconfig.xml) so it is not so hidden
• MEH: adding limited w refstack processing on ippx nodes along side dvopsps processing -- x051-x064

Wednesday : 2017.07.12

• MEH: cleaned up misc K2+MD07 nightly products
• MEH: continuing to add limited refstack processing on ippx nodes along side dvopsps processing -- x030-x050
• MEH: ipp121 ganglia down, restarted gmond and reporting again so any issues can be logged...
• 16:00 CZW: power cycled stare01, which I seem to have killed accidentally.

Thursday : 2017.07.13

• 06:50 MEH: a very long list of summitcopy fault 200 -- using revertcopy to clear

  request failed: 500 EOF when chunk header expected at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/dsget line 155.
  

• 07:20 MEH: lost connection to summit for a few mins at least

  request failed: 503 Service Temporarily Unavailable at /data/ippc64.1/ippitc/psconfig/ipp-20170121.lin64/bin/dsget line 155.
  

• MEH: network issue summit faulted files before manually started revertcopy -- manually set fault 0 in gpc1 to re-download

  7 summit faults
    exp_name         registered      fault
  o7947g0105o   2017-07-13 14:44:34   203
  o7947g0107o   2017-07-13 14:46:03   203
  o7947g0108o   2017-07-13 14:46:44   203
  o7947g0109o   2017-07-13 14:47:29   203
  o7947g0110o   2017-07-13 14:48:10   203
  o7947g0111o   2017-07-13 14:48:51   203
  o7947g0116o   2017-07-13 14:52:28   203
  

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD