PS1 IPP Czar Logs for the week 2017-03-20 - 2017-03-26

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

Tuesday : 2017-03-21

  • 16:18 CZW: Setting ipp004-ipp066 to down to check for problems prior to their shutdown next week.
  • 16:20 CZW: Restarting ippitc pantasks servers.
  • 21:45 EAM: summit copy does not work at night with the proxy; returned to conductor for the rest of the night.
  • 22:50 EAM: khcorrect astrom detrend was not on any non-repair node : turned ipp104 to repair and grabbed one there; replicated it 5x at ITC

Wednesday : 2017.03.22

  • MEH: special PSNSC/QUB test processing using ITC nodes with data to ipp105,117 only in the afternoon
  • 16:45 CZW: Restarted ippitc pantasks.

Thursday : 2017.03.23

  • MEH: ippc106 was out of processing since logins broken -- works now but using for QUB processing

Friday : YYYY.MM.DD

Saturday : 2017.03.25

  • MEH: %.20170322 cleanup was missed due to stdscience being down during initday -- manually sending to cleanup
    • ipp100-104.0 nodes also filing up faster -- w/ ipp106-116 most full, re-target data to the other datanodes up at the ITC now
  • MEH: sent some broken IASC/MOPS stamp request to cleanup since not needed

Sunday : 2017.03.26

  • MEH: ippMonitor fixes..
    • ipp115 ippMonitor user not have REPLICATION CLIENT grant -- added and now can see ipp115 neb replication status, hack out other repl_ db in /var/www/localhost/htdocs/ippMonitor/site.php to remove the connection stall until those are back up
    • fault logs broken -- ~ippitc/ippconfig/ippmonitor.config was pointing to ippc64.0 not ippc64.1... hardcoded uses locally hacked to use ippc70...