PS1 IPP Czar Logs for the week 2017-03-20 - 2017-03-26

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

Tuesday : 2017-03-21

• 16:18 CZW: Setting ipp004-ipp066 to down to check for problems prior to their shutdown next week.
• 16:20 CZW: Restarting ippitc pantasks servers.
• 21:45 EAM: summit copy does not work at night with the proxy; returned to conductor for the rest of the night.
• 22:50 EAM: khcorrect astrom detrend was not on any non-repair node : turned ipp104 to repair and grabbed one there; replicated it 5x at ITC

Wednesday : 2017.03.22

• MEH: special PSNSC/QUB test processing using ITC nodes with data to ipp105,117 only in the afternoon
• 16:45 CZW: Restarted ippitc pantasks.

Thursday : 2017.03.23

• MEH: ippc106 was out of processing since logins broken -- works now but using for QUB processing

Friday : YYYY.MM.DD

Saturday : 2017.03.25

• MEH: %.20170322 cleanup was missed due to stdscience being down during initday -- manually sending to cleanup
  • ipp100-104.0 nodes also filing up faster -- w/ ipp106-116 most full, re-target data to the other datanodes up at the ITC now
• MEH: sent some broken IASC/MOPS stamp request to cleanup since not needed

Sunday : 2017.03.26

• MEH: ippMonitor fixes..
  • ipp115 ippMonitor user not have REPLICATION CLIENT grant -- added and now can see ipp115 neb replication status
  • fault logs broken -- ~ippitc/ippconfig/ippmonitor.config was pointing to ippc64.0 not ippc64.1... ipp_filename.pl hardcoded uses nebserver.ipp.ifa.hawaii.edu... locally hacked to use ippc70...