PS1 IPP Czar Logs for the week 2017-02-06 - 2017-02-13

(Up to PS1 IPP Czar Logs)

Monday : 2017-02-06

• 13:28 CZW: Restarted ipptest/replication due to laggy pantasks server. Set npend at 400, which hopefully won't overload any particular host. 300 seemed to be working fine over the weekend.
• 17:15 CZW: Started script running on stare04 to replicate a copy of the detrend data to the ITC cluster (randomly selected host/volume) and to the new ATRC B nodes (randomly selected from ippb07-ippb15). This is running single threaded, and is not pushing very hard, but the number of files included in this transfer is fairly small (34 individual det_id values, ~2000 total files) so this shouldn't be an issue.
• 19:00 EAM : PS1 is down for mirror cleaning

Tuesday : 2017.02.07

• 21:00 EAM : PS1 is down for dome (and weather)
• 21:00 EAM : I rsynced both the gpc1 and nebulous mysqls from the ipp116 & ipp117 replicants to ipp114 & ipp115 today. gpc1 (ipp114) finished in the afternoon, so I restarted ipp116 & started mysql for ipp114 with replication running on both. nebulous just finished. I have restarted ipp117 and started replication there, but I need to build the 5.6 version of mysql before starting ipp115.

Wednesday : YYYY.MM.DD

• MEH: ipp001 disk full per the regular email warning -- running the gpc1 db dump cleanup script
• MEH: ipp089 overloaded from excessive remote_md5sum.pl jobs since ~1700 yesterday -- if expected, then probably needed to take out of processing and put neb-host repair to isolate and notice sent to group, if not expected then should be check in on and not fireoff and forget
  • after ~hour of no response, have stopped ~ipptest/replication in order to get other work done

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD