PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2017.03.06

• MEH: ipp001 disk space low, running mysql cleanup scripts for gpc1 dumps
• MEH: ipp078 in repair again so Haydn can work on it this week -- did today, swapped SATA cables and backup okay -- leave neb-host repair w/o nightly load, just Heather may have jobs again on it
• MEH: Gene fix for dvoImagesAtCoords -- rebuild ipp ipp-20170121/Ohana onward and pstamp seems to produce chips okay again

Tuesday : YYYY.MM.DD

Wednesday : YYYY.MM.DD

Thursday : 2017-03-09

• 17:20 CZW: Set ITC nodes to xattr=0 in nebulous. This field controls replication/new copy status, and will now allow first copies to be generated on the ITC nodes (ipp105-117).
• 18:40 CZW: Stopped apache on ippc17 and ippc30 to ensure no problems with the ITC switchover test tonight.

Friday : 2017-03-10

• 13:30 CZW: Restarted ipptest/replication with 50 jobs to prevent overloading of MRTCB-ITC connection.
• MEH: running ippqub WSdiffs and updates during ippitc processing of MOPS test chunk
• MEH: all nodes ipp054-ipp104 into repair to prevent scattered data going there in preparation to move to ITC -- all mrtc-b nightly to target ipp118-122
• MEH: distribution uses targeting parameters in gpc1:rcDestination:dbhost for ippc17, will take a little time to setup again for ipp113 and distribution is off for now -- manually changed in ippdb08 and working for meh-tests, QUB-tests, ps1-3piWS-cat
• MEH: pstamp@mrtc-b (just to keep all active processing there) seems mostly functional --
• MEH: restarting summitcopy, registration, stdscience with updated nodes (replace ippx001-12 and ipp090,91,92 shutdown) and for targeting ipp118-122 only -- running @mrtc-b, all individually started w/ start_server.sh
• MEH: cleanup wasn't on/cleared before ipp090-092 were turned off and packed -- may be issues -- leaving off unless need to clear issues
• MEH: MD07 on obsplan for tonight -- start stack pantasks
• MEH: ippdb03 replication stalled since 20170307 -- turning dump off in ipp@ippdb03 crontab
• MEH: WSdiff alternate queuing started again and running in ippqub:stdscience_ws
• MEH: change in datastore/ippRequestServer setup has added many old, broken/cleaned entries -- will need to clean up at least ps1-3piWS-cat as the very long list can cause problems

Saturday : 2017.03.11

• MEH: using ippqub:stdscience_ws to do large number of critical updates for SNIaF before datanodes packed up

Sunday : YYYY.MM.DD