PS1 IPP Czar Logs for the week 2019.01.28 - 2019.02.03

(Up to PS1 IPP Czar Logs)

Monday : 2019.01.28

• MEH: manually sent old update chips to cleanup to free up space where possible since some nodes were going red -- also included all WSdiffs up through past few days and WWdiffs only through end of 2018 since the WWdiff update bug still exists
• TdB: IPP backup machines ippb16,ippb17,ipp19 taken down for Haydn to convert to Ubuntu

Tuesday : 2019.01.29

• TdB: Summitcopy for gpc2 has been updated to include compression.

Wednesday : 2019.01.30

• TdB: IPP backup machines ippb16,ippb17,ipp19 set back to up and SSH keys reset for users ippitc,ipp,ipps2

Thursday : 2019.01.31

• TdB: IPP backup machines ippb16-20 had automount problems. Gavin fixed those, and machines are set back to up.

Friday : 2019.02.01

• TdB: IPP processing failed due to an issue related to the permissions set on the recently updated backup machines ippb16-20. Owner of nebulous disks was set to "ipp users" instead of "apache nebulous". Machines set to repair while the issue is being fixed and processing has now continued.
• TdB: ippb16-20 permissions for nebulous drives fixed to apache, and Gene is running chown on all files on these machines. Will be set back to up on monday, due to Friday being a bad day for system-wise changes.
• TdB: machine ipp70 removed from /home/panstarrs/ipp/local/bin/apachedisk_chk.sh since it is down (and had already been removed from nebulous), but still spawning warning emails every 4 hours.

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD