PS1 IPP Czar Logs for the week 2017-09-04 - 2017-09-10

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

Tuesday : 2017.09.05

• 18:25 CZW: Started rsync of retired node poorly replicated data to B nodes. I've started slowly, with ipp006-ipp012 transferring to ippb07-ippb10. I'll likely ramp this up to cover more hosts tomorrow.
• 21:25 EAM: ipp121 crashed at some point earlier in the night. I was unable to get in via console, so I power cycled it. it came back up.

Wednesday : YYYY.MM.DD

Thursday : 2017.09.07

• MEH: rather than having the ippc70-c75 critical nebulous apache nodes with various levels of space, reset all the tmp/nebulous_server.log files
  • turn on logrotation for nodes as well to prevent those logs from eating up space -- root crontab for mid-week@10am --

    1 10 * * 3 /etc/cron.daily/logrotate.cron 2>&1
    

  • turn on disk space check like done for homedir -- also crontab ippc19 to check every 4 hours

    0  */4  *  *  *  /bin/bash /home/panstarrs/ipp/local/bin/apachedisk_chk.sh > /dev/null
    

• MEH: doing more misc cleanup
  • including retry of another buildup of error_cleaned chip/warp/diff... *
  • set regularly red data nodes to neb-host repair -- ipp076, ipp102.0
  • set non-red data nodes to neb-host up again -- ipp115.1, ipp104.0
• MEH: ippdb09 now primary nebulous DB but no plots in ganglia for status... -- /etc/ganglia/gmond.conf change 10.10.20.16 (ippc18..) --> 10.10.20.17 and restart

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD