Version 3 (modified by mhuber@…, 3 months ago)

--

PS1 IPP Czar Logs for the week 2017-09-04 - 2017-09-10

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

Tuesday : 2017.09.05

  • 18:25 CZW: Started rsync of retired node poorly replicated data to B nodes. I've started slowly, with ipp006-ipp012 transferring to ippb07-ippb10. I'll likely ramp this up to cover more hosts tomorrow.
  • 21:25 EAM: ipp121 crashed at some point earlier in the night. I was unable to get in via console, so I power cycled it. it came back up.

Wednesday : YYYY.MM.DD

Thursday : 2017.09.07

  • MEH: rather than having the ippc70-c75 critical nebulous apache nodes with various levels of space, reset all the tmp/nebulous_server.log files
    • turn on logrotation for nodes as well to prevent those logs from eating up space -- root crontab for mid-week@10am --
      1 10 * * 3 /etc/cron.daily/logrotate.cron 2>&1
      
    • turn on disk space check like done for homedir -- also crontab ippc19 to check every 4 hours
      0  */4  *  *  *  /bin/bash /home/panstarrs/ipp/local/bin/apachedisk_chk.sh > /dev/null
      
  • MEH: doing more misc cleanup
    • including retry of another buildup of error_cleaned chip/warp/diff... *
    • set regularly red data nodes to neb-host repair -- ipp076, ipp102.0
    • set non-red data nodes to neb-host up again -- ipp115.1, ipp104.0
  • MEH: ippdb09 now primary nebulous DB but no plots in ganglia for status... -- /etc/ganglia/gmond.conf change 10.10.20.16 (ippc18..) --> 10.10.20.17 and restart

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD