PS1 IPP Czar Logs for the week 2017-09-04 - 2017-09-10

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

Tuesday : 2017.09.05

  • 18:25 CZW: Started rsync of retired node poorly replicated data to B nodes. I've started slowly, with ipp006-ipp012 transferring to ippb07-ippb10. I'll likely ramp this up to cover more hosts tomorrow.
  • 21:25 EAM: ipp121 crashed at some point earlier in the night. I was unable to get in via console, so I power cycled it. it came back up.

Wednesday : YYYY.MM.DD

Thursday : 2017.09.07

  • MEH: rather than having the ippc70-c75 critical nebulous apache nodes with various levels of space, reset all the tmp/nebulous_server.log files
    • turn on logrotation for nodes as well to prevent those logs from eating up space -- root crontab for mid-week@10am --
      1 10 * * 3 /etc/cron.daily/logrotate.cron 2>&1
    • turn on disk space check like done for homedir -- also crontab ippc19 to check every 4 hours
      0  */4  *  *  *  /bin/bash /home/panstarrs/ipp/local/bin/ > /dev/null
  • MEH: doing more misc manually triggered cleanup
    • including retry of another buildup of error_cleaned chip/warp/diff...
    • including apparently missed nights 20170903,20170904 as well
    • set regularly red data nodes to neb-host repair -- ipp076, ipp102.0
    • set non-red data nodes to neb-host up again -- ipp115.1, ipp104.0
  • MEH: ippdb09 now primary nebulous DB but no plots in ganglia for status... -- /etc/ganglia/gmond.conf change (ippc18..) --> and restart

Friday : 2017.09.08

  • MEH: Hadyn repaired ippb02 -- putting neb-host repair again so can access its data
  • MEH: Haydn/Gavin working with ippb00 to start the changeover to the 10G switch -- may be offline for a bit
    • success so doing rest -- ippb01-03,06-15 neb-host down
    • DNS may take up to 48 hours to propagate
    • @1600 can ssh to all w/ ippbXX from ipp117
    • fixed the ~ipp/.ssh/known_hosts conflict
  • MEH: ipp086,088,089 have stuck gmond and appear to be down but are not -- appears to be some gmond defunct jobs not clearing, unclear why
  • MEH: ipp@ipp117 nebdiskd is reporting not able to connect to ippdb09 -- manually running nebdiskd -p XXXX having the proper password seemed to fix the problem...
  • MEH: UH has required all machines open to the world to have openssh upgraded to v7.5 -- this impacts ippb04,ippb05 access from ITC and forces password entry -- Gavin is looking into it, but should be ok to set neb-host repair so data can be read from them (though they won't show up in the ippmonitor disk plot)
    • Gavin fixed -- "required to enable dsa in the ssh session (appears to be disabled by default from openssh 7.0 and later) -- This may all go away in future revisions"

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD