PS1 IPP Czar Logs for the week 2016.10.03 - 2016.10.09

(Up to PS1 IPP Czar Logs)

Monday : 2016.10.03

  • 16:30 HAF daily restart of panstasks

Tuesday : 2016.10.04

  • 16:10 HAF daily restart of pantasks

Wednesday : 2016.10.05

Thursday : 2016.10.06

  • MEH: apparently ganglia on ippdb01 (nebulous) hasn't been reporting for several months? gmond status shows running, but doing restart and now ganglia reporting status again
    • also notice similar issue w/ ipp017 non-reporting for past couple weeks, running again
  • 17:40 EAM : restarting pantasks
  • 21:55 MEH: ipp068 seems to be having trouble -- take out of pantasks and neb-host repair for now

Friday : 2016.10.07

  • MEH: good number of data nodes going red, doing misc cleanup where possible -- ps_ud_%, sniaf updates, QUB.WS not needed,
  • MEH: removing nodes following from processing and setting repair to try an avoid stalls -- ipp092, 067, 091, 054

Saturday : 2016.10.08

MEH: restarting pantasks w/ ipp092,091,067 manually removed from processing and pstamp tweak bump again to help clear large stamp request

Sunday : 2016.10.09