PS1 IPP Czar Logs for the week 2016.10.03 - 2016.10.09

(Up to PS1 IPP Czar Logs)

Monday : 2016.10.03

• 16:30 HAF daily restart of panstasks

Tuesday : 2016.10.04

• 16:10 HAF daily restart of pantasks

Wednesday : 2016.10.05

Thursday : 2016.10.06

• MEH: apparently ganglia on ippdb01 (nebulous) hasn't been reporting for several months? gmond status shows running, but doing restart and now ganglia reporting status again
  • also notice similar issue w/ ipp017 non-reporting for past couple weeks, running again
• 17:40 EAM : restarting pantasks
• 21:55 MEH: ipp068 seems to be having trouble -- take out of pantasks and neb-host repair for now

Friday : 2016.10.07

• MEH: good number of data nodes going red, doing misc cleanup where possible -- ps_ud_%, sniaf updates, QUB.WS not needed,
• MEH: removing nodes following from processing and setting repair to try an avoid stalls -- ipp092, 067, 091, 054

Saturday : 2016.10.08

MEH: restarting pantasks w/ ipp092,091,067 manually removed from processing and pstamp tweak bump again to help clear large stamp request

Sunday : 2016.10.09