PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2016.02.29

  • 10:02 MEH: SNIaF updates running in normal stdscience (turn down/off as needed) and will be cleaned before nightly processing tonight
  • 20:10 EAM: stopping and restarting pantasks for the night

Tuesday : 2016.03.01

  • 06:00 EAM: cleared a stuck exposure in summitcopy (118o)
  • 11:00 CZW: restart replication as ipptest on ippc05 for ippb03.1 ota data.
  • 17:15 EAM: restarting pantasks

Wednesday : 2016.03.02

  • 08:30 MEH: clearing fault 5 from 20160301 and last night before the accelerated cleanup get to them..
  • 08:54 MEH: ipp087 BBU was fixed last week, it should have been restored as a data target host in ~ipp/psconfig/ipp-20141024.lin64/share/pantasks/modules/ipphosts.mhpcc.config...

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

  • 15:00 EAM: stopping pantasks to adjust the processing lists (remove ipp070-ipp097 for now -- needed to run relastro)
  • 15:15 EAM: I have adjusted the processing assignments to remove the _s5 set from the list (ipp070 - ipp097). I have replaced them with x2 and x3 for stdscience and s4 for summitcopy and registration (2 clients should not be an excessive additional load there).

Saturday : 2016.03.05

  • 05:45 EAM: stdscience is far behind. not sure if this is due to the relastro load or not, but i'm restarting stdscience pantasks since it is not filling its queues.
    • update: it looks like ippdb05 is responding badly -- i'll quickly restart the mysql
    • stopping all ipp user pantasks to restart ippdb05 mysql
    • stopped addstar pantasks on ipp071, ipp072, ipp073, ipp074, ipp075, ipp076
  • 06:15 EAM: mysql has been restarted, and processing has also been restarted. not sure if the mysql restart helped or if the addstars were putting too much load on the system or what.
  • 07:55 EAM : forgot to restart RoboCzar? & CzarPoll? on ippc11 -- now restarted

Sunday : 2016.03.06

  • 21:00 EAM: stopping and restarting the pantaskses