PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : YYYY.MM.DD

Tuesday : YYYY.MM.DD

• ken noted problems - downloads were slow (not sure why), but they were proceeding. Heather turned off addstars to decrease load on ippdb01 (from 15 - (11 or 12ish))
• 07:40 MEH: local stdlanl stopped + c2 hosts off -- nodes can be used to boost nightly processing catchup (worked on last thursday, ipp067,068,069,070 were able to handle load, have handled local stdlanl being left on during nightly processing as well, so..), also loading in the ippsXX nodes
• 08:00 MEH: stdsci also needs a regular restart, polls and active jobs remain low.. -- wow so much better
• 08:10 MEH: also tweak_wsdiff to not queue jobs until this afternoon
• 11:00 HAF: turned addstars back on
• 11:20 MEH: looking ~noon to finish nightly, since took things apart will put hosts back and set local stdlanl back to run when finish. leaving ippsXX nodes in stdsci for the WS diff unless instructed differently.
• 12:20 MEH: nightly finished, publishing nearly so -- ps_ud_WEB.BIG,ps_ud_WEB.UP,OSS.WS.nightlyscience labels back into stdsci, c2 back to local stdlanl and set to run, WS running with 3x ippsXX machines in stdsci
  • was running stdsci 430 nodes over the regular ~315, saw ~30% rate increase (would be less if downloads still going). local stdlanl ~160 node nominal.
  • by ~1230 ippdb01 was being overloaded again, local stdlanl part of problem?
• 16:20 MEH: cannot rebuild nightly ops tag w/ SHUTOUTC mod until WS are finished otherwise will have issue w/ already made warps -- have to rebuild nightly ops tag tomorrow

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD