PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2013.03.18

mark is czar

• 01:00 MEH: putting 1x compute3 back on in deepstack for the staticsky outside of the GP for now. leaving normal allocation of 1x compute3 in stack now that mostly caught up, will have nightly science stacks to do. only 2/4x compute3 in stdscience manually for now.
• Serge looked at mysql on ipp0012 and ippdb02, restarted and slowly catching up.
• 15:20 MEH: odd, processing almost came to halt.. looks like stdscience pantasks is underloaded.. slowly coming back up. might as well do regular pantasks restart

Tuesday : 2013.03.19

mark is czar

• 00:30 MEH: datastore having intermittent proxy errors
• 07:05 MEH: nightly science still processing, turning chip.off to push the nightly warps out. looks like an OSS and a few other publishing jobs had fault and revert not automatic.
• 09:20 MEH: warps+diffs only running, something limiting processing (stsci06/07 over-targeted?) -- 3PI nightly now mostly finished, 3PI WSdiff a good number to do still
• 11:15 MEH: 3PI WSdiff finished
• 11:20 MEH: ippc17 is down? appears so, no info on console. power-cycled and restarted pantasks
• 11:50 MEH: stopping stdscience for normal restart and will do ipp020 reboot now nightly science is finished
• 12:00 MEH: ipp020 has note in neb-host as problem machine to be in repair, but was neb-host up.. crash on 3/15 was similar to before when not in repair so placing back into repair
• 15:30 Serge: ippc05 and ippc06 are both down. Power-cycled. Nothing but the prompt on the console.
• 15:35 Serge: pantasks: stop, shutdown, start.server, setup, hostoff ipp020, run. Missing stack and detrend since c05 and c06 are recovering
• 16:10 Serge: stack and detrend restarted
• 18:30 MEH: stdscience pantasks in odd state, restarting. update in same state, setup not run?

Wednesday : YYYY.MM.DD

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD