PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2013.03.18

mark is czar

• 01:00 MEH: putting 1x compute3 back on in deepstack for the staticsky outside of the GP for now. leaving normal allocation of 1x compute3 in stack now that mostly caught up, will have nightly science stacks to do. only 2/4x compute3 in stdscience manually for now.
• Serge looked at mysql on ipp0012 and ippdb02, restarted and slowly catching up.
• 15:20 MEH: odd, processing almost came to halt.. looks like stdscience pantasks is underloaded.. slowly coming back up. might as well do regular pantasks restart

Tuesday : 2013.03.19

mark is czar

• 00:30 MEH: datastore having intermittent proxy errors
• 07:05 MEH: nightly science still processing, turning chip.off to push the nightly warps out. looks like an OSS and a few other publishing jobs had fault and revert not automatic.
• 09:20 MEH: warps+diffs only running, something limiting processing (stsci06/07 over-targeted?) -- 3PI nightly now mostly finished, 3PI WSdiff a good number to do still
• 11:15 MEH: 3PI WSdiff finished
• 11:20 MEH: ippc17 is down? appears so, no info on console. power-cycled and restarted pantasks
• 11:50 MEH: stopping stdscience for normal restart and will do ipp020 reboot now nightly science is finished
• 12:00 MEH: ipp020 has note in neb-host as problem machine to be in repair, but was neb-host up.. crash on 3/15 was similar to before when not in repair so placing back into repair
• 15:30 Serge: ippc05 and ippc06 are both down. Power-cycled. Nothing but the prompt on the console.
• 15:35 Serge: pantasks: stop, shutdown, start.server, setup, hostoff ipp020, run. Missing stack and detrend since c05 and c06 are recovering
• 16:10 Serge: stack and detrend restarted
• 18:30 MEH: stdscience pantasks in odd state, restarting. update in same state, setup not run?
• 22:15: MEH: looks like ippc17 crashed again ~1hr ago, looking into -- nothing on console, rebooting again..
  • date came up wrong again on power-cycle. going to move pstamp and update to another host -- pstamp+update on ippc06 for now, detrend off and deepstack off
  • manually set date to w/in ~1sec (as per Bill's email in the morning, need to do this before starting pstamp where ever it is)

Wednesday : 2013.03.20

• 09:30 and later Bills has disabled dvodist access because it is overloading the apache server some way and causing data store errors.
• 09:50 Bill started pantasks as bills from ~bills/relgroup that is simply looking for completed lapGroups and queuing staticsky runs. The pantasks is running on ippc17
• 10:15 Bill started deepstack pantasks
• 11:05 Bill restarted apache on ipp017 with the number of clients reduced to 127 This is configured in /etc/apache2/modules.d/00_mpm.conf
• 11:50 Bill stopped apache on ippc02 moving /tmp/nebulous log to a partition that is not full
• 3:50pm heather added ssp to summitcopy
• 16:33 restarted update and pstamp pantasks because they were down ippc06 crashed and was rebooted.

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD