PS1 IPP Czar Logs for the week 2014.11.03 -- 2014.11.09

(Up to PS1 IPP Czar Logs)

Monday : 2014.11.03

• 16:00 HAF : removed ipp008 from the hosts input file in ippconfig ( gene has declared it too flaky to be used)
• 20:30 EAM : added ipp008 to the ignore list (to avoid overloading the machine); restarting all servers
• 20:40 EAM : restarting stdlocal as well. I've also added a 'storage.hosts.half.on,storage.hosts.half.off' pair so we can keep the storage hosts somewhat loaded.

Tuesday : 2014.11.04

• 11:48 Bill : started pantasks from ~ippsky/staticsky using the hi memory nodes to run full force tests on part of the SAS area. Turned them off in stdlocal
• 21:55 EAM : the processing on ippsky/staticsky on himem nodes appears to be done, so I'm putting them back in stdlocal for the night.
• 10:00 MEH: MD needs nodes for processing, will be using them (to be clear himem=ippsXX nodes) then off and on -- took a long time for the stacks to clear... finished for a bit so now back into lanl stdlocal

Wednesday : 2014.11.05

• 08:40 MEH: ippsXX nodes off in lanl stdlocal -- now on again
• 13:40 MEH: ippsXX nodes off in lanl stdlocal -- now on again in stdlocal
• 15:03 Bill: added label sas.fftest.20141104 to the distribution pantasks
• 15:26 EAM : stopping stdlocal stacks to restart the pantasks
• 16:20 MEH: set stdlocal to stop as wasn't loading anything anyways (needs a restart) -- also taking the ippsXX nodes for MD again now
• 17:15 MEH: ippsXX nodes back on in stdlocal
• 20:15 MEH: nightly processing hanging up.. many timeouts.. stopping stdlocal, removing update labels
  • stdsci chips picked up fairly quickly, update labels causing problem for stdsci?
  • ippdb01 gpc1 probably needs to be restarted in morning
  • stsci00 also started heavy load ~2000 and may be related to problem as well
  • odd/unannounced script (do_check.pl) running on ippc18 during nightly, does it need to be?
  • still seeing many timeouts in summitcopy, stdlocal off again -- summitcopy+registration timeouts much less as often..
• 21:05 MEH: stdlocal run again -- things seem better.. so something else unknown running?

Thursday : 2014.11.06

• 05:56 Bill restarted the pstamp pantasks
• 07:45 Bill: stdscience was idle so added the update labels back in. The chip poll task seems to be taking forever. This is likely due to the

fact that so many chipRuns are in state update. If the situation doesn't improve I will set them to goto_cleaned.

Friday : 2014.11.07

Saturday : 2014.11.08

Sunday : 2014.11.09