PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.11.17

• 01:20 MEH: ippsXX in use for MD PV3 -- ~ippmd/stdscience/ptolemy.rc if need to turn off
• 12:50 MEH: going to start using ippx001-020 for MD processing rate tests, turning off slowly in stdlocal
  • ippx013 many faults -- Gavin fixed missing entries in autofs

Tuesday : 2014.11.18

• 07:30 MEH: stdsci underloading, will need a restart after nightly finishes along w/ summit and reg. removing update labels for now -- was well underloaded so restarted
  • lanl stdlocal also seems to be using storage nodes when nightly not finished, 2x stacks on upp039 (only 8 cores as well..) was causing chips to run ~5x longer than normal.. manually turning storage nodes off..
• 09:45 MEH: nightly finally finished
• 09:30 MEH: ippsXX and ippxNNN off for ippmd, turned on in lanl stdlocal for SAS processing
• 10:55 EAM: restarted stdlocal with the following modified logic for loading:
  • storage hosts are either all on (daytime, 6AM HST - 7PM HST) or all off (nighttime)
  • chip-warp processing is blocked only if > 10k nightly science items are outstanding
  • ippxNNN nodes are loaded in 2 blocks: x0 and x1, corresponding to machines on the same PDU
  • ippxNNN and ippsNN nodes are not ON by default, but are loaded for ease of pcontrol operations
  • added the SAS.20141118 label to process the newest SAS run.
• 11:10 MEH: turning ippsXX and ippxNNN on then in stdlocal for SAS prioity
• 21:35 MEH: registration backed up >100 because of a fault? had to manually run

  regtool -updateprocessedimfile -exp_id 819547 -class_id XY75 -set_state pending_burntool -dbname gpc1
  

• 22:00 MEH: 100+ exposures behind, ippsXX off in stdlocal and into stdsci to catchup, then ippsXX into MD since SAS stacks ~done
• 23:00 MEH: stdlocal also stop until caught up -- nightly rate seems to increase, too many chip-warp jobs?
• 00:45 MEH: mostly caught up, stdlocal back on w/o ippsXX nodes

Wednesday : 2014.11.19

• 00:55 MEH: ippsXX in use by ~ippmd/stdscience/ptolemy.rc
• 05:05 EAM: ipp036 crashed, nothing on console. rebooting
• 05:15 EAM: stopping & restarting stdlocal. xnodes on, snodes off
• 07:30 MEH: registration stuck again -- 174 exposures behind. oddly chip processing seems to have stopped ~0500 on the ippmonitor plots..
  • needed to manually run

    regtool -updateprocessedimfile -exp_id 820015 -class_id XY55 -set_state pending_burntool -dbname gpc1
    

  • agaiin flipping on ippsXX nodes in stdsci and stdlocal off

Thursday : YYYY.MM.DD

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD