PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2014.11.17

• 01:20 MEH: ippsXX in use for MD PV3 -- ~ippmd/stdscience/ptolemy.rc if need to turn off
• 12:50 MEH: going to start using ippx001-020 for MD processing rate tests, turning off slowly in stdlocal
  • ippx013 many faults -- Gavin fixed missing entries in autofs

Tuesday : 2014.11.18

• 07:30 MEH: stdsci underloading, will need a restart after nightly finishes along w/ summit and reg. removing update labels for now -- was well underloaded so restarted
  • lanl stdlocal also seems to be using storage nodes when nightly not finished, 2x stacks on upp039 (only 8 cores as well..) was causing chips to run ~5x longer than normal.. manually turning storage nodes off..
• 09:45 MEH: nightly finally finished
• 09:30 MEH: ippsXX and ippxNNN off for ippmd, turned on in lanl stdlocal for SAS processing
• 10:55 EAM: restarted stdlocal with the following modified logic for loading:
  • storage hosts are either all on (daytime, 6AM HST - 7PM HST) or all off (nighttime)
  • chip-warp processing is blocked only if > 10k nightly science items are outstanding
  • ippxNNN nodes are loaded in 2 blocks: x0 and x1, corresponding to machines on the same PDU
  • ippxNNN and ippsNN nodes are not ON by default, but are loaded for ease of pcontrol operations
  • added the SAS.20141118 label to process the newest SAS run.
• 11:10 MEH: turning ippsXX and ippxNNN on then in stdlocal for SAS prioity
• 21:35 MEH: registration backed up >100 because of a fault? had to manually run

  regtool -updateprocessedimfile -exp_id 819547 -class_id XY75 -set_state pending_burntool -dbname gpc1
  

• 22:00 MEH: 100+ exposures behind, ippsXX off in stdlocal and into stdsci to catchup, then ippsXX into MD since SAS stacks ~done
• 23:00 MEH: stdlocal also stop until caught up -- nightly rate seems to increase, too many chip-warp jobs?
• 00:45 MEH: mostly caught up, stdlocal back on w/o ippsXX nodes

Wednesday : 2014.11.19

• 00:55 MEH: ippsXX in use by ~ippmd/stdscience/ptolemy.rc
• 05:05 EAM: ipp036 crashed, nothing on console. rebooting
• 05:15 EAM: stopping & restarting stdlocal. xnodes on, snodes off
• 07:30 MEH: registration stuck again -- 174 exposures behind. oddly chip processing seems to have run out of chips ~0400-0500 on the ippmonitor plots..
  • needed to manually run

    regtool -updateprocessedimfile -exp_id 820015 -class_id XY55 -set_state pending_burntool -dbname gpc1
    

  • agaiin flipping on ippsXX nodes in stdsci and stdlocal off
• 09:20 MEH: nightly mostly finished, ippsXX back to ippmd, ippxNNN back to stdlocal
• 09:30 MEH: failing skycell

  cannot build growth curve (psf model is invalid everywhere)
  
  warptool -dbname gpc1 -updateskyfile -fault 0 -set_quality 42 -warp_id 1201568 -skycell_id skycell.1787.029
  

• 12:45 MEH: ippmd will be taking ippxNNN x0 group from stdlocal (doesn't seem to be fully loaded)
• 19:30 MEH: another build up of reg faults
• 21:00 MEH: seeing chip jobs taking ~2x long than normal, network is reaching 3GB/s so turning 2x x1 off in stdlocal and 2x off all in ippmd
• 22:10 MEH: stdsci poll underloaded.. doing quick restart
• 23:00 MEH: looks like more stacks being run by stdlocal now, stack poll increased? chip times better probably from fewer chips overall being processed, could bump ippx nodes up possibly but cannot monitor right now

Thursday : 2014.11.20

• 04:10 EAM : ipp035 crashed, nothing on console, rebooting.

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD