== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2014.11.17 ===
 * 01:20 MEH: ippsXX in use for MD PV3 -- ~ippmd/stdscience/ptolemy.rc if need to turn off
 * 12:50 MEH: going to start using ippx001-020 for MD processing rate tests, turning off slowly in stdlocal
  * ippx013 many faults -- Gavin fixed missing entries in autofs 

=== Tuesday : 2014.11.18 ===
 * 07:30 MEH: stdsci underloading, will need a restart after nightly finishes along w/ summit and reg. removing update labels for now -- was well underloaded so restarted
  * lanl stdlocal also seems to be using storage nodes when nightly not finished, 2x stacks on upp039 (only 8 cores as well..) was causing chips to run ~5x longer than normal.. manually turning storage nodes off.. 
 * 09:45 MEH: nightly finally finished
 * 09:30 MEH: ippsXX and ippxNNN off for ippmd, turned on in lanl stdlocal for SAS processing
 * 10:55 EAM: restarted stdlocal with the following modified logic for loading:
    * storage hosts are either all on (daytime, 6AM HST - 7PM HST) or all off (nighttime)
    * chip-warp processing is blocked only if > 10k nightly science items are outstanding
    * ippxNNN nodes are loaded in 2 blocks: x0 and x1, corresponding to machines on the same PDU
    * ippxNNN and ippsNN nodes are not ON by default, but are loaded for ease of pcontrol operations
    * added the SAS.20141118 label to process the newest SAS run.
 * 11:10 MEH: turning ippsXX and ippxNNN on then in stdlocal for SAS prioity
 * 21:35 MEH: registration backed up >100 because of a fault? had to manually run
{{{
regtool -updateprocessedimfile -exp_id 819547 -class_id XY75 -set_state pending_burntool -dbname gpc1
}}}
 * 22:00 MEH: 100+ exposures behind, ippsXX off in stdlocal and into stdsci to catchup, then ippsXX into MD since SAS stacks ~done
 * 23:00 MEH: stdlocal also stop until caught up -- nightly rate seems to increase, too many chip-warp jobs?
 * 00:45 MEH: mostly caught up, stdlocal back on w/o ippsXX nodes

=== Wednesday : 2014.11.19 ===

 * 00:55 MEH: ippsXX in use by ~ippmd/stdscience/ptolemy.rc
 * 05:05 EAM: ipp036 crashed, nothing on console.  rebooting

=== Thursday : YYYY.MM.DD ===

=== Friday : YYYY.MM.DD ===

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===