== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2016.03.14 ===
 * 04:38 MEH: registration backing up it looks like because exposure o7461g0601 stalled in summitcopy ~1500s, manually clearing..
 * 08:30 EAM: processing is done for the night.  i'm restarting relastro clients
 * 13:55 MEH: stdsci and cleanup to stop while changing label to OSS.nightlyscience.mopshold for both sets of R13S2 WW diffims from OSS.20160314 (normally would be cleaned tomorrow afternoon) -- Serge will let us know when can clean up
 * 16:00 MEH: regular restart of pantasks now that cleanup is finished
 * 22:18 MEH: the current 6x group x2+x3 gives stdsci ~614 active nodes is probably excessive (was ~530? and well overpowered), particularly w/ the odd behavior the xnodes seems to have.
=== Tuesday : 2016.03.15 ===
 * 07:10 MEH: clear OSS fault 4 warp (cannot build growth curve) problem
{{{
warptool -dbname gpc1 -updateskyfile -set_quality 42 -skycell_id skycell.1115.041  -warp_id 1703288 -fault 0
}}}
 * using only 3x group x2+3 loaded instead of 6x and processing rate kept up to the normal ~60 exp/hr
 * 07:30 MEH: 73 WWdiff .multi type running, thought these were going to be turned off..
=== Wednesday : 2016-03-16 ===
 * 13:30 MEH: sending OSS.nightlyscience.mopshold, OSS.20160314 to cleanup now that MOPS is finished with them
 * 15:45 CZW Restarting ipptest/replication to cull down over-replicated copies.  This will then replicate copies that were missed in the previous iteration.
 * 15:55 CZW Restarting ipp/* pantasks for nightly reboot.  I am also installing the regtool update that prevents the sunset/burntool issue from the weekend from popping up in the future.
=== Thursday : 2016.03.17 ===
 * 01:25 MEH: (email notes to add) -- for o7464g0319o ota14
 * 01:42 MEH: watching another exposure stalled on check_burntool case o7464g0395o ota14, no error in registration log but been >10 min so far -- will manually clear if doesn't w/in 35 minutes -- didn't clear w/in 35 minutes and registration getting ~100 behind again... manually clearing..
=== Friday : 2016.03.18 ===
 * 20:45 EAM : stopping pantasks for restart

=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===