== PS1 IPP Czar Logs for the week 2018.05.21 - 2018.05.28 == 
[[PageOutline]]

(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])

=== Monday : 2018.05.21 ===
  * 08:45 EAM : I needed to manually run the end-of-night version of nightly_science.pl to get the desperate diffs to trigger for gpc2:
{{{
nightly_science.pl --force_registration --verbose --queue_diffs --dbname gpc2 --camera GPC2
}}}

  * 10:00 EAM : pole reprocessing stacks are continuing to run under ~ipptest/stacks/ptolemy.rc. It is using 3x x0 and x1.  I've also added 3x c2 (ippc26 - c63) to get some statistics on the network issues reported here.

 * MEH: moving s5,s6,s7,s8 datanodes from gpc1:stdscience to gpc1:stdscience_ws pantasks where the extra power is actually needed now that more WSdiff possible w/ new PSNSC.wref.20170707 stacks
 * MEH: ipp067-097 still in repair so put up -- ipp084 BBU failed so leave in repair
 * MEH: locally setting gpc1:nightly_science.config to not automatically cleanup warps now (RETENTION_TIME -1)
 * MEH: Angie notes odd image issue @2231 HST-- turn off WSdiff distribution until checked in morning to avoid a tadpole-like flood
=== Tuesday : 2018.05.22 ===
 * MEH: PS1 observer log noted the specific exposures having telescope issue --  WWdiff visit alternatives that could be made 
{{{
o8260g0079o OSSR.R13N5.6.Q.i ps1_22_5166 visit 1 -- v23?
o8260g0080o OSSR.R13N5.6.Q.i ps1_22_5143 visit 1 -- v23?
o8260g0085o OSSR.R13N5.6.Q.i ps1_22_5144 visit 1 -- no alt
o8260g0136o OSSR.R13N5.6.Q.i ps1_22_5144 visit 4 -- no alt
o8260g0103o OSSR.R13N5.6.Q.i ps1_22_5123 visit 2 -- no alt
o8260g0120o OSSR.R13N5.6.Q.i ps1_22_5123 visit 3 -- no alt
o8260g0137o OSSR.R13N5.6.Q.i ps1_22_5123 visit 4 -- v23?
o8260g0142o OSSR.R13N5.6.Q.i ps1_22_5105 visit 4 -- v23?
o8260g0143o OSSR.R13N5.6.Q.i ps1_22_5124 visit 4 -- v23?
o8260g0144o OSSR.R13N5.6.Q.i ps1_22_5107 visit 4 -- v23?
o8260g0145o OSSR.R13N5.6.Q.i ps1_22_4986 visit 4 -- v23?
o8260g0146o OSSR.R13N5.6.Q.i ps1_22_5125 visit 4 -- v23?
o8260g0258o OSSR.R14N6.6.Q.i ps1_22_4960 visit 4 -- v23?
o8260g0259o OSSR.R14N6.6.Q.i ps1_22_5082 visit 4 -- v23?
o8260g0326o OSSR.R15N6.6.Q.i ps1_22_4948 visit 2 -- v13?
}}}
 * MEH: PS1 WS distribution turned back on in morning after exposures with telescope issue had .telejump added to label & data_group for chip,cam,warp,WWdiff,WSdiff entries -- unfortunately over half had already been distributed to QUB by the time learned of problem and turned off distribution
 * MEH: PS1 WS distribution disabled again until tomorrow morning after checking/renaming any with odd exposures
 * MEH: running new PS1 test chunk WSdiff for MOPS in normal ~ippqub/stdscience_ws
 * MEH: running PS2 exposure/quad WSdiff tests for MOPS in normal ~ippps2/stdscience
=== Wednesday : 2018.05.23 ===
 * MEH: remaining WS distribution finished in morning after observers log noted no telescope jump issues -- leave off for couple more nights to monitor
 * MEH: broken updates for MOPS stamps -- o6937g0284o,o6501g0503o basic chip-cam-warp reprocessing done again so MOPS can get their data while looking into problem
  * yet another broken warp file -- o6937g0284o, skycell.2088.054 warp state cleaned with some skycells data_state full, but with broken LAP.PV3 files from the stsci shuffle as described again early last week
  * memory fault in o6501g0503o, skycell.2290.009 on ippc98 -- revert seemed to clear fine, unclear if fault 4 isn't auto-reverted in update/pstamp (had fault count=2, but unclear if revert does only fault 2 or 2-4) or if just needed a few tries like when happens in normal nightly processing
  * these updates highlighted another oddity where a skycell has a data_state cleaned and fault 2 due to a past odd update or manual DB modification -- skycell seems to clear and update fine when made to update normally, but does show up on the ippmonitor page as a fault (essentially a red herring)
=== Thursday : 2018.05.24 ===
 * MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
=== Friday : 2018.05.25 ===
 * MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
=== Saturday : YYYY.MM.DD ===

=== Sunday : YYYY.MM.DD ===