PS1 IPP Czar Logs for the week 2018.05.21 - 2018.05.28

(Up to PS1 IPP Czar Logs)

Monday : 2018.05.21

• 08:45 EAM : I needed to manually run the end-of-night version of nightly_science.pl to get the desperate diffs to trigger for gpc2:

  nightly_science.pl --force_registration --verbose --queue_diffs --dbname gpc2 --camera GPC2
  

• 10:00 EAM : pole reprocessing stacks are continuing to run under ~ipptest/stacks/ptolemy.rc. It is using 3x x0 and x1. I've also added 3x c2 (ippc26 - c63) to get some statistics on the network issues reported here.

• MEH: moving s5,s6,s7,s8 datanodes from gpc1:stdscience to gpc1:stdscience_ws pantasks where the extra power is actually needed now that more WSdiff possible w/ new PSNSC.wref.20170707 stacks
• MEH: ipp067-097 still in repair so put up -- ipp084 BBU failed so leave in repair
• MEH: locally setting gpc1:nightly_science.config to not automatically cleanup warps now (RETENTION_TIME -1)
• MEH: Angie notes odd image issue @2231 HST-- turn off WSdiff distribution until checked in morning to avoid a tadpole-like flood

Tuesday : 2018.05.22

• MEH: PS1 observer log noted the specific exposures having telescope issue -- WWdiff visit alternatives that could be made

  o8260g0079o OSSR.R13N5.6.Q.i ps1_22_5166 visit 1 -- v23?
  o8260g0080o OSSR.R13N5.6.Q.i ps1_22_5143 visit 1 -- v23?
  o8260g0085o OSSR.R13N5.6.Q.i ps1_22_5144 visit 1 -- no alt
  o8260g0136o OSSR.R13N5.6.Q.i ps1_22_5144 visit 4 -- no alt
  o8260g0103o OSSR.R13N5.6.Q.i ps1_22_5123 visit 2 -- no alt
  o8260g0120o OSSR.R13N5.6.Q.i ps1_22_5123 visit 3 -- no alt
  o8260g0137o OSSR.R13N5.6.Q.i ps1_22_5123 visit 4 -- v23?
  o8260g0142o OSSR.R13N5.6.Q.i ps1_22_5105 visit 4 -- v23?
  o8260g0143o OSSR.R13N5.6.Q.i ps1_22_5124 visit 4 -- v23?
  o8260g0144o OSSR.R13N5.6.Q.i ps1_22_5107 visit 4 -- v23?
  o8260g0145o OSSR.R13N5.6.Q.i ps1_22_4986 visit 4 -- v23?
  o8260g0146o OSSR.R13N5.6.Q.i ps1_22_5125 visit 4 -- v23?
  o8260g0258o OSSR.R14N6.6.Q.i ps1_22_4960 visit 4 -- v23?
  o8260g0259o OSSR.R14N6.6.Q.i ps1_22_5082 visit 4 -- v23?
  o8260g0326o OSSR.R15N6.6.Q.i ps1_22_4948 visit 2 -- v13?
  

• MEH: PS1 WS distribution turned back on in morning after exposures with telescope issue had .telejump added to label & data_group for chip,cam,warp,WWdiff,WSdiff entries -- unfortunately over half had already been distributed to QUB by the time learned of problem and turned off distribution
• MEH: PS1 WS distribution disabled again until tomorrow morning after checking/renaming any with odd exposures
• MEH: running new PS1 test chunk WSdiff for MOPS in normal ~ippqub/stdscience_ws
• MEH: running PS2 exposure/quad WSdiff tests for MOPS in normal ~ippps2/stdscience

Wednesday : 2018.05.23

• MEH: remaining WS distribution finished in morning after observers log noted no telescope jump issues -- leave off for couple more nights to monitor
• MEH: broken updates for MOPS stamps -- o6937g0284o,o6501g0503o basic chip-cam-warp reprocessing done again so MOPS can get their data while looking into problem
  • yet another broken warp file -- o6937g0284o, skycell.2088.054 warp state cleaned with some skycells data_state full, but with broken LAP.PV3 files from the stsci shuffle as described again early last week
  • memory fault in o6501g0503o, skycell.2290.009 on ippc98 -- revert seemed to clear fine, unclear if fault 4 isn't auto-reverted in update/pstamp (had fault count=2, but unclear if revert does only fault 2 or 2-4) or if just needed a few tries like when happens in normal nightly processing
  • these updates highlighted another oddity where a skycell has a data_state cleaned and fault 2 due to a past odd update or manual DB modification -- skycell seems to clear and update fine when made to update normally, but does show up on the ippmonitor page as a fault (essentially a red herring)

Thursday : 2018.05.25

• MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first

Friday : YYYY.MM.DD

Saturday : YYYY.MM.DD

Sunday : YYYY.MM.DD