PS1 IPP Czar Logs for the week 2018.05.21 - 2018.05.28

(Up to PS1 IPP Czar Logs)

Monday : 2018.05.21

• 08:45 EAM : I needed to manually run the end-of-night version of nightly_science.pl to get the desperate diffs to trigger for gpc2:

  nightly_science.pl --force_registration --verbose --queue_diffs --dbname gpc2 --camera GPC2
  

• 10:00 EAM : pole reprocessing stacks are continuing to run under ~ipptest/stacks/ptolemy.rc. It is using 3x x0 and x1. I've also added 3x c2 (ippc26 - c63) to get some statistics on the network issues reported here.

• MEH: moving s5,s6,s7,s8 datanodes from gpc1:stdscience to gpc1:stdscience_ws pantasks where the extra power is actually needed now that more WSdiff possible w/ new PSNSC.wref.20170707 stacks
• MEH: ipp067-097 still in repair so put up -- ipp084 BBU failed so leave in repair
• MEH: locally setting gpc1:nightly_science.config to not automatically cleanup warps now (RETENTION_TIME -1)
• MEH: Angie notes odd image issue @2231 HST-- turn off WSdiff distribution until checked in morning to avoid a tadpole-like flood

Tuesday : 2018.05.22

• MEH: PS1 observer log noted the specific exposures having telescope issue -- WWdiff visit alternatives that could be made

  o8260g0079o OSSR.R13N5.6.Q.i ps1_22_5166 visit 1 -- v23?
  o8260g0080o OSSR.R13N5.6.Q.i ps1_22_5143 visit 1 -- v23?
  o8260g0085o OSSR.R13N5.6.Q.i ps1_22_5144 visit 1 -- no alt
  o8260g0136o OSSR.R13N5.6.Q.i ps1_22_5144 visit 4 -- no alt
  o8260g0103o OSSR.R13N5.6.Q.i ps1_22_5123 visit 2 -- no alt
  o8260g0120o OSSR.R13N5.6.Q.i ps1_22_5123 visit 3 -- no alt
  o8260g0137o OSSR.R13N5.6.Q.i ps1_22_5123 visit 4 -- v23?
  o8260g0142o OSSR.R13N5.6.Q.i ps1_22_5105 visit 4 -- v23?
  o8260g0143o OSSR.R13N5.6.Q.i ps1_22_5124 visit 4 -- v23?
  o8260g0144o OSSR.R13N5.6.Q.i ps1_22_5107 visit 4 -- v23?
  o8260g0145o OSSR.R13N5.6.Q.i ps1_22_4986 visit 4 -- v23?
  o8260g0146o OSSR.R13N5.6.Q.i ps1_22_5125 visit 4 -- v23?
  o8260g0258o OSSR.R14N6.6.Q.i ps1_22_4960 visit 4 -- v23?
  o8260g0259o OSSR.R14N6.6.Q.i ps1_22_5082 visit 4 -- v23?
  o8260g0326o OSSR.R15N6.6.Q.i ps1_22_4948 visit 2 -- v13?
  

• MEH: PS1 WS distribution turned back on in morning after exposures with telescope issue had .telejump added to label & data_group for chip,cam,warp,WWdiff,WSdiff entries -- unfortunately over half had already been distributed to QUB by the time learned of problem and turned off distribution
• MEH: PS1 WS distribution disabled again until tomorrow morning after checking/renaming any with odd exposures
• MEH: running new PS1 test chunk WSdiff for MOPS in normal ~ippqub/stdscience_ws
• MEH: running PS2 exposure/quad WSdiff tests for MOPS in normal ~ippps2/stdscience

Wednesday : 2018.05.23

• MEH: remaining WS distribution finished in morning after observers log noted no telescope jump issues -- leave off for couple more nights to monitor
• MEH: broken updates for MOPS stamps -- o6937g0284o,o6501g0503o basic chip-cam-warp reprocessing done again so MOPS can get their data while looking into problem
  • yet another broken warp file -- o6937g0284o, skycell.2088.054 warp state cleaned with some skycells data_state full, but with broken LAP.PV3 files from the stsci shuffle as described again early last week
  • memory fault in o6501g0503o, skycell.2290.009 on ippc98 -- revert seemed to clear fine, unclear if fault 4 isn't auto-reverted in update/pstamp (had fault count=2, but unclear if revert does only fault 2 or 2-4) or if just needed a few tries like when happens in normal nightly processing
  • these updates highlighted another oddity where a skycell has a data_state cleaned and fault 2 due to a past odd update or manual DB modification -- skycell seems to clear and update fine when made to update normally, but does show up on the ippmonitor page as a fault (essentially a red herring)

Thursday : 2018.05.24

• MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
  • large number of MOPS pstamp request with various fault cases -- turn cleanup off for ps_ud% and pstamp until sorted out -- set pstamp/input pstamp.cleanup.off and add macro pstamp.update.cleanup.off to also turn off the ps_ud% cleanup
  • large number of MOPS pstamp requests faulting, looks like the erroneous state/data_state cleaned/full with broken pixel files has not been tested/reset for updates -- start testing and looks like setting state goto_cleaned will not working due to a block in warp cleanup to not do anything if the .mdc file is broken and the warp data_state will need to be directly set to cleaned instead (done with warp_id,skycell_id DB update warpSkyfile set data_state="cleaned" ...)
  • setup for reprocessing variously broken older processing versions (ie ecliptic.rp) --

Friday : 2018.05.25

• MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
  • finished creating listing of warp state/data_state cleaned/full with broken files for LAP.PV3 --
  • another case of chip state/data_state cleaned/full with broken/missing LAP.PV3 files not being fixed with code changes preventing warps from updating -- looks like it also has the block in cleanup of the chip if .mdc is missing and needs the data_state directly set cleaned in the DB

• MEH: setting ipp067-126 for skycell data local in ipphosts.mhpcc.config now that those nodes have been in neb-host up state for few days to really test data io use there with all the MOPS updates and reprocessing -- orig config can be copied back if problem
• MEH: ganglia not reporting for ippc17,db03,c18,c27,c127,x052,x071,b06,b09 -- gmond restart cleared
• MEH: setup for mops to also use MOPS.2 for large update requests

Saturday : 2018.05.26

• MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
  • using the faulted exposures, sent warp to cleanup in order to uncover any missed warp state/data_state cleaned/full cases with broken/missing files --
  • another large batch of broken burn.tbl to repair -- for broken neb key need to manually neb-mv .bad before running ipp_apply_burntool_fix.pl otherwise the neb key error breaks the repair, 0 vs 13 case looks to be caused here by 0 size burn.tbl file and appears to regenerate okay using ipp_apply_burntool_fix.pl
  • missing SMF and camera mask files -- generally reprocessed the following exposures with label/data_group

• MEH: many many nodes wanted data from ipp054 (detrends?) and created a terrible bottleneck.. set neb-host down to prevent additional connections until large updates finish

Sunday : 2018.05.27

• MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
  • pstampDependent appears to also slowly compound the extensive broken/missing problem from the stsci file shuffle -- once triggered it prevents a broken dependent from returning a stamp (fault 25)