PS1 IPP Czar Logs for the week 2018.05.21 - 2018.05.28

(Up to PS1 IPP Czar Logs)

Monday : 2018.05.21

• 08:45 EAM : I needed to manually run the end-of-night version of nightly_science.pl to get the desperate diffs to trigger for gpc2:

  nightly_science.pl --force_registration --verbose --queue_diffs --dbname gpc2 --camera GPC2
  

• 10:00 EAM : pole reprocessing stacks are continuing to run under ~ipptest/stacks/ptolemy.rc. It is using 3x x0 and x1. I've also added 3x c2 (ippc26 - c63) to get some statistics on the network issues reported here.

• MEH: moving s5,s6,s7,s8 datanodes from gpc1:stdscience to gpc1:stdscience_ws pantasks where the extra power is actually needed now that more WSdiff possible w/ new PSNSC.wref.20170707 stacks
• MEH: ipp067-097 still in repair so put up -- ipp084 BBU failed so leave in repair
• MEH: locally setting gpc1:nightly_science.config to not automatically cleanup warps now (RETENTION_TIME -1)
• MEH: Angie notes odd image issue @2231 HST-- turn off WSdiff distribution until checked in morning to avoid a tadpole-like flood

Tuesday : 2018.05.22

• MEH: PS1 observer log noted the specific exposures having telescope issue -- WWdiff visit alternatives that could be made

  o8260g0079o OSSR.R13N5.6.Q.i ps1_22_5166 visit 1 -- v23?
  o8260g0080o OSSR.R13N5.6.Q.i ps1_22_5143 visit 1 -- v23?
  o8260g0085o OSSR.R13N5.6.Q.i ps1_22_5144 visit 1 -- no alt
  o8260g0136o OSSR.R13N5.6.Q.i ps1_22_5144 visit 4 -- no alt
  o8260g0103o OSSR.R13N5.6.Q.i ps1_22_5123 visit 2 -- no alt
  o8260g0120o OSSR.R13N5.6.Q.i ps1_22_5123 visit 3 -- no alt
  o8260g0137o OSSR.R13N5.6.Q.i ps1_22_5123 visit 4 -- v23?
  o8260g0142o OSSR.R13N5.6.Q.i ps1_22_5105 visit 4 -- v23?
  o8260g0143o OSSR.R13N5.6.Q.i ps1_22_5124 visit 4 -- v23?
  o8260g0144o OSSR.R13N5.6.Q.i ps1_22_5107 visit 4 -- v23?
  o8260g0145o OSSR.R13N5.6.Q.i ps1_22_4986 visit 4 -- v23?
  o8260g0146o OSSR.R13N5.6.Q.i ps1_22_5125 visit 4 -- v23?
  o8260g0258o OSSR.R14N6.6.Q.i ps1_22_4960 visit 4 -- v23?
  o8260g0259o OSSR.R14N6.6.Q.i ps1_22_5082 visit 4 -- v23?
  o8260g0326o OSSR.R15N6.6.Q.i ps1_22_4948 visit 2 -- v13?
  

• MEH: PS1 WS distribution turned back on in morning after exposures with telescope issue had .telejump added to label & data_group for chip,cam,warp,WWdiff,WSdiff entries -- unfortunately over half had already been distributed to QUB by the time learned of problem and turned off distribution
• MEH: PS1 WS distribution disabled again until tomorrow morning after checking/renaming any with odd exposures
• MEH: running new PS1 test chunk WSdiff for MOPS in normal ~ippqub/stdscience_ws
• MEH: running PS2 exposure/quad WSdiff tests for MOPS in normal ~ippps2/stdscience

Wednesday : 2018.05.23

• MEH: remaining WS distribution finished in morning after observers log noted no telescope jump issues -- leave off for couple more nights to monitor
• MEH: broken updates for MOPS stamps -- o6937g0284o,o6501g0503o basic chip-cam-warp reprocessing done again so MOPS can get their data while looking into problem
  • yet another broken warp file -- o6937g0284o, skycell.2088.054 warp state cleaned with some skycells data_state full, but with broken LAP.PV3 files from the stsci shuffle as described again early last week
  • memory fault in o6501g0503o, skycell.2290.009 on ippc98 -- revert seemed to clear fine, unclear if fault 4 isn't auto-reverted in update/pstamp (had fault count=2, but unclear if revert does only fault 2 or 2-4) or if just needed a few tries like when happens in normal nightly processing
  • these updates highlighted another oddity where a skycell has a data_state cleaned and fault 2 due to a past odd update or manual DB modification -- skycell seems to clear and update fine when made to update normally, but does show up on the ippmonitor page as a fault (essentially a red herring)

Thursday : 2018.05.24

• MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
  • large number of MOPS pstamp request with various fault cases -- turn cleanup off for ps_ud% and pstamp until sorted out -- set pstamp/input pstamp.cleanup.off and add macro pstamp.update.cleanup.off to also turn off the ps_ud% cleanup
  • large number of MOPS pstamp requests faulting, looks like the erroneous state/data_state cleaned/full with broken pixel files has not been tested/reset for updates -- start testing and looks like setting state goto_cleaned will not working due to a block in warp cleanup to not do anything if the .mdc file is broken and the warp data_state will need to be directly set to cleaned instead (done with warp_id,skycell_id DB update warpSkyfile set data_state="cleaned" ...)
  • setup for reprocessing variously broken/missing files from older processing versions (ie ecliptic.rp), estimate ~10k exposures (~1k for recent MOPS requests) will need this that not done in PSNSC.wref.2016xxxx -- label/data_group PSNSC.wfix.20180525/PSNSC.wfix.20180525.<orig data_group>

Friday : 2018.05.25

• MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
  • finished creating listing of warp state/data_state cleaned/full with broken files for LAP.PV3 --
  • another case of chip state/data_state cleaned/full with broken/missing LAP.PV3 files not being fixed with code changes preventing warps from updating -- looks like it also has the block in cleanup of the chip if .mdc is missing and needs the data_state directly set cleaned in the DB --
  • broken/missing ecliptic.rp files in chip update missing .psf triggering psphot rerun -- looks like triggered from PSPHOT.EXT.DIFF.ALTERNATE in psphot.config not in old processing -- just setting a quality bad case for the chip isn't desired as it would prevent MOPS from getting the pixels they are asking for and reprocessing will continue on this data

    ppImage: psphotSourceSize.c:122: psphotSourceSizeReadout: Assertion `status' failed. 
    

    • PSNSC.wfix.20180525 chip_id 2074262 odd error --
    • all broken ecliptic.rp exposures moved out of way to label ps_ud_MOPSeclip
• MEH: setting ipp067-126 for skycell data local in ipphosts.mhpcc.config now that those nodes have been in neb-host up state for few days to really test data io use there with all the MOPS updates and reprocessing -- orig config can be copied back if problem
• MEH: ganglia not reporting for ippc17,db03,c18,c27,c127,x052,x071,b06,b09 -- gmond restart cleared
• MEH: setup for MOPS to also use MOPS.2 for large update requests so that they won't block nightly processing or their small, fast requests

Saturday : 2018.05.26

• MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
  • using the faulted exposures, sent warp to cleanup in order to uncover any missed warp state/data_state cleaned/full cases with broken/missing files --
  • another large batch of broken burn.tbl to repair -- for broken neb key need to manually neb-mv .bad before running ipp_apply_burntool_fix.pl otherwise the neb key error breaks the repair, 0 vs 13 case looks to be caused here by 0 size burn.tbl file and appears to regenerate okay using ipp_apply_burntool_fix.pl -- looks like ippb00 being down is the primary issue, but is only an issue because the .burn.tbl tables were not replicated when ipp051 was decomissioned (>90% wanted second copy from there)
    • reprocessing ecliptic.rp also being hit by missing burn.tbl and needed to be cleared of course
  • 22 more broken/missing chip pixel files not cleaning/updating properly until set data_state to cleaned directly -- o6994g0386o.830104.ch.1462236.XY11, o7009g0625o.840293.ch.1462244.XY42 etc o7009g0611o.840278.ch.1462649.XY71
  • missing SMF and camera mask files -- generally reprocessed for the following exposures through warp with label/data_group mopsfix/mops.fixbrokenLAP.20180323 -- leaving broken version in ps_ud_MOPS warp stage

    o6217g0057o.532854.cm.1352549.smf
    o5770g0457o.369703.cm.460324.smf
    o5846g0077o.406120.cm.1352530.XY23.mk.fits
    o5788g0253o.376382.cm.1352507.XY23.mk.fits
    o6584g0115o.666314.cm.1351561.XY22.mk.fits
    o5280g0502o.151314.cm.1403372.XY21.mk.fits
    o5636g0719o.311728.cm.1403592.XY41.mk.fits
    o5636g0719o.311728.cm.1403592.XY41.mk.fits
    o6172g0060o.517837.cm.1403811.XY74.mk.fits
    o6172g0083o.517860.cm.1403835.XY65.mk.fits
    o6006g0596o.467992.cm.1403922.XY15.mk.fits
    o5878g0045o.419357.cm.1229211.XY34.mk.fits
    o6049g0422o.483711.cm.1352379.XY67.mk.fits
    o6049g0434o.483722.cm.1352388.XY21.mk.fits
    o6584g0061o.666261.cm.1352440.XY36.mk.fits
    

• MEH: many many nodes wanted data from ipp054 (detrends in addition to raw?) and created a terrible bottleneck.. set neb-host down to prevent additional connections until large updates finish

Sunday : 2018.05.27

• MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
  • pstampDependent appears to also slowly compound the extensive broken/missing problem from the stsci file shuffle -- once triggered it prevents a broken dependent from returning a stamp (fault 25), manually setting with pstamptool to state stop (normal state is full) which avoids it from being a block and keeps it as a note how often this case is being triggered