Changes between Version 13 and Version 14 of PS1_IPP_Czarlog_20180521

Timestamp:

May 28, 2018, 5:24:59 PM (8 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20180521

@@ -39 +39 @@
  * MEH: running new PS1 test chunk WSdiff for MOPS in normal ~ippqub/stdscience_ws
  * MEH: running PS2 exposure/quad WSdiff tests for MOPS in normal ~ippps2/stdscience
+
 === Wednesday : 2018.05.23 ===
  * MEH: remaining WS distribution finished in morning after observers log noted no telescope jump issues -- leave off for couple more nights to monitor
@@ -45 +46 @@
   * memory fault in o6501g0503o, skycell.2290.009 on ippc98 -- revert seemed to clear fine, unclear if fault 4 isn't auto-reverted in update/pstamp (had fault count=2, but unclear if revert does only fault 2 or 2-4) or if just needed a few tries like when happens in normal nightly processing
   * these updates highlighted another oddity where a skycell has a data_state cleaned and fault 2 due to a past odd update or manual DB modification -- skycell seems to clear and update fine when made to update normally, but does show up on the ippmonitor page as a fault (essentially a red herring)
+
 === Thursday : 2018.05.24 ===
  * MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
+  * large number of MOPS pstamp request with various fault cases -- turn cleanup off for ps_ud% and pstamp until sorted out -- set pstamp/input pstamp.cleanup.off and add macro pstamp.update.cleanup.off to also turn off the ps_ud% cleanup
+  * large number of MOPS pstamp requests faulting, looks like the erroneous state/data_state cleaned/full with broken pixel files has not been tested/reset for updates -- start testing and looks like setting state goto_cleaned will not working due to a block in warp cleanup to not do anything if the .mdc file is broken and the warp data_state will need to be directly set to cleaned instead (done with warp_id,skycell_id DB update warpSkyfile set data_state="cleaned" ...)
+  * setup for reprocessing variously broken older processing versions (ie ecliptic.rp) -- 
+
 === Friday : 2018.05.25 ===
  * MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
+  * finished creating listing of warp state/data_state cleaned/full with broken files for LAP.PV3 --  
+  * another case of chip state/data_state cleaned/full with broken/missing LAP.PV3 files not being fixed with code changes preventing warps from updating -- looks like it also has the block in cleanup of the chip if .mdc is missing and needs the data_state directly set cleaned in the DB
+
  * MEH: setting ipp067-126 for skycell data local in ipphosts.mhpcc.config now that those nodes have been in neb-host up state for few days to really test data io use there with all the MOPS updates and reprocessing -- orig config can be copied back if problem
  * MEH: ganglia not reporting for ippc17,db03,c18,c27,c127,x052,x071,b06,b09 -- gmond restart cleared 
  * MEH: setup for mops to also use MOPS.2 for large update requests 
+
 === Saturday : 2018.05.26 ===
  * MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
+  * using the faulted exposures, sent warp to cleanup in order to uncover any missed warp state/data_state cleaned/full cases with broken/missing files -- 
+  * another large batch of broken burn.tbl to repair -- for broken neb key need to manually neb-mv .bad before running ipp_apply_burntool_fix.pl otherwise the neb key error breaks the repair, 0 vs 13 case looks to be caused here by 0 size burn.tbl file and appears to regenerate okay using ipp_apply_burntool_fix.pl
+  * 
+  * missing SMF and camera mask files -- generally reprocessed the following exposures with label/data_group
+
  * MEH: many many nodes wanted data from ipp054 (detrends?) and created a terrible bottleneck.. set neb-host down to prevent additional connections until large updates finish
+
 === Sunday : 2018.05.27 ===
  * MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
+  * pstampDependent appears to also slowly compound the extensive broken/missing problem from the stsci file shuffle -- once triggered it prevents a broken dependent from returning a stamp (fault 25)
+
+