Changes between Version 15 and Version 16 of PS1_IPP_Czarlog_20180521

Timestamp:

May 28, 2018, 5:49:02 PM (8 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20180521

@@ -51 +51 @@
   * large number of MOPS pstamp request with various fault cases -- turn cleanup off for ps_ud% and pstamp until sorted out -- set pstamp/input pstamp.cleanup.off and add macro pstamp.update.cleanup.off to also turn off the ps_ud% cleanup
   * large number of MOPS pstamp requests faulting, looks like the erroneous state/data_state cleaned/full with broken pixel files has not been tested/reset for updates -- start testing and looks like setting state goto_cleaned will not working due to a block in warp cleanup to not do anything if the .mdc file is broken and the warp data_state will need to be directly set to cleaned instead (done with warp_id,skycell_id DB update warpSkyfile set data_state="cleaned" ...)
-  * setup for reprocessing variously broken older processing versions (ie ecliptic.rp) -- 
+  * setup for reprocessing variously broken/missing files from older processing versions (ie ecliptic.rp), estimate ~10k exposures (~1k for recent MOPS requests) will need this that not done in PSNSC.wref.2016xxxx -- label/data_group PSNSC.wfix.20180525/PSNSC.wfix.20180525.<orig data_group>
 
 === Friday : 2018.05.25 ===
  * MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
   * finished creating listing of warp state/data_state cleaned/full with broken files for LAP.PV3 --  
-  * another case of chip state/data_state cleaned/full with broken/missing LAP.PV3 files not being fixed with code changes preventing warps from updating -- looks like it also has the block in cleanup of the chip if .mdc is missing and needs the data_state directly set cleaned in the DB
+  * another case of chip state/data_state cleaned/full with broken/missing LAP.PV3 files not being fixed with code changes preventing warps from updating -- looks like it also has the block in cleanup of the chip if .mdc is missing and needs the data_state directly set cleaned in the DB -- 
+  * broken/missing ecliptic.rp files in chip update missing .psf triggering psphot rerun -- looks like triggered from PSPHOT.EXT.DIFF.ALTERNATE in psphot.config not in old processing -- just setting a quality bad case for the chip isn't desired as it would prevent MOPS from getting the pixels they are asking for and reprocessing will continue on this data
+{{{ ppImage: psphotSourceSize.c:122: psphotSourceSizeReadout: Assertion `status' failed. }}}
+  * PSNSC.wfix.20180525 chip_id 2074262 odd error -- 
 
  * MEH: setting ipp067-126 for skycell data local in ipphosts.mhpcc.config now that those nodes have been in neb-host up state for few days to really test data io use there with all the MOPS updates and reprocessing -- orig config can be copied back if problem
  * MEH: ganglia not reporting for ippc17,db03,c18,c27,c127,x052,x071,b06,b09 -- gmond restart cleared 
- * MEH: setup for mops to also use MOPS.2 for large update requests 
+ * MEH: setup for MOPS to also use MOPS.2 for large update requests so that they won't block nightly processing or their small, fast requests
 
 === Saturday : 2018.05.26 ===
@@ -66 +69 @@
   * using the faulted exposures, sent warp to cleanup in order to uncover any missed warp state/data_state cleaned/full cases with broken/missing files -- 
   * another large batch of broken burn.tbl to repair -- for broken neb key need to manually neb-mv .bad before running ipp_apply_burntool_fix.pl otherwise the neb key error breaks the repair, 0 vs 13 case looks to be caused here by 0 size burn.tbl file and appears to regenerate okay using ipp_apply_burntool_fix.pl -- looks like ippb00 being down is the primary issue, but is only an issue because the .burn.tbl tables were not replicated when ipp051 was decomissioned (>90% wanted second copy from there)
-  * 
-  * missing SMF and camera mask files -- generally reprocessed the following exposures with label/data_group
+   * reprocessing ecliptic.rp also being hit by missing burn.tbl and needed to be cleared of course  
+  * 22 more broken/missing chip pixel files not cleaning/updating properly until set data_state to cleaned directly -- o6994g0386o.830104.ch.1462236.XY11, o7009g0625o.840293.ch.1462244.XY42 etc o7009g0611o.840278.ch.1462649.XY71
+  * missing SMF and camera mask files -- generally reprocessed the following exposures with label/data_group mopsfix/mops.fixbrokenLAP.20180323 -- o6217g0057o 1352549, 
+   
 
  * MEH: many many nodes wanted data from ipp054 (detrends?) and created a terrible bottleneck.. set neb-host down to prevent additional connections until large updates finish
@@ -73 +78 @@
 === Sunday : 2018.05.27 ===
  * MEH: continue working on cleaning up broken issues for MOPS getting warp stamps -- make no changes to system without discussing with me first
-  * pstampDependent appears to also slowly compound the extensive broken/missing problem from the stsci file shuffle -- once triggered it prevents a broken dependent from returning a stamp (fault 25)
+  * pstampDependent appears to also slowly compound the extensive broken/missing problem from the stsci file shuffle -- once triggered it prevents a broken dependent from returning a stamp (fault 25), manually setting with pstamptool to state stop (normal state is full) which avoids it from being a block and keeps it as a note how often this case is being triggered
+  * 
 
-