Summary of MDdeeptest_pixrej_20140403

(back to MediumDeepFields summary)

For a while it has been bothersome to find junk in the MD stacks through just simple visual inspection as would not be expected in N=60 (refstack) and definitely not for N>>60 (deepstack), and they can also come and go depending on the inputs used. This seems to be at least configs needing tuning, at worst code bugs.

A related issue also to be addressed is the ratty edge of the MD stacks due to poorer edge chips and smaller number of input pixels. Possible alternatives for a solution are

  • hard cut by radius field by field and filter -- difficult to implement
  • combine MD+3PI to help improve by including better 3PI pixels -- non-trivial setup and unknown config behavior for both types of data sets (MD and 3PI)
  • only include pixel if N input > some minimum, similar to the SAFE (N=2) option but with a larger N minimum -- edge may vary greatly between filters
  • often edge is dominated by SUSPECT mask pixels, set those to bad and reject -- potential to impact entire skycell (may be good since SUSPECT may actually be bad)
  • any others options?

The bugs found will apply to all stacks (MD and 3PI).

Changes are noted in the MDdeeptest_pixrej_20140403#ConfigurationMods section for the MD deep stacks only (so far, suspect will be necessary for refstacks but unclear how night stacks will configured as of yet).

Example Junk and Bugs

Large regions of high level, bad pixels ( md04s065i -- diffract spike chunks (1129,4256), the moth (2193,2062) ) -- bug

img mk wt num exp expwt

  • due to NAN/0 bug from unprotected denominator being 0, setting mean and sigma to NAN (then all inputs used...)

Excess hot pixels/CR-like ( md04s065i (4099,4651),(4752,4853) ) -- bug


  • due to bug, showing up more after NAN/0 bug fixed, sigma sometimes being set to a negative value resulting in final mean and sigma going NAN (then all inputs again used...)

Large region of low level, poor/bad pixels ( md04s013g (1090,5600), md09s013g (1090,5600) ) -- bug? much can be reduced by config mods

img mk num wt

  • greatly reduced by lowering THRESHOLD.MASK 0.5->0.01 (may need further modifications) or reject a set of entire inputs as poor/bad if can be identified.

Edge junk and webbing ( md04s013g (800,2850) ) -- bug? much can be reduced by config mods and added N input minimum per pixel (NMINPIX)

img mk num wt

  • webbing greatly reduced by lowering THRESHOLD.MASK 0.5->0.3 (may need further modifications, 0.01 even better)
  • edge junk also cleaned up by rejecting pixels with <20 inputs (NMINPIX 20). Based on the large rise of variance values still, could probably be increased 30-50 inputs generally.
img mk num wt

Other case of webbing (md09s012g (1421,4850) ) -- SUSPECT pixels bad not really help. Different cause, appears to be incomplete rejection of bright, scattered light. Bug? Lower threshold (THRESHOLD.MASK) helps, but likely need to reject a set of entire inputs as poor/bad.

img img (suspect bad)

Other streak-like features still (md09s046g (3235,3368) ) -- another bug? Lower threshold (THRESHOLD.MASK) helps but not completely.

img (thresh 0.3) img (thresh 0.1)

Edge of skycell junk -- strip along skycell with fair amount of junk ignored, kernel size (all stacks have this problem)

  • KERNEL.SIZE=20 pixels, MD.V3 skycell overlap is 320 pixels -- this area should just be entirely MASKED
img img (mask border 20 pixels)

Setting SUSPECT input pixels as bad -- general input loss per pixel ~1-5% weblike over the field but more often than not with shapes that are clearly poor (i.e. ghost rings)

img1 num1 img2 (suspect bad) num2 num2/num1

Concern with lower threshold (THRESHOLD.MASK 0.5 to 0.3 to 0.1) will over-clip input pixels particularly in brighter sources -- generally only 1-2% loss and 10-20% loss in brighter sources.

img1 (thresh 0.5) num1 img2 (thresh 0.3) num2 num2/num1

img1 (thresh 0.3) num1 img2 (thresh 0.1) num2 num2/num1

Full Field Test Samples

All stacks should be available via the Postage Stamp Server (PSS), if large or full sets are desired then they can be found as usual on the datastore under ps1-deeptest with the data_group name the same as the label. Warp examples should be obtainable via the PSS and will not be pushed to the distribution server.

MD04 PV2 warps -- the full test runs below used all available PV2 through the end of the survey (MD.PV2)

  • warp label: MD04.pv2.20130903 -- all filters
  • not all examples below use the full set of warps, many use only the same warps that went into MD04.cut2009deeptest.20131103 in order to provide a similar stack for comparison.

3PI exposures overlapping with MD04 -- span Apr. 2009 to Feb. 17 2014, however any exposures before June 2010 will be excluded just like the MD04 exposures (i.e., cut2009 condition described at MD04.pv2_stacktest.20130903#DeepStacks)

  • warp label: MD04.test3pistack.20140217 -- g-band only
  • (need to add total and used numbers)

MD04+3PI overlap test with just bug fixes to compare the edge quality -- will include additional PV2 warps through end of survey for MD04 and similarly 3PI though Feb. 2014

  • label: MD04_3PI.cut2009teststack.20140404 -- g-band only

MD04+3PI overlap test with bug fixes and config modifications to compare the edge quality -- will include same available warps as prior test

  • label: MD04_3PI.cut2009teststack.20140411 -- again g-band only
  • will take work to evaluate impact of config modifications between the 3PI and MD here.. -- likely will have to consider this option after much more testing

MD04 with bug fixes, config modifications, N>20 inputs per pixel to clean up the field edge -- can compare to MD04.cut2009deeptest.20131103 since PV2 inputs limited by date here through end of Sept. 2013

  • label: MD04.cut2009n20deeptest.20140413 -- i-band only, same inputs as MD04.cut2009deeptest.20131103

MD04 with bug fixes, smaller config modifications, hard coded N>30 inputs per pixel -- but other configs not changed as much from MD04.cut2009deeptest.20131103

  • label: MD04.cut2009n30deeptest.20140416 -- i-band only, same inputs as MD04.cut2009deeptest.20131103

MD09 example as a shifted center field case -- MD09.PV1_deepstack_tests and manual re-test showed a large amount of junk, so also set SUSPECT input pixels as BAD pixels, edge properties unknown so keep pixels N>10 inputs (NMINPIX)

  • label: MD09.cut2009n10suspectbaddeeptest.20140427 -- g-band only, full PV2 set after 6/2010 like cut2009
  • NMINPIX 10

MD09 example to try exclude extra junk -- NMINPIX now 30, adding a MASK.BLANKBORDER to mask the edge of the skycell since it goes through essentially no rejection process and often a pile of junk. Is set 20 pixels since the border is set in code to be KERNEL.SIZE in the convolution and overlap setup. No pixels are lost since the MD skycells have an overlap of 320 pixels, now they will overlap by only a mere 300 pixels.

  • label: MD09.cut2009n30th01suspectbadborder.deeptest.20140502 -- g-band only, full PV2 set after 6/2010 like cut2009
  • NMINPIX 30

MD04 example with changes made in the MD09 above and with imposed smaller target PSF -- limit the target PSF for convolution to check if smaller THRESHOLD.MASK has any problems

  • label: MD04.cut2009n30th01suspectbad7pix.deeptest.20140504 -- g-band only, same inputs as MD04.cut2009deeptest.20131103
  • NMINPIX 10

Configuration Mods

This should be done properly before the final version, but will take some time to do. Config params that impact the pixel rejection

COMBINE.ITER    F32     0.5             # Number of rejection iterations per input
COMBINE.REJ     F32     3.5             # Rejection threshold in combination (sigma)
COMBINE.SYS     F32     0.1             # Relative systematic error in combination
COMBINE.DISCARD F32     0.2             # Discard fraction for Olympic weighted mean

MASK.VAL        STR     MASK.VALUE,CONV.BAD,GHOST # Mask value of input bad pixels
MASK.BAD        STR     BLANK           # Mask value to give bad pixels
MASK.POOR       STR     CONV.POOR       # Mask value to give poor pixels

POOR.FRACTION   F32     0.25            # Maximum fraction of bad variance for poor pixels
THRESHOLD.MASK  F32     0.5             # Threshold for mask deconvolution (0..1)
IMAGE.REJ       F32     0.1             # Rejected pixel fraction threshold for rejecting entire image
VARIANCE        BOOL    TRUE            # Use variance in rejection?
SAFE            BOOL    TRUE            # Play safe when combining small number of values?

  • MASK.VAL move SUSPECT to list from MASK.SUSPECT
  • THRESHOLD.MASK lower to 0.1

New options added (defaults):

NMINPIX         S32     1              # Minimum input per pixel 
MASK.BLANKBORDER S32    0              # Mask blank border in final stack output
  • NMINPIX 30 appears to be a reasonable starting value, may be better if 40-50. The edges of the MD field is poor, 3PI overlap in that area will often be better.
  • MASK.BLANKBORDER 20 is same as KERNEL.SIZE in ppSub.config to exclude differently/un-masked pixel region

Unclear how a mix of MD+3PI will be impacted by config mods... a strong reason to not make MD+3PI stacks yet.