Summary of MDdeeptest_pixrej_20140403 (still under construction)

(back to MediumDeepFields summary)

For a while it has been bothersome to find junk in the MD stacks through just simple visual inspection as would not be expected in N=60 (refstack) and definitely not for N>>60 (deepstack), and they can also come and go depending on the inputs used. This seems to be at least configs needing tuning, at worst code bugs.

A related issue also to be addressed is the ratty edge of the MD stacks due to poorer edge chips and smaller number of input pixels. Possible alternatives for a solution are

• hard cut by radius field by field and filter -- difficult to implement
• combine MD+3PI to help improve by including better 3PI pixels -- non-trivial setup and unknown configs for both types of data sets
• only include pixel if N input > some minimum, similar to the SAFE (N=2) option but with a larger N minimum -- edge may vary greatly between filters
• often edge is dominated by SUSPECT mask pixels, set those to bad and reject -- potential to impact entire skycell (may be good since SUSPECT may actually be bad)
• any others options?

---

Example Junk and Bugs

Large regions of high level, bad pixels (s065i -- diffract spike chunks [1129,4256], the moth []) -- bug

img

mk

wt

num

exp

expwt

• due to NAN/0 bug from unprotected denominator being 0, setting mean and sigma to NAN (then all inputs used...)

Excess hot pixels/CR-like (s065i [4099,4651],[4752,4853]) -- bug

img

• due to bug shown after NAN/0 bug fixed, sigma sometimes being set to a negative value resulting in final mean and sigma going NAN (then all inputs used...)

Large region of low level, bad pixels (s013g []) -- bug? much can be reduced by config mods

img

mk

num

wt

• greatly reduced by lowering THRESHOLD.MASK 0.5->0.01 (may need further modifications)

Edge pomegranate-like webbing (s013g [], md09s012g extensive) -- bug? much can be reduced by config mods

img

mk

num

wt

• greatly reduced by lowering THRESHOLD.MASK 0.5->0.3 (may need further modifications, to 0.01 even better)

• md09s12g example -- similar or different case?

Misc edge junk -- add 3PI or impose a Nin miminum per pixel?

Specially skipped edge junk -- strip along skycell with fair amount of junk ignored, 2*kernel size..

• 2*kernel=20 pixels, MD.V3 skycell overlap is 320 pixels -- this should just be entirely MASKED

---

Test Samples

All stacks should be available via the Postage Stamp Server (PSS), if large or full sets are desired then they can be found as usual on the datastore under ps1-deeptest with the data_group name the same as the label. Warp examples should be obtainable via the PSS but will not be pushed to the distribution server.

MD04 PV2 warps -- the full test runs below used all available PV2 through the end of the survey (MD.PV2)

• warp label: MD04.pv2.20130903 -- all filters
• not all examples below use the full set of warps, many use only the same warps that went into MD04.cut2009deeptest.20131103 in order to provide a similar stack for comparison.

3PI exposures overlapping with MD04 -- span Apr. 2009 to Feb. 17 2014, however any exposures before June 2010 will be excluded just like the MD04 exposures (i.e., cut2009 condition described at MD04.pv2_stacktest.20130903)

• warp label: MD04.test3pistack.20140217 -- g-band only
• (need to add total and used numbers)

MD04+3PI overlap test with just bug fixes to compare the edge quality -- will include additional PV2 warps through end of survey for MD04 and similarly 3PI though Feb. 2014

• label: MD04_3PI.cut2009teststack.20140404 -- g-band only

MD04+3PI overlap test with bug fixes and config modifications to compare the edge quality -- will include same available warps as prior test

• label: MD04_3PI.cut2009teststack.20140411 -- again g-band only
• THRESHOLD.MASK 0.01
• COMBINE.REJ 2.5
• will take work to evaluate impact of config modifications between the 3PI and MD here.. -- likely will have to consider this option after much more testing

MD04 with bug fixes, config modifications, N>20 inputs per pixel to clean up the field edge -- can compare to MD04.cut2009deeptest.20131103 since PV2 inputs limited by date here through end of Sept. 2013

• label: MD04.cut2009n20deeptest.20140413 -- i-band only, same inputs as MD04.cut2009deeptest.20131103
• THRESHOLD.MASK 0.01
• COMBINE.REJ 2.5

MD04 with bug fixes, smaller config modifications, hard coded N>30 inputs per pixel -- but other configs not changed as much from MD04.cut2009deeptest.20131103

• label: MD04.cut2009n30deeptest.20140416 -- i-band only, same inputs as MD04.cut2009deeptest.20131103
• THRESHOLD.MASK 0.3
• COMBINE.REJ 3.5

MD09 example as a shifted center field case -- pre-test showed a large amount of junk, so also set SUSPECT input pixels as BAD pixels, edge properties unknown so keep pixels N>10 inputs (NMINPIX)

• label: MD09.cut2009n10suspectbaddeeptest.20140427 -- g-band only, full PV2 set after 6/2010 like cut2009
• THRESHOLD.MASK 0.3
• COMBINE.REJ 3.5
• NMINPIX 10
• MASK.VAL MASK.VALUE,CONV.BAD,GHOST,SUSPECT

MD09 example to try exclude extra junk -- NMINPIX now 30, adding a MASK.BLANKBORDER to mask the edge of the skycell since it goes through essentially no rejection process and often a pile of junk. Is set 20 pixels since the border is set in code to be KERNEL.SIZE in the convolution and overlap setup. No pixels are lost since the MD skycells have an overlap of 320 pixels, now they will overlap by only a mere 300 pixels.

• label: MD09.cut2009n30th01suspectbadborder.deeptest.20140502 -- g-band only, full PV2 set after 6/2010 like cut2009
• THRESHOLD.MASK 0.1
• COMBINE.REJ 3.0
• NMINPIX 30
• MASK.VAL MASK.VALUE,CONV.BAD,GHOST,SUSPECT
• MASK.BLANKBORDER 20

MD04 example with changes made in the MD09 above and with imposed smaller target PSF -- limit the target PSF for convolution to check if smaller THRESHOLD.MASK has any problems

• label: MD04.cut2009n30th01suspectbad7pix.deeptest.20140504 -- g-band only, same inputs as MD04.cut2009deeptest.20131103
• THRESHOLD.MASK 0.1
• COMBINE.REJ 3.0
• NMINPIX 10
• MASK.VAL MASK.VALUE,CONV.BAD,GHOST,SUSPECT
• MASK.BLANKBORDER 20
• PSF.INPUT.MAX 7.0

---

Configuration Mods

This should be done properly before the final version, but will take some time to do. Config params that impact the pixel rejection

COMBINE.ITER    F32     0.5             # Number of rejection iterations per input
COMBINE.REJ     F32     3.5             # Rejection threshold in combination (sigma)
COMBINE.SYS     F32     0.1             # Relative systematic error in combination
COMBINE.DISCARD F32     0.2             # Discard fraction for Olympic weighted mean

MASK.VAL        STR     MASK.VALUE,CONV.BAD,GHOST # Mask value of input bad pixels
MASK.SUSPECT    STR     SUSPECT,BURNTOOL,SPIKE,STREAK,STARCORE,CONV.POOR # Mask value of suspect pixels
MASK.BAD        STR     BLANK           # Mask value to give bad pixels
MASK.POOR       STR     CONV.POOR       # Mask value to give poor pixels

POOR.FRACTION   F32     0.25            # Maximum fraction of bad variance for poor pixels
THRESHOLD.MASK  F32     0.5             # Threshold for mask deconvolution (0..1)
IMAGE.REJ       F32     0.1             # Rejected pixel fraction threshold for rejecting entire image
VARIANCE        BOOL    TRUE            # Use variance in rejection?
SAFE            BOOL    TRUE            # Play safe when combining small number of values?

New options added:

NMINPIX         S32     30              # Minimum input per pixel 
MASK.BLANKBORDER S32    20              # Mask blank border in final stack output

Unclear how a mix of MD+3PI will be impacted by config mods... a strong reason to not make MD+3PI stacks yet.