= Summary of MDdeeptest_pixrej_20140403 =
[[PageOutline]]

(back to [wiki:MediumDeepFields] summary)

For a while it has been bothersome to find junk in the MD stacks through just simple visual inspection as would not be expected in N=60 (refstack) and definitely not for N>>60 (deepstack), and they can also come and go depending on the inputs used. This seems to be at least configs needing tuning, at worst code bugs.


A related issue also to be addressed is the ratty edge of the MD stacks due to poorer edge chips and smaller number of input pixels. Possible alternatives for a solution are
 * hard cut by radius field by field and filter --
 * combine MD+3PI to help improve by including better 3PI pixels -- 
 * only include pixel if N input > some minimum, similar to the SAFE (N=2) option but with a larger N minimum -- 
 * any others options?


----

= Example Junk and Bugs = 

Large regions of high level, bad pixels (s065i -- diffract spike chunks [1129,4256], the moth []) -- bug

|| img || mk || wt || num || exp || expwt || 
[[Image(md04s065i_stackjunk_1_cut2009_ds9.jpg,600px)]] 

[[Image(md04s065i_stackjunk_2_cut2009_ds9.jpg,600px)]]

 * due to NAN/0 bug from unprotected denominator being 0, setting mean and sigma to NAN (then all inputs used...)
 

Excess hot pixels/CR-like (s065i [4099,4651],[4752,4853]) -- bug 

|| img || 

[[Image(md04s065i_stackjunk_crlike_1_ds9.jpg,300px)]]

[[Image(md04s065i_stackjunk_crlike_2_ds9.jpg,300px)]]

 * due to bug shown after NAN/0 bug fixed, sigma sometimes being set to a negative value resulting in final mean and sigma going NAN (then all inputs used...)


Large region of low level, bad pixels (s013g []) -- bug? much can be reduced by config mods

|| img || mk || num || wt || 

[[Image(md04s013g_stackjunk_1_ds9.jpg,600px)]]

 * greatly reduced by lowering THRESHOLD.MASK 0.5->0.01 (may need further modifications)


Edge ''pomegranate-like webbing'' (s013g []) -- bug? much can be reduced by config mods

|| img || mk || num || wt || 

[[Image(md04s013g_stackjunk_edgepom_2_ds9.jpg,600px)]]

 * greatly reduced by lowering THRESHOLD.MASK 0.5->0.3 (may need further modifications, to 0.01 even better) 

[[Image(md04s013g_stackjunk_edgepom_2_n20thr03_ds9.jpg,600px)]]



Misc edge junk -- add 3PI or impose a Nin miminum per pixel?


----


= Test Samples = 

All should be available via the PSS, if large or full sets are desired then can be push to the datastore (let us know)

MD04 PV2 warps -- MD04.cut2009deeptest.20131103 for manual testing, the full test runs below used all available PV2 through the end of the survey ([wiki:MD.PV2])
 * label: MD04.pv2_stacktest.20130903


3PI exposures overlapping with MD04 -- span Apr. 2009 to Feb. 17 2014, however any exposures before June 2010 will be excluded just like the MD04 exposures (i.e., cut2009 condition described at [wiki:MD04.pv2_stacktest.20130903#DeepStacks])
 * label: MD04.test3pistack.20140217 -- 
 * (need to add total and used numbers)


MD04+3PI overlap test with just the bug fixes to compare the edge quality -- will include additional PV2 warps through end of survey for MD04, but just 3PI though Feb. 2014
 * label: MD04_3PI.cut2009teststack.20140404 -- g-band only


MD04+3PI overlap test with bug fixes and config mods to compare the edge quality -- will include same available warps 
 * label: MD04_3PI.cut2009teststack.20140411 -- again g-band only
 * THRESHOLD.MASK  0.01
 * COMBINE.REJ     2.5

 
MD04 with bug fixes, config mods, N>20 inputs per pixel to clean up the field edge -- can compare to MD04.cut2009deeptest.20131103 since inputs limited by date here
 * label: MD04.cut2009n20deeptest.20140413 -- i-band only
 * THRESHOLD.MASK  0.01
 * COMBINE.REJ     2.5



----

= Configuration Mods = 

This should be done properly before the final version, but will take some time to do. Config params that impact the pixel rejection 
{{{

COMBINE.ITER    F32     0.5             # Number of rejection iterations per input
COMBINE.REJ     F32     3.5             # Rejection threshold in combination (sigma)
COMBINE.SYS     F32     0.1             # Relative systematic error in combination
COMBINE.DISCARD F32     0.2             # Discard fraction for Olympic weighted mean

MASK.VAL        STR     MASK.VALUE,CONV.BAD,GHOST # Mask value of input bad pixels
MASK.SUSPECT    STR     SUSPECT,BURNTOOL,SPIKE,STREAK,STARCORE,CONV.POOR # Mask value of suspect pixels
MASK.BAD        STR     BLANK           # Mask value to give bad pixels
MASK.POOR       STR     CONV.POOR       # Mask value to give poor pixels

POOR.FRACTION   F32     0.25            # Maximum fraction of bad variance for poor pixels
THRESHOLD.MASK  F32     0.5             # Threshold for mask deconvolution (0..1)
IMAGE.REJ       F32     0.1             # Rejected pixel fraction threshold for rejecting entire image
VARIANCE        BOOL    TRUE            # Use variance in rejection?
SAFE            BOOL    TRUE            # Play safe when combining small number of values?


-- NINMIN -- some Nin minimum pixel threshold

}}}