Summary of MD10.refstack.20130715

(back to MediumDeepFields summary)

Similar to the revised MD10.refstack.20120804 (MD10.refstack.20120705) refstacks for the 2012 season, these include recent improvements for the PV2. Overall fewer inputs to the refstack were done this round and may not be as deep as last season.

With the concurrent, new ipp-20130712 tag including PV2 modifications, MD10 was the guinea pig and includes additional testing and config modifications

  • PV2 1-D convolution initially failed for MD, the original ISIS method was used for the active refstack, MD10.refstack.20130715_1dg_v3 is working version of 1D for MD10
  • PV2 dynamic masks working, however many ghosts are not masked due to their centers being in gaps and will need to be addressed in PV3
  • stack photometry appears to have odd/poor solutions for normal PSF photometry on some skycells (and seeing more oddities showing up in normal nightly processing..)
  • long outstanding change to the nightly stack definition -- in progress
  • combining MD with overlapping 3PI to improve the edge mess -- in progress

MD10 observations:

  • starting late June 2013 with pre-observations, regular observations mid-July
  • end July 2012 through start Jan. 2013
  • mid-July 2011 through early Dec. 2011
  • late June 2010 through early Dec. 2010
  • end Aug. 2009 through start Jan. 2010
  • (some 2008 but pre-survey)
Filter N total N FWHM<8 pixels N (FWHM<8 pixels for 2012 N FWHM<6 pixels N (FWHM<6 pixels for 2012
g 627 591 120 478 72
r 613 584 111 515 83
i 608 565 145 483 117
z 660 610 180 559 150
y 557 504 217 396 150
  • 2012 wasn't a great season for MD10, but better than MD09

Status:

  • 07/15/13 selected exposure reprocessing chip->warp for refstack started
  • 07/18 reftests finished. detrend issues OTA47, redoing subset of exposures for most of 2010 to use revised video dark.. full refstack started
  • 07/19 refstack finished, checked and problem skycells found/fixed if possible. staticsky single filter only finished and mappings started -- seeing psphot issue reported in DRAVG still..
  • (07/20) mappings finished (some oddities showing up), 1-d conv PV2 rejecting more than expected, diffim testing, non-1d conv refstack setup , recently missed exposures setup for diffims and finished
  • (07/21) reftest+refstack to distribution
  • 07/25 finished looking into oddities for now, updating wiki
  • 8/17 improved 1D convolution stacks made (MD10.refstack.20130715_1dg_v3)

MD10.V3 Processing (chip->warp + refstack)

Exposure reprocessing was done for only a limited number (~150) of the best images from 2010-2012 to provide a balance of N input warps versus best seeing, excluding exposures with large background, low zeropoints and large elongation.

Processing chip-warp and the refstack itself has the same label, data_group

  • label: MD10.refstack.20130715 --> once set as active refstack for diffims, stack label will change to MD10.refstack, chip/cam/warp label will remain the same
  • data_group: MD10.refstack.20130715

The ipp-20130712 tag used

  • GPC1_Detrend_Documentation#Currentdetrends -- new video darks
  • dynamic masks now propagating through to warps -- ghost mask is slightly offset or missing at times
  • 1-d convolution option for PV2 -- more rejection than expected
  • changes to psphot, PSF_MAJOR,MINOR root(2) different but appropriate now and 2*root(2) for FWHM; may be stability issue in PSF fitting
  • SCL_ keyword keeps scaling value and not reassigned to 1. when stack rejected now
  • new reduction class with clear name MD_REF_STACK
  • SAFE config now set to false

Histograms of the FWHM major and minor and elongation from camera stage value (full exposure) for the sample of reprocessed exposures (red) and used for refstack.20130715 (top blue) and used for refstack.20120804 (bottom blue) in grizy order

Test stack (reftest) sample also made again for skycell.055 with different number of input warps.

  • label: MD10.reftest.20130715
  • data_group:
    • MD10.reftest.20130715.c#f<FWHM cut> where #=comparison run, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
    • MD10.reftest.20130715.all.<FWHM cut> where all means 05/2010+2011+2012 data possible, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
    • MD10.reftest.20130715.2012.<FWHM cut> where 2012 means 2012 data only for season stack example case, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
  • the main field center shift was through ~Dec 2009, but is a very small shift. Excluding 2009 since usually not very good, or uncharacteristically too good (avoiding problems basically)
  • c1 cuts
    filter: (FWHM set), elongation, zeropoint, background limits
    y: (3.5, 3.8, 4.1, 4.6), 1.3, 22.8, 5000 -- bkg bit higher
    z: (3.6, 3.7, 3.8, 4.0), 1.3, 24.0, 4000 -- bkg bit higher 
    i: (3.7, 3.9, 4.1, 4.3), 1.25, 24.3, 2000 --
    r: (4.2, 4.4, 4.5, 4.7), 1.25, 24.3, 600 -- 
    g: (4.6, 4.8, 5.0, 5.2), 1.25, 24.3, 400 -- 
    --> refstack starts with 5.0, 4.5, 4.1, 3.8, 4.1 pixels for upper cut in grizy (may go smaller based on MEAN+1.0*STDDEV)
    

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055) after running psphot on the stacks.

| data_group                        | filter  | stack_id | warps_in | warps_rej |  FWHM_MAJ   IQ_FW1 (pixels)
+-----------------------------------+---------+----------+----------+-----------+
--> 2013 refstack
| MD10.refstack.20130715            | g.00000 |  2294005 |       94 |        16 | 	 5.01  4.03 
| MD10.refstack.20130715            | r.00000 |  2293945 |       84 |        15 | 	 4.52  3.74 
| MD10.refstack.20130715            | i.00000 |  2293885 |       86 |        11 | 	 3.78  3.10 
| MD10.refstack.20130715            | z.00000 |  2293825 |       85 |        16 | 	 3.16  2.99 
| MD10.refstack.20130715            | y.00000 |  2293765 |       87 |        15 | 	 3.42  3.14 

--> reftest comparisons						
| MD10.reftest.20130715.all.49      | g.00000 |  2291338 |       95 |	     20 | 	 4.74  3.88 
| MD10.reftest.20130715.2012.49     | g.00000 |  2291339 |       13 |	      4 | 	 4.77  3.56 
| MD10.reftest.20130715.c1f46       | g.00000 |  2291365 |       25 |	      6 |	 4.82  3.59 
| MD10.reftest.20130715.c1f48       | g.00000 |  2291366 |       56 |	      9 | 	 4.71  3.84 
| MD10.reftest.20130715.c1f50       | g.00000 |  2291367 |       95 |	     17 | 	 4.48  4.04 
| MD10.reftest.20130715.c1f52       | g.00000 |  2291368 |      124 |	     21 | 	 4.95  4.15 
| MD10.reftest.20130715.all.44      | r.00000 |  2291336 |       90 |	     17 | 	 4.26  3.56 
| MD10.reftest.20130715.2012.44     | r.00000 |  2291337 |       17 |	      3 | 	 3.88  3.12 
| MD10.reftest.20130715.c1f42       | r.00000 |  2291361 |       31 |	      4 | 	 4.21  3.38 
| MD10.reftest.20130715.c1f44       | r.00000 |  2291362 |       65 |	     11 | 	 4.41  3.66 
| MD10.reftest.20130715.c1f45       | r.00000 |  2291363 |       84 |	     15 | 	 4.38  3.74 
| MD10.reftest.20130715.c1f47       | r.00000 |  2291364 |      121 |	     16 | 	 4.51  3.83 
| MD10.reftest.20130715.all.40      | i.00000 |  2291334 |       90 |	     12 | 	 3.74  3.06 
| MD10.reftest.20130715.2012.40     | i.00000 |  2291335 |       13 |	      2 | 	 3.33  3.20 
| MD10.reftest.20130715.c1f37       | i.00000 |  2291357 |       30 |	      6 | 	 3.48  2.98 
| MD10.reftest.20130715.c1f39       | i.00000 |  2291358 |       54 |	     10 | 	 4.61  3.02 
| MD10.reftest.20130715.c1f41       | i.00000 |  2291359 |       87 |	     10 | 	 3.77  3.10 
| MD10.reftest.20130715.c1f43       | i.00000 |  2291360 |      124 |	     20 | 	 3.86  3.17 
| MD10.reftest.20130715.all.37      | z.00000 |  2291332 |       85 |	     21 | 	 3.39  2.95 
| MD10.reftest.20130715.2012.37     | z.00000 |  2291333 |       18 |	      3 | 	 3.58  3.03 
| MD10.reftest.20130715.c1f36       | z.00000 |  2291353 |       46 |	      8 | 	 3.36  2.93 
| MD10.reftest.20130715.c1f37       | z.00000 |  2291354 |       66 |	     11 | 	 3.43  2.96 
| MD10.reftest.20130715.c1f38       | z.00000 |  2291355 |       85 |	     18 | 	 3.15  2.98 
| MD10.reftest.20130715.c1f40       | z.00000 |  2291356 |      118 |	     23 | 	 3.22  3.04 
| MD10.reftest.20130715.all.38      | y.00000 |  2291329 |       82 |	     20 |	 3.37  3.07 
| MD10.reftest.20130715.2012.38     | y.00000 |  2291331 |       28 |	      4 | 	 3.37  2.89 
| MD10.reftest.20130715.c1f35       | y.00000 |  2291349 |       27 |	      5 | 	 3.29  2.85 
| MD10.reftest.20130715.c1f38       | y.00000 |  2291350 |       59 |	     10 | 	 3.38  3.06 
| MD10.reftest.20130715.c1f41       | y.00000 |  2291351 |       88 |	     17 | 	 3.41  3.15 
| MD10.reftest.20130715.c1f46       | y.00000 |  2291352 |      118 |	     18 | 	 3.50  3.26 
	
--> previous 2012 refstack				
| MD10.refstack.20120804            | g.00000 |  1108704 |      103 |	     36 | 	 5.08  4.10 
| MD10.refstack.20120804            | r.00000 |  1112755 |       98 |	     21 | 	 4.25  3.86 
| MD10.refstack.20120804            | i.00000 |  1112815 |       89 |	     13 | 	 3.50  3.15 
| MD10.refstack.20120804            | z.00000 |  1112695 |      100 |	     19 | 	 3.41  3.12 
| MD10.refstack.20120804            | y.00000 |  1108632 |       71 |	     11 | 	 3.64  3.32 

As before, the FWHM_MAJOR with the central skycell can be variable and more so with largely different input sets. Comparison with the psf_major/minor maps are more useful between refstack versions.

Problems

  • as with all past refstacks, generally edge skycells have issues -- need better containment of those still. (see core coverage below).
  • like others, greater % of inputs rejected (20-30%) so per pixel input is less, but still of order 30 inputs per pixel
  • some convolved stacks have poorer FWHM PSF but not significant enough to be likely improved by reprocessing using DEEP_STACK config as in earlier MD fields
  • skycell notes:
    • y: s025,026,036,037,047,057 bad cell bands still.. s032 odd patch
    • z: s012 quality fault, rerun with same MD_REF_STACK but change elong cut <1.25 from 1.3 to remove odd exposures messing up Target PSF; s046,063,065,066 odd,raised background (bad cell?)
    • i: s065 like some other cells, some odd raised background (bad cell? glint?)
    • r: s014,015,048,064,065,066,084 poor ghost mask/mess
    • g: s014,015,022,023,027,032,037,045,048,058,064,065,067,085 poor ghost mask/mess; s038 edge really ratty; s044,054 poor cell arc (wonder if could/should mask cell as seen in MD10.refstack.20120705)

Ghost mask example -- skycell.045 g-band has a cluster of donuts, sample of example input warps

neb://any/gpc1/MD10.refstack.20130715/MD10.V3/skycell.045/MD10.V3.skycell.045.stk.2293997

--> one is missing 
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5416g0465o.204618/o5416g0465o.204618.wrp.814962.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5776g0435o.372304/o5776g0435o.372304.wrp.815009.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5779g0510o.373485/o5779g0510o.373485.wrp.815020.skycell.045
... likely others

--> slightly missed position, ~40 pixels
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5425g0536o.208404/o5425g0536o.208404.wrp.814971.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5443g0459o.219428/o5443g0459o.219428.wrp.814976.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5443g0461o.219430/o5443g0461o.219430.wrp.814978.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5443g0456o.219426/o5443g0456o.219426.wrp.814979.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5443g0458o.219427/o5443g0458o.219427.wrp.814981.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5767g0498o.368083/o5767g0498o.368083.wrp.815004.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5776g0433o.372302/o5776g0433o.372302.wrp.815008.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5776g0437o.372306/o5776g0437o.372306.wrp.815010.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5776g0438o.372307/o5776g0438o.372307.wrp.815012.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5779g0505o.373480/o5779g0505o.373480.wrp.815013.skycell.045
... and others

Core coverage -- one of the shifted pointings so different from normal distribution of skycells (also see detection efficiency maps below)

082-087
071-078
061-068
051-058
041-048
031-038
021-028
012-017
  • skycells with fewer inputs (typical for edges): 012,017,021,028,071,078,082,087
  • skycells with <50% or more of typical input number is filter dependent - usually s017,028,078,087; sometimes none except s078

Stackphot (staticsky) oddity/problem -- skycell.024 z-band main example (other hot region/skycells in plots below) in the single filter photometry cases (1x). The 5 filter (Nx) ones are a little better and without an odd deteff in z-band, but still show hot spots more than previous tag.

-- log snippet from ipp-20130715 -- neb://any/gpc1/MD10.refstack.20130715.1x20130719/MD10.V3/skycell.024/MD10.V3.skycell.024.sky.446833.log
        fwhm (psf): 6.367312,2.439518 (moments): 3.011427,3.011427
        PSF extra parameter 1: -0.232193 +/- 0.461459
    select psf model: 1.384724 sec
      psf model PS_MODEL_PS1_V1, ApResid: 0.141451 +/- 0.008141
    built models for 4767 objects: 0.210036 sec (12358026 pixels)


-- log snippet from previous ipp-20130307 -- neb://any/gpc1/MD10.refstack.20130715.1x20130724.redoipp130307/MD10.V3/skycell.024/MD10.V3.skycell.024.sky.446953.log
        fwhm (psf): 3.516740,3.209410 (moments): 3.032322,3.032322
        PSF extra parameter 1: -0.052530 +/- 0.030366
    select psf model: 4.576834 sec
      psf model PS_MODEL_PS1_V1, ApResid: 0.121756 +/- 0.009789
    built models for 4905 objects: 0.252904 sec

Refstack Summary

Stack phomtometry on the refstacks done through the staticsky and skycal stages using the unconvolved images

  • single filter mode so basic psphot with deteff measurements that has the zero-point recalculated for the test plots below
    • label: MD10.refstack.20130715, MD10.refstack.20130715
    • data_group (single filter): MD10.refstack.20130715.1x20130719, MD10.refstack.20130715.1x20130719.cal20130719
  • multi-filter (Nx) also run
    • label: MD10.refstack.20130715.5x20130725, MD10.refstack.20130715.5x20130725.cal20130725
    • data_group: MD10.refstack.20130715.5x20130725, MD10.refstack.20130715.5x20130725.cal20130725 (all skycells have 5 filters matched)
    • on datastore under ps1-md-staticsky, but only the staticsky version, the skycal product still has no method for distribution

Source photometry psf_major,_minor maps (grizy): all new plots here scaled by root(2) to match past scaling (but not on the plot label yet)

Magnitude of sources@5-sigma maps (grizy):

Detection efficiency magnitude @50% recovery (500 fakes, grizy): 0 rej for all now that the SCL_ keyword has its true value other than 1. when rejected in stacking


1D convolution mode PV2

Same naming format as generally above, adding _1dg to the data_group (as will be seen on the datastore)

  • label: MD10.reftest.20130715
  • data_group:
    • MD10.reftest.20130715.c#f<FWHM cut>_1dg where #=comparison run, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
    • MD10.reftest.20130715.all.<FWHM cut>_1dg where all means 05/2010+2011+2012 data possible, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
    • MD10.reftest.20130715.2012.<FWHM cut>_1dg where 2012 means 2012 data only for season stack example case, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
  • c1 cuts (same as above)

Full refstack:

  • label: MD10.refstack.20130715_1dg
  • data_group: MD10.refstack.20130715_1dg

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055) after running psphot on the stacks.

| data_group                        | filter  | stack_id | warps_in | warps_rej |  FWHM_MAJ   IQ_FW1 (pixels)
+-----------------------------------+---------+----------+----------+-----------+
| MD10.refstack.20130715_1dg        | y.00000 |  2294065 |       87 |        15 | 
| MD10.refstack.20130715_1dg        | z.00000 |  2294125 |       85 |        32 | 
| MD10.refstack.20130715_1dg        | i.00000 |  2294185 |       86 |        14 | 
| MD10.refstack.20130715_1dg        | r.00000 |  2294245 |       84 |        24 | 
| MD10.refstack.20130715_1dg        | g.00000 |  2294305 |       94 |        27 | 

| MD10.reftest.20130715.all_1dg.38  | y.00000 |  2291330 |       82 |        14 | 
| MD10.reftest.20130715.2012_1dg.38 | y.00000 |  2291340 |       28 |         3 | 
| MD10.reftest.20130715.all_1dg.37  | z.00000 |  2291341 |       85 |        30 | 
| MD10.reftest.20130715.2012_1dg.37 | z.00000 |  2291342 |       18 |         2 | 
| MD10.reftest.20130715.all_1dg.40  | i.00000 |  2291343 |       90 |        17 | 
| MD10.reftest.20130715.2012_1dg.40 | i.00000 |  2291344 |       13 |         1 | 
| MD10.reftest.20130715.all_1dg.44  | r.00000 |  2291345 |       90 |        15 | 
| MD10.reftest.20130715.2012_1dg.44 | r.00000 |  2291346 |       17 |         3 | 
| MD10.reftest.20130715.all_1dg.49  | g.00000 |  2291347 |       95 |        23 | 
| MD10.reftest.20130715.2012_1dg.49 | g.00000 |  2291348 |       13 |         5 | 
| MD10.reftest.20130715.c1f35_1dg   | y.00000 |  2291369 |       27 |         5 | 
| MD10.reftest.20130715.c1f38_1dg   | y.00000 |  2291370 |       59 |         9 | 
| MD10.reftest.20130715.c1f41_1dg   | y.00000 |  2291371 |       88 |        17 | 
| MD10.reftest.20130715.c1f46_1dg   | y.00000 |  2291372 |      118 |        34 | 
| MD10.reftest.20130715.c1f36_1dg   | z.00000 |  2291373 |       46 |        17 | 
| MD10.reftest.20130715.c1f37_1dg   | z.00000 |  2291374 |       66 |        23 | 
| MD10.reftest.20130715.c1f38_1dg   | z.00000 |  2291375 |       85 |        17 | 
| MD10.reftest.20130715.c1f40_1dg   | z.00000 |  2291376 |      118 |        25 | 
| MD10.reftest.20130715.c1f37_1dg   | i.00000 |  2291377 |       30 |         7 | 
| MD10.reftest.20130715.c1f39_1dg   | i.00000 |  2291378 |       54 |        18 | 
| MD10.reftest.20130715.c1f41_1dg   | i.00000 |  2291379 |       87 |        17 | 
| MD10.reftest.20130715.c1f43_1dg   | i.00000 |  2291380 |      124 |        31 | 
| MD10.reftest.20130715.c1f42_1dg   | r.00000 |  2291381 |       31 |         8 | 
| MD10.reftest.20130715.c1f44_1dg   | r.00000 |  2291382 |       65 |        17 | 
| MD10.reftest.20130715.c1f45_1dg   | r.00000 |  2291383 |       84 |        26 | 
| MD10.reftest.20130715.c1f47_1dg   | r.00000 |  2291384 |      121 |        31 | 
| MD10.reftest.20130715.c1f46_1dg   | g.00000 |  2291385 |       25 |         9 | 
| MD10.reftest.20130715.c1f48_1dg   | g.00000 |  2291386 |       56 |         9 | 
| MD10.reftest.20130715.c1f50_1dg   | g.00000 |  2291387 |       95 |        26 | 
| MD10.reftest.20130715.c1f52_1dg   | g.00000 |  2291388 |      124 |        35 | 

PROBLEM -- new 1D convolution code produced unexpected excess rejection and cannot be used for the refstack -- investigate further below

1D Excess Rejection

data_group/log:

MD10.reftest.20130715.c1f46 -- 18 rej, all cut at start
neb://any/gpc1/MD10.reftest.20130715/MD10.V3/skycell.055/MD10.V3.skycell.055.stk.2291352.log

MD10.reftest.20130715.c1f46_1dg -- 34 rej, so 14 additionally rej by chi2 
neb://any/gpc1/MD10.reftest.20130715/MD10.V3/skycell.055/MD10.V3.skycell.055.stk.2291372.log

Target PSF ~4.67 pixels

input index 16,37,38,39,45,46,49,50,62,65,70,71,74,93,96,111 -- many close to the 4.67 target (or even over at 4.7 for 50)

range 4.5-4.7 pixels

possible tests:

  • manually run with manually set target psf 4.8 to see if all pass
  • increase MATCH.REJ from 4 to >10 -- would be necessary to include many rej other than #50

Stack Photometry 1D version

Like the default refstack, also run for the 1D convolution version and the multi-filter version is on the datastore (only staticsky product as the skycal has no method for distribution)

  • label (staticsky, skycal): MD10.refstack.20130715_1dg.5x20130726, MD10.refstack.20130715_1dg.5x20130726.cal20130726
  • data_group: MD10.refstack.20130715_1dg.5x20130726, MD10.refstack.20130715_1dg.5x20130726.cal20130726

Odd Features in 1D version

Odd streaks seem more common in 1D convolved stacking when should be plenty of inputs to reject pixels.

unconv stack 1D conv stack 1D conv mask unconv .num. 1D .num. unconv .exp. 1D .exp

Appears that the .exp. files for the >30 input stacks are mostly NAN. Suspect the compression had a problem. Has been happening for over a year. Added a .EXP.NOCOMP option (one already existed for EXPWT) and if a stack is made uncompressed (as is with deep stacks) then this is used.

  • filesize change 933 KB -->163 MB, a large change without compression, but only for deep stacks (MD ref and deep types only) --

1D convolution v3 Rejection OK

r35925 appears to produce similar rejection to the original ISIS method, and includes a fix the the other rejection issue found in the night stack sample below. Target PSF for convolved stack is now set to the max FWHM of the input warps that pass all input cuts with an additional epsilon of 0.1 pixels.

    PSF.TARGET.AS.MAX  BOOL  TRUE
    PSF.TARGET.AS.MAX.EPSILON F32 0.1
  • label and data_group -- MD10.refstack.20130715_1dg_v3
| data_group                    | filter  | stack_id | skycell_id  | warps_in | warps_rej |
+-------------------------------+---------+----------+-------------+----------+-----------+

| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455233 | skycell.051 |       87 |        16 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455234 | skycell.052 |       87 |        13 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455235 | skycell.053 |       87 |        14 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455236 | skycell.054 |       87 |        11 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455237 | skycell.055 |       87 |        15 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455238 | skycell.056 |       87 |        16 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455239 | skycell.057 |       87 |        14 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455240 | skycell.058 |       87 |        10 | 

| MD10.refstack.20130715 | y.00000 |  2293761 | skycell.051 |       87 |        19 | 
| MD10.refstack.20130715 | y.00000 |  2293762 | skycell.052 |       87 |        13 | 
| MD10.refstack.20130715 | y.00000 |  2293763 | skycell.053 |       87 |        15 | 
| MD10.refstack.20130715 | y.00000 |  2293764 | skycell.054 |       87 |        11 | 
| MD10.refstack.20130715 | y.00000 |  2293765 | skycell.055 |       87 |        15 | 
| MD10.refstack.20130715 | y.00000 |  2293766 | skycell.056 |       87 |        16 | 
| MD10.refstack.20130715 | y.00000 |  2293767 | skycell.057 |       87 |        15 | 
| MD10.refstack.20130715 | y.00000 |  2293768 | skycell.058 |       87 |        12 |

| MD10.refstack.20130715 | y.00000 |  2293763 | skycell.053 |       87 |        15 | 
| MD10.refstack.20130715_1dg | y.00000 |  2294063 | skycell.053 |       87 |        29 | 
| MD10.refstack.20130715_1dg_v2 | y.00000 |  2450747 | skycell.053 |       87 |        29 | 

1D convolution full stack photometry

Will use MD09 1D convolved stacks (MD09.refstack.20130613#a1Dconvolutionmode) as test first since it overlaps with SAS.


Night Stacks and PV2

Night stack configuration has been unchanged since magic requirements (i.e. >=4 inputs) and a number of the marginal or problem nights are being "missed" when a stack of 2-3 could suffice.

Sugggested configuration modifications to include PV2 and 1D convolution chages --

  • SAFE false -- should really only impact the edges in normal night stacks, but in N~2-3 could help fill regions across the FPA
  • 1D convolution --
  • MATCH.REJ 4.0 --
  • PSF.INPUT.MAX 12.0 -- normally limited by camera stage of only processing exposures with FWHM <12 pixels, but there can be cases of edge skycells being just poor or mis-measured and probably do not want to include those
  • N input warp minimun 2 -- was originally set to 4 for magic restrictions, then on partial field nights will end up missing stacks and a stack of two warps is better than none. For nights of 1 warp, there are WS diffims. All reprocessed night stacks use 2 as the minimum.

How many nights have been missed due to <4 exposures?

| dateobs             | comment                      | count(exp_name) |
+---------------------+------------------------------+-----------------+
| 2008-09-18 11:47:48 | md10 test exposure           |               1 | 
| 2008-10-09 10:14:56 | md10_2008-10-08 MD10 center  |               2 | 
| 2009-12-27 05:35:05 | MD10 i MJD55082 MD10 center  |               1 | 
| 2010-09-19 08:10:45 | MD10 g,r N5458 r MD10 center |               2 | 
| 2011-08-28 14:12:18 | MD10 y N5800 MD10 center     |               3 | 
| 2011-10-28 06:11:09 | MD10 y N5862 MD10 center     |               2 | 
| 2011-11-20 06:17:02 | MD10 i N5885 MD10 center     |               1 | 
| 2012-08-17 13:07:29 | MD10 y N6156 MD10 center     |               2 | 
| 2012-11-09 05:13:26 | MD10 y N6240 MD10 center     |               3 | 
| 2012-11-30 04:33:23 | MD10 y N6261 MD10 center     |               3 | 
| 2012-12-11 05:48:44 | MD10 r,i N6272 i MD10 center |               2 | 
--> so 11/444 nights, ~2.4% -- other fields similar ~1.4--3.6%

Test 1D Night Stacks

Night 20130707 g and z -- using ipp-trunk build r35868 with OPTIMUM.TOL setup to relax rejection for FWHM near target PSF and config modifications above

  • label: meh_md10nstest20130707, meh_md10nstest20130707_1dg
  • data_group: meh_md10nstest20130707<f>, meh_md10nstest20130707<f>_1dg -- <f> is g or z filter

Another 1D Rejection Problem

Another rejection problem -- excess rejection even with trunk modification to use OPTIMUM.TOL, entire images (not convolved because > target PSF) are being fully rejected in pixel rejection stage. Also from example log notice Target seeing FWHM appears not well matched and probably the actual trigger for this (manually setting larger and is fine). Wonder if is related to the diffim PSF problems with recent psphot PSF fixes.

| data_group                  | filter  | stack_id | warps_in | warps_rej |
+-----------------------------+---------+----------+----------+-----------+
| meh_md10nstest20130707g     | g.00000 |  2355589 |        8 |         0 | 
| meh_md10nstest20130707z     | z.00000 |  2355649 |        8 |         0 | 
| meh_md10nstest20130707g_1dg | g.00000 |  2355709 |        8 |         0 | 
| meh_md10nstest20130707z_1dg | z.00000 |  2355769 |        8 |         5 | 

-- not just central poor PSF skycells either, see at various % over FPA in both filters

| meh_md10nstest20130707g_1dg | g.00000 |  2355692 | skycell.034 |        8 |         4 | 

neb://any/meh/meh_md10nstest20130707z_1dg/MD10.V3/skycell.055/MD10.V3.skycell.055.stk.2355769.log

      Input seeing FWHMs:
      Input 0: 5.394163
      Input 1: 5.390012
      Input 2: 5.393055
      Input 3: 5.100585
      Input 4: 5.308995
      Input 5: 4.463570
      Input 6: 4.224805
      Input 7: 3.980880
      Input FWHMs : 4.907008 +/- 0.588663
      PSF FWHM distribution: Used simple clip method.  Limit set to 12.000000.

      Target seeing FWHM: 4.742117

      Stage 2: Generate Convolutions and Save: 161.477153 sec
      Summary of images for initial combination:
      Image 0:  Good.
      Image 1:  Good.
      Image 2:  Good.
      Image 3:  Good.
      Image 4:  Good.
      Image 5:  Good.
      Image 6:  Good.
      Image 7:  Good.
      Stage 3: Make Initial Stack: 290.986779 sec
    Rejection on image 0 didn't work --- reject entire image.
      Time to perform rejection on image 0: 0.304300 sec
    Rejection on image 1 didn't work --- reject entire image.
      Time to perform rejection on image 1: 0.272519 sec
    Rejection on image 2 didn't work --- reject entire image.
      Time to perform rejection on image 2: 0.274936 sec
    Rejection on image 3 didn't work --- reject entire image.
      Time to perform rejection on image 3: 0.274820 sec
    Rejection on image 4 didn't work --- reject entire image.
      Time to perform rejection on image 4: 0.275285 sec
      Convolve image: 5.637316 sec
      150010 pixels rejected from image 5 (0.4%)
      Time to perform rejection on image 5: 6.448552 sec
      Convolve image: 5.874424 sec
      102450 pixels rejected from image 6 (0.3%)
      Time to perform rejection on image 6: 6.665843 sec
      Convolve image: 5.654609 sec
      93433 pixels rejected from image 7 (0.2%)
      Time to perform rejection on image 7: 6.459099 sec

      Stage 4: Pixel Rejection: 21.048442 sec
      Summary of images for final combination:
      Image 0:  Rejection exceeded threshold.
      Image 1:  Rejection exceeded threshold.
      Image 2:  Rejection exceeded threshold.
      Image 3:  Rejection exceeded threshold.
      Image 4:  Rejection exceeded threshold.
      Image 5:  Good.
      Image 6:  Good.
      Image 7:  Good.
      Stage 5: Final Stack: 12.869377 sec

Odd Core NAN Masking Oddity in 1D version

If the target PSF is << input PSF then even though image is triggered to not be convolved it is rejected in the pixel rejection stage. The resulting images have many sources with NAN cores. Example skycell.055, z-filter

  • unknown if this is due to the odd rejection case due to PSF > target PSF or if due to bug when image rejection done in pixel rejection stage
  • also appears to be some print through of burns and the like
1D convolved stack stack mask input warps 1..N input masks 1..N

  • once the modification was made to set the target PSF to the maximum input warp, this problem is not longer happening (r35925).

Masking of poor versus suspect pixels may have issues -- example skycell.055, z-filter near center skycell

  • appears to be many, likely suspect, pixels getting through and consideration of rejecting additional types of masked pixels (i.e. adding to the poor set) may be necessary
1D convolved stack stack mask sample input warp sample input mask

Successful 1D Night Stack

Using r35943 a successful set of night stacks are made with the suggested configuration modifications above --

  • g-filter label, data_group -- meh_md10nstest20130707g_1dg_v2, meh_md10nstest20130707g_1dg_v2
  • z-filter label, data_group -- meh_md10nstest20130707z_1dg_v2, meh_md10nstest20130707z_1dg_v2
| data_group                          | filter  | stack_id | warps_in | warps_rej |
+-------------------------------------+---------+----------+----------+-----------+
| meh_md10nstest20130707g             | g.00000 |  2355589 |        8 |         0 | 
| meh_md10nstest20130707z             | z.00000 |  2355649 |        8 |         0 | 
| meh_md10nstest20130707g_1dg         | g.00000 |  2355709 |        8 |         0 | 
| meh_md10nstest20130707z_1dg         | z.00000 |  2355769 |        8 |         5 | 
| meh_md10nstest20130707g_1dg_v2      | g.00000 |  2447129 |        8 |         0 | 
| meh_md10nstest20130707z_1dg_v2      | z.00000 |  2447289 |        8 |         0 | 

Now can move on to testing the configuration changes and 1D convolution on a sample of normal and marginal type of night stacks..

PV2 Update Sample

To remake the night stacks, cleaned up chip/warps need to be updated. Part of PV2 is the inclusion of the dynamic mask during chip->warp updates by also updating the camera stage SMF and this should be tested for the MD before fully applied to the planned MD04 updates for making the MD04 deep stack (MD04.pv2_deeptest.20130903).

Sample MD10 chip+camera+warp to be updated to PV2 -- when updating PV2, label is changed to reflect this while data_group is left as the original to be able to trace back. As camera stage is not normally cleaned up, the camera label is the key indicator to which processing is PV2.

| count(warp_id) | label             | data_group             | dateobs             | filter  |
+----------------+-------------------+------------------------+---------------------+---------+
|              4 | MD10.pv2.20130828 | MD10.20120825          | 2012-08-25 13:20:29 | y.00000 | 
|             16 | MD10.pv2.20130828 | MD10.20121014          | 2012-10-14 07:14:56 | g.00000 | 
|              2 | MD10.pv2.20130828 | MD10.20121211          | 2012-12-11 05:52:58 | i.00000 | 
|             19 | MD10.pv2.20130828 | MD10.20121215          | 2012-12-15 05:47:47 | y.00000 | 

|              5 | MD10.pv2.20130828 | MD10.pv1.20130324      | 2010-11-03 08:01:45 | g.00000 | 
|              3 | MD10.pv2.20130828 | MD10.refstack.20120705 | 2010-11-03 08:14:32 | g.00000 | 

|             22 | MD10.pv2.20130828 | MD10.20121023          | 2012-10-23 07:07:21 | y.00000 | 
|             18 | MD10.pv2.20130828 | MD10.refstack.20130715 | 2012-10-23 07:28:34 | y.00000 | 

|              8 | MD10.pv2.20130828 | MD10.20130714          | 2013-07-14 13:13:17 | i.00000 | -- already PV2 so a normal update process without updating the camera stage

When doing the update process, have found cases where only a single, valid instance exists for a not insignificant amount of raw data.. this is tracked more in the full MD04 update (MD04.pv2_deeptest.20130903).

PV2 Update Masking Losses in Warp

Odd mask lost in PV2 updates -- seems to be limited to

    PM_SOURCE_MODE_CR_LIMIT
    PM_SOURCE_MODE_BLEND

Was suggested that possibly from photometry not being run on update -- since masks being remade in PV2 updates then lost, while a normal update uses the already defined mask? Since PV2 is mainly for improved inputs to stacks, should be minor and just rejected? (except for low N..)

Example -- one masked CR is lost (full reprocessing retains it while the PV2 update method loses it) and one is gained (as with the full reprocessing and should be)

exposure.exp_id: o5503g0240o.250240
new chip/cam/warp_id = 883673 / 854492 / 835018
old/pv1/up chip/cam/warp_id = 810251 / 786993 / 766825
skycell.045
  • Image panels --
full reprocessing pv1 normal update pv2 s/ camera update cleanup and normal update of pv2

  • Mask panels left to right -- (same sequence)


PV2 Update Night Stacks

Only focusing on the unconvolved stacks as those are only used for the SSdiff, the convolved stacks will be looked at in more detail when stack photometry is evaluated.

Default/current operations config version with PV2 update warps (also using ISIS convolution) --

  • label -- meh_md10ns20130831
  • data_group -- meh_md10.<obsdate>ns20130831 (i.e., meh_md10.20101103ns20130831)

Suggested config option version with PV2 update warps (using the 1D convolution) --

  • label -- meh_md10nstest20130830
  • data_group -- meh_md10.<obsdate>nstest20130830 (i.e. meh_md10.20101103ns20130830)
| count(warp_id) | label             | data_group             | dateobs             | filter  |
+----------------+-------------------+------------------------+---------------------+---------+
|              4 | MD10.pv2.20130828 | MD10.20120825          | 2012-08-25 13:20:29 | y.00000 | 
--> only 4/8 as the other exposures have >12 pixel FWHM and fail the camera stage. Remaining 4 have near limit FWHM
|             16 | MD10.pv2.20130828 | MD10.20121014          | 2012-10-14 07:14:56 | g.00000 | 
--> large number of input warps with large FWHM and large FWHM sigma
|              2 | MD10.pv2.20130828 | MD10.20121211          | 2012-12-11 05:52:58 | i.00000 | 
--> large-ish FWHM, but <4 inputs and would normally be skipped in making a night stack
|             19 | MD10.pv2.20130828 | MD10.20121215          | 2012-12-15 05:47:47 | y.00000 | 
--> large number of input warps with large FWHM and large FWHM sigma
|              8 | MD10.pv2.20130828 | MD10.20130714          | 2013-07-14 13:13:17 | i.00000 | 
--> already PV2 as originally processed with ipp-20130712 tag, basically a normal night stack, example of a typical night stack
|              5 | MD10.pv2.20130828 | MD10.pv1.20130324      | 2010-11-03 08:01:45 | g.00000 | 
|              3 | MD10.pv2.20130828 | MD10.refstack.20120705 | 2010-11-03 08:14:32 | g.00000 | 
--> example of a typical night stack
|             22 | MD10.pv2.20130828 | MD10.20121023          | 2012-10-23 07:07:21 | y.00000 | 
|             18 | MD10.pv2.20130828 | MD10.refstack.20130715 | 2012-10-23 07:28:34 | y.00000 | 
--> large number of input warps with large FWHM and large FWHM sigma, degraded ZPT and high background
| filter  | dateobs             | Nobs | ZPTav  | ZPTsig | FWHMav | FWHMsig | ELONGav | ELONGsig | BGav      | BGSTDav |
+---------+---------------------+------+--------+--------+--------+---------+---------+----------+-----------+---------+
| g.00000 | 2010-11-03 08:06:03 |    8 | 24.481 | 0.005  | 5.607  | 0.156   | 1.127   | 0.016    | 79.086    | 12.578  | 
| y.00000 | 2012-08-25 13:10:16 |    4 | 23.253 | 0.008  | 12.379 | 0.375   | 1.050   | 0.012    | 1,642.069 | 74.031  | 
| g.00000 | 2012-10-14 07:14:56 |   16 | 24.459 | 0.015  | 9.836  | 1.250   | 1.051   | 0.014    | 81.849    | 15.993  | 
| y.00000 | 2012-10-23 07:07:21 |   22 | 21.493 | 0.683  | 5.601  | 1.540   | 1.134   | 0.069    | 5,387.800 | 108.619 | 
| i.00000 | 2012-12-11 05:48:44 |    2 | 21.347 | 0.793  | 7.522  | 1.353   | 1.224   | 0.172    | 1,404.812 | 95.901  | 
| y.00000 | 2012-12-15 05:47:47 |   19 | 23.370 | 0.013  | 9.039  | 2.055   | 1.040   | 0.017    | 1,733.706 | 122.356 | 
| i.00000 | 2013-07-14 13:13:17 |    8 | 24.566 | 0.008  | 4.974  | 0.256   | 1.590   | 0.207    | 641.060   | 68.630  | 

Rejection comparison --

  • original type night stack can have issue with excessive rejection at times from oddly determined target PSF
  • suggested config night stack seems to reject small numbers -- will need to look into source of the rejection --
------------------------------ skycell.055 (center) ------------------------------------
-- original type night stack --
| data_group                  | filter  | stack_id | warps_in | warps_rej | good_frac |
+-----------------------------+---------+----------+----------+-----------+-----------+
| meh_md10.20120825ns20130831 | y.00000 |  2543164 |        4 |         0 |  0.941    | 
| meh_md10.20121014ns20130831 | g.00000 |  2543222 |       16 |         0 |  0.982    | 
| meh_md10.20121023ns20130831 | y.00000 |  2543282 |       32 |        13 |  0.978    | -- 3 chi2, 10 PSF matching failed
| meh_md10.20121211ns20130831 | i.00000 |  2543339 |        2 |         0 |  0.623    | 
| meh_md10.20121215ns20130831 | y.00000 |  2543396 |       19 |         0 |  0.980    | 
| meh_md10.20130714ns20130831 | i.00000 |  2543456 |        8 |         0 |  0.984    | 
| meh_md10.20101103ns20130831 | g.00000 |  2543514 |        8 |         0 |  0.981    | 
| meh_md10.20101123ns20130831 | y.00000 |  2543571 |        8 |         0 |  0.985    | 

-- suggested config night stack --
| data_group                      | filter  | stack_id | warps_in | warps_rej | good_frac |
+---------------------------------+---------+----------+----------+-----------+-----------+
| meh_md10.20120825nstest20130830 | y.00000 |  2533428 |        4 |         3 |  0.799    | -- FWHM 12.6-12.8 >12pix limit
| meh_md10.20121014nstest20130830 | g.00000 |  2533486 |       16 |         1 |  0.999    | -- FWHM 13.2 >12pix limit (others 8-12 so..)
| meh_md10.20121023nstest20130830 | y.00000 |  2533546 |       32 |         1 |  0.999    | -- FWHM 14.3 >12pix limit (others 4-8 so.. good call)
| meh_md10.20121211nstest20130830 | i.00000 |  2533603 |        2 |         0 |  0.948    | 
| meh_md10.20121215nstest20130830 | y.00000 |  2533660 |       19 |         4 |  0.998    | -- FWHM 12-13 >12 pix limit (others 6-12 so..)
| meh_md10.20130714nstest20130830 | i.00000 |  2533720 |        8 |         0 |  0.998    | 
| meh_md10.20101103nstest20130830 | g.00000 |  2533778 |        8 |         0 |  0.998    | 
| meh_md10.20101123nstest20130830 | y.00000 |  2533835 |        8 |         0 |  0.998    | 


------------------------------ skycell.051 (edge) ------------------------------------
-- original type night stack --
| data_group                  | filter  | stack_id | warps_in | warps_rej | good_frac |
+-----------------------------+---------+----------+----------+-----------+-----------+
| meh_md10.20120825ns20130831 | y.00000 |  2543160 |        4 |         0 | 0.736     | 
| meh_md10.20121014ns20130831 | g.00000 |  2543218 |       16 |         0 | 0.894     | 
| meh_md10.20121023ns20130831 | y.00000 |  2543278 |       32 |        15 | 0.896     | -- 4 chi2, 11 PSF matching failed
| meh_md10.20121211ns20130831 | i.00000 |  2543335 |        2 |         0 | 0.333     | 
| meh_md10.20121215ns20130831 | y.00000 |  2543392 |       18 |         2 | 0.892     | -- 1 PSF matching failed, 1 chi2
| meh_md10.20130714ns20130831 | i.00000 |  2543452 |        8 |         0 | 0.891     | 
| meh_md10.20101103ns20130831 | g.00000 |  2543510 |        8 |         0 | 0.855     | 
| meh_md10.20101123ns20130831 | y.00000 |  2543567 |        8 |         0 | 0.582     | 

-- suggested config night stack --
| data_group                      | filter  | stack_id | warps_in | warps_rej | good_frac |
+---------------------------------+---------+----------+----------+-----------+-----------+
| meh_md10.20120825nstest20130830 | y.00000 |  2533424 |        4 |         0 | NULL      | -- all are FWHM 12.3-13.3 >12pix limit and throws error code
| meh_md10.20121014nstest20130830 | g.00000 |  2533482 |       16 |         1 | 0.926     | -- FWHM 13.2 >12pix limit
| meh_md10.20121023nstest20130830 | y.00000 |  2533542 |       32 |         0 | 0.939     | -- are all really ok?
| meh_md10.20121211nstest20130830 | i.00000 |  2533599 |        2 |         0 | 0.742     | 
| meh_md10.20121215nstest20130830 | y.00000 |  2533656 |       18 |         4 | 0.909     | -- all 4 rej as FWHM 12-13 >12pix limit (others 6-12 so..)
| meh_md10.20130714nstest20130830 | i.00000 |  2533716 |        8 |         0 | 0.929     | 
| meh_md10.20101103nstest20130830 | g.00000 |  2533774 |        8 |         0 | 0.911     | 
| meh_md10.20101123nstest20130830 | y.00000 |  2533831 |        8 |         0 | 0.647     | 

Skycell coverage comparison -- edges and central dither

  • original config night stack
    large image order:
    | 20120825 | 20121014 | 20121023 |
    | 20121211 | 20121215 | 20130714 |
    | 20101103 | 20101123 | ... |
    
skycell.051 skycell.055
zoom slice image order:
| 20120825 | 20121014 | 20121023 | 20121211 | 20121215 | 20130714 | 20101103 | 20101123 |
skycell.051 skycell.055
  • suggested config night stack
large image order similar -- except skycell.051 missing 20120825 (first in sequence)
skycell.051 skycell.055
zoom slice image order similar -- except skycell.051 missing 20120825 (first in sequence)
skycell.051 skycell.055

Detection number and Depth comparison -- good example to test any issues with PSF photometry on the stacks themselves as well


PV2 Update Night Stack SSdiff

SSdiff made using the PV2 sample night stacks and the fix MD10.refstack.20130715_1dg_v3 refstack.

Night stacks using the default/current operations config version with PV2 update warps --

  • label -- meh_md10ssdiff20130831
  • data_group -- meh_md10ssdifftest.20130715_1dg_v3.<obsdate>ns20130831 (i.e., meh_md10ssdifftest.20130715_1dg_v3.20101123ns20130831)

Night stacks using the suggested config option version with PV2 update warps --

  • label -- meh_md10ssdifftest20130830
  • data_group -- meh_md10ssdifftest.20130715_1dg_v3.<obsdate>nstest20130830 (i.e. meh_md10ssdifftest.20130715_1dg_v3.20101103nstest20130830)

Skycell coverage/refstack overlap comparison --

------------------------------ skycell.055 (center) ------------------------------------
-- original type night stack --
| data_group                                            | filter  | diff_id | good_frac | sources |
+-------------------------------------------------------+---------+---------+-----------+---------+
| meh_md10ssdifftest.20130715_1dg_v3.20101103ns20130831 | g.00000 |  473881 | 0.969     |     239 | 
| meh_md10ssdifftest.20130715_1dg_v3.20101123ns20130831 | y.00000 |  473882 | 0.970     |     396 | 
| meh_md10ssdifftest.20130715_1dg_v3.20120825ns20130831 | y.00000 |  473883 | 0.925     |     201 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121014ns20130831 | g.00000 |  473884 | 0.970     |     193 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121023ns20130831 | y.00000 |  473885 | 0.966     |     447 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121211ns20130831 | i.00000 |  473886 | 0.600     |     428 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121215ns20130831 | y.00000 |  473889 | 0.968     |      99 | 
| meh_md10ssdifftest.20130715_1dg_v3.20130714ns20130831 | i.00000 |  473890 | 0.972     |    1067 | 

-- suggested config night stack --
| data_group                                                | filter  | diff_id | good_frac | sources |
+-----------------------------------------------------------+---------+---------+-----------+---------+
| meh_md10ssdifftest.20130715_1dg_v3.20101103nstest20130830 | g.00000 |  473564 | 0.986     |     111 | 
| meh_md10ssdifftest.20130715_1dg_v3.20101123nstest20130830 | y.00000 |  473565 | 0.986     |     426 | 
| meh_md10ssdifftest.20130715_1dg_v3.20120825nstest20130830 | y.00000 |  473566 | 0.783     |     266 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121014nstest20130830 | g.00000 |  473567 | 0.986     |      83 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121023nstest20130830 | y.00000 |  473568 | 0.987     |     277 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121211nstest20130830 | i.00000 |  473569 | 0.936     |     491 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121215nstest20130830 | y.00000 |  473570 | 0.986     |      67 | 
| meh_md10ssdifftest.20130715_1dg_v3.20130714nstest20130830 | i.00000 |  473571 | 0.986     |     957 |


------------------------------ skycell.051 (edge) ------------------------------------
-- original type night stack --
| data_group                                            | filter  | diff_id | good_frac | sources |
+-------------------------------------------------------+---------+---------+-----------+---------+
| meh_md10ssdifftest.20130715_1dg_v3.20101103ns20130831 | g.00000 |  473881 | 0.847     |     226 | 
| meh_md10ssdifftest.20130715_1dg_v3.20101123ns20130831 | y.00000 |  473882 | 0.569     |     549 | 
| meh_md10ssdifftest.20130715_1dg_v3.20120825ns20130831 | y.00000 |  473883 | 0.725     |     453 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121014ns20130831 | g.00000 |  473884 | 0.885     |     281 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121023ns20130831 | y.00000 |  473885 | 0.887     |     364 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121211ns20130831 | i.00000 |  473886 | 0.329     |    1074 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121215ns20130831 | y.00000 |  473889 | 0.883     |     194 | 
| meh_md10ssdifftest.20130715_1dg_v3.20130714ns20130831 | i.00000 |  473890 | 0.882     |    1284 | 

-- suggested config night stack --
| data_group                                                | filter  | diff_id | good_frac | sources |
+-----------------------------------------------------------+---------+---------+-----------+---------+
| meh_md10ssdifftest.20130715_1dg_v3.20101103nstest20130830 | g.00000 |  473564 | 0.902     |     149 | 
| meh_md10ssdifftest.20130715_1dg_v3.20101123nstest20130830 | y.00000 |  473565 | 0.639     |     386 | 
---> missing NS ---
| meh_md10ssdifftest.20130715_1dg_v3.20121014nstest20130830 | g.00000 |  473567 | 0.915     |     106 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121023nstest20130830 | y.00000 |  473568 | 0.919     |     448 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121211nstest20130830 | i.00000 |  473569 | 0.734     |     699 | 
| meh_md10ssdifftest.20130715_1dg_v3.20121215nstest20130830 | y.00000 |  473570 | 0.901     |     109 | 
| meh_md10ssdifftest.20130715_1dg_v3.20130714nstest20130830 | i.00000 |  473571 | 0.917     |    1112 | 

  • original config night stack
    zoom slice image order --  skycell.051 dropped 20120825 (3rd in sequence) to match loss in config night stack sample
    skycell.051 -- | 20101103 | 20101123 | 20121014 | 20121023 | 20121211 | 20121215 | 20130714 |
    skycell.055 -- | 20101103 | 20101123 | 20120825 | 20121014 | 20121023 | 20121211 | 20121215 | 20130714 |
    
skycell.051 skycell.055
  • suggested config night stack
    zoom slice image order -- except skycell.051 missing 20120825 (3rd in sequence)
    -- skycell.051 -- | 20101103 | 20101123 | 20121014 | 20121023 | 20121211 | 20121215 | 20130714 |
    -- skycell.055 -- | 20101103 | 20101123 | 20120825 | 20121014 | 20121023 | 20121211 | 20121215 | 20130714 |
    
skycell.051 skycell.055

Detection number comparison -- see table above for now, break down of numbers by good/bad flag types probably useful..

PV2 Night Stack Summary

Was at least a good test to help flush out issues with the 1D convolution method. Basically want to decide if there is an improvement without a decrease in robustness, maybe also use for reprocessing only.

  • hard cut for FWHM ~12 to match the camera stage rejection
    • does not fully work well, extra allowance needs to be made for edges by ~2 pixels.
    • will help prevent excessively large target PSF from being chosen when 1D convolution is used (select target PSF as maximum FWHM input)
    • the functional method used for making the LAP stacks may also be an alternative.
  • 1D convolution
    • appears equivalent in terms of the unconvolved stacks
    • will be more robust in avoiding crazy target PSF unlike the ISIS method that uses PSFEnvelope to determine and thus image rejections will also be more stable
  • N input warp minimum 2 (instead of >=4)
    • useful in filling in regions or actually producing stacks in nights with fewer inputs
  • SAFE false (single input pixels allowed) is somewhat uncertain
    • more area on edges is definitely seen, how useful it is as a single warp is the question versus just using the warps for them.

Can push any of these data products to the datastore under meh-tests


MD+3PI stack

Gene has suggested including overlapping 3PI exposures may improve the edges of the deeper stacks. A test sample run has been made with MD10 and 1D convolution for comparison to the pure MD10 refstack (would really only be used for the deep stacks, but refstacks are faster and easier to produce).

  • label, data_group -- MD10_3PI.teststack.20130728_1dg, MD10_3PI.teststack.20130728 (need to fix this)

Like the refstack itself, 2009 data is not used and only the exposures from 5/2010--6/2013 were used -- this reduced the number of input warps from ~22-->12

Edge image samples --

only MD MD+3PI
Error: Macro Image(.jpg,500px) failed
Attachment 'wiki:MD10.refstack.20130715: .jpg' does not exist.
Error: Macro Image(.jpg,500px) failed
Attachment 'wiki:MD10.refstack.20130715: .jpg' does not exist.

Detection comparisons --

Photometry comparisons --

MD+3PI Stack Summary

Attachments