Summary of MD10.refstack.20130715

(back to MediumDeepFields summary)

Similar to the revised MD10.refstack.20120804 (MD10.refstack.20120705) refstacks for the 2012 season, these include recent improvements for the PV2. Overall fewer inputs to the refstack were done this round and may not be as deep as last season.

With the concurrent, new ipp-20130712 tag including PV2 modifications, MD10 was the guinea pig and includes additional testing and config modifications

• PV2 1-D convolution initially failed for MD, the original ISIS method was used for the active refstack, MD10.refstack.20130715_1dg_v3 is working version of 1D for MD10
• PV2 dynamic masks working, however many ghosts are not masked due to their centers being in gaps and will need to be addressed in PV3
• stack photometry appears to have odd/poor solutions for normal PSF photometry on some skycells (and seeing more oddities showing up in normal nightly processing..)
• long outstanding change to the nightly stack definition -- in progress
• combining MD with overlapping 3PI to improve the edge mess -- in progress

MD10 observations:

• starting late June 2013 with pre-observations, regular observations mid-July
• end July 2012 through start Jan. 2013
• mid-July 2011 through early Dec. 2011
• late June 2010 through early Dec. 2010
• end Aug. 2009 through start Jan. 2010
• (some 2008 but pre-survey)

Filter	N total	N FWHM<8 pixels	N (FWHM<8 pixels for 2012	N FWHM<6 pixels	N (FWHM<6 pixels for 2012
g	627	591	120	478	72
r	613	584	111	515	83
i	608	565	145	483	117
z	660	610	180	559	150
y	557	504	217	396	150

• 2012 wasn't a great season for MD10, but better than MD09

Status:

• 07/15/13 selected exposure reprocessing chip->warp for refstack started
• 07/18 reftests finished. detrend issues OTA47, redoing subset of exposures for most of 2010 to use revised video dark.. full refstack started
• 07/19 refstack finished, checked and problem skycells found/fixed if possible. staticsky single filter only finished and mappings started -- seeing psphot issue reported in DRAVG still..
• (07/20) mappings finished (some oddities showing up), 1-d conv PV2 rejecting more than expected, diffim testing, non-1d conv refstack setup , recently missed exposures setup for diffims and finished
• (07/21) reftest+refstack to distribution
• 07/25 finished looking into oddities for now, updating wiki
• 8/17 improved 1D convolution stacks made (MD10.refstack.20130715_1dg_v3)

MD10.V3 Processing (chip->warp + refstack)

Exposure reprocessing was done for only a limited number (~150) of the best images from 2010-2012 to provide a balance of N input warps versus best seeing, excluding exposures with large background, low zeropoints and large elongation.

Processing chip-warp and the refstack itself has the same label, data_group

• label: MD10.refstack.20130715 --> once set as active refstack for diffims, stack label will change to MD10.refstack, chip/cam/warp label will remain the same
• data_group: MD10.refstack.20130715

The ipp-20130712 tag used

• GPC1_Detrend_Documentation -- new video darks
• dynamic masks now propagating through to warps -- ghost mask is slightly offset or missing at times
• 1-d convolution option for PV2 -- more rejection than expected
• changes to psphot, PSF_MAJOR,MINOR root(2) different but appropriate now and 2*root(2) for FWHM; may be stability issue in PSF fitting
• SCL_ keyword keeps scaling value and not reassigned to 1. when stack rejected now
• new reduction class with clear name MD_REF_STACK
• SAFE config now set to false

Histograms of the FWHM major and minor and elongation from camera stage value (full exposure) for the sample of reprocessed exposures (red) and used for refstack.20130715 (top blue) and used for refstack.20120804 (bottom blue) in grizy order

Test stack (reftest) sample also made again for skycell.055 with different number of input warps.

• label: MD10.reftest.20130715
• data_group:
  • MD10.reftest.20130715.c#f<FWHM cut> where #=comparison run, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
  • MD10.reftest.20130715.all.<FWHM cut> where all means 05/2010+2011+2012 data possible, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
  • MD10.reftest.20130715.2012.<FWHM cut> where 2012 means 2012 data only for season stack example case, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
• the main field center shift was through ~Dec 2009, but is a very small shift. Excluding 2009 since usually not very good, or uncharacteristically too good (avoiding problems basically)
• c1 cuts

  filter: (FWHM set), elongation, zeropoint, background limits
  y: (3.5, 3.8, 4.1, 4.6), 1.3, 22.8, 5000 -- bkg bit higher
  z: (3.6, 3.7, 3.8, 4.0), 1.3, 24.0, 4000 -- bkg bit higher 
  i: (3.7, 3.9, 4.1, 4.3), 1.25, 24.3, 2000 --
  r: (4.2, 4.4, 4.5, 4.7), 1.25, 24.3, 600 -- 
  g: (4.6, 4.8, 5.0, 5.2), 1.25, 24.3, 400 -- 
  --> refstack starts with 5.0, 4.5, 4.1, 3.8, 4.1 pixels for upper cut in grizy (may go smaller based on MEAN+1.0*STDDEV)
  

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055) after running psphot on the stacks.

| data_group                        | filter  | stack_id | warps_in | warps_rej |  FWHM_MAJ   IQ_FW1 (pixels)
+-----------------------------------+---------+----------+----------+-----------+
--> 2013 refstack
| MD10.refstack.20130715            | g.00000 |  2294005 |       94 |        16 | 	 5.01  4.03 
| MD10.refstack.20130715            | r.00000 |  2293945 |       84 |        15 | 	 4.52  3.74 
| MD10.refstack.20130715            | i.00000 |  2293885 |       86 |        11 | 	 3.78  3.10 
| MD10.refstack.20130715            | z.00000 |  2293825 |       85 |        16 | 	 3.16  2.99 
| MD10.refstack.20130715            | y.00000 |  2293765 |       87 |        15 | 	 3.42  3.14 

--> reftest comparisons						
| MD10.reftest.20130715.all.49      | g.00000 |  2291338 |       95 |	     20 | 	 4.74  3.88 
| MD10.reftest.20130715.2012.49     | g.00000 |  2291339 |       13 |	      4 | 	 4.77  3.56 
| MD10.reftest.20130715.c1f46       | g.00000 |  2291365 |       25 |	      6 |	 4.82  3.59 
| MD10.reftest.20130715.c1f48       | g.00000 |  2291366 |       56 |	      9 | 	 4.71  3.84 
| MD10.reftest.20130715.c1f50       | g.00000 |  2291367 |       95 |	     17 | 	 4.48  4.04 
| MD10.reftest.20130715.c1f52       | g.00000 |  2291368 |      124 |	     21 | 	 4.95  4.15 
| MD10.reftest.20130715.all.44      | r.00000 |  2291336 |       90 |	     17 | 	 4.26  3.56 
| MD10.reftest.20130715.2012.44     | r.00000 |  2291337 |       17 |	      3 | 	 3.88  3.12 
| MD10.reftest.20130715.c1f42       | r.00000 |  2291361 |       31 |	      4 | 	 4.21  3.38 
| MD10.reftest.20130715.c1f44       | r.00000 |  2291362 |       65 |	     11 | 	 4.41  3.66 
| MD10.reftest.20130715.c1f45       | r.00000 |  2291363 |       84 |	     15 | 	 4.38  3.74 
| MD10.reftest.20130715.c1f47       | r.00000 |  2291364 |      121 |	     16 | 	 4.51  3.83 
| MD10.reftest.20130715.all.40      | i.00000 |  2291334 |       90 |	     12 | 	 3.74  3.06 
| MD10.reftest.20130715.2012.40     | i.00000 |  2291335 |       13 |	      2 | 	 3.33  3.20 
| MD10.reftest.20130715.c1f37       | i.00000 |  2291357 |       30 |	      6 | 	 3.48  2.98 
| MD10.reftest.20130715.c1f39       | i.00000 |  2291358 |       54 |	     10 | 	 4.61  3.02 
| MD10.reftest.20130715.c1f41       | i.00000 |  2291359 |       87 |	     10 | 	 3.77  3.10 
| MD10.reftest.20130715.c1f43       | i.00000 |  2291360 |      124 |	     20 | 	 3.86  3.17 
| MD10.reftest.20130715.all.37      | z.00000 |  2291332 |       85 |	     21 | 	 3.39  2.95 
| MD10.reftest.20130715.2012.37     | z.00000 |  2291333 |       18 |	      3 | 	 3.58  3.03 
| MD10.reftest.20130715.c1f36       | z.00000 |  2291353 |       46 |	      8 | 	 3.36  2.93 
| MD10.reftest.20130715.c1f37       | z.00000 |  2291354 |       66 |	     11 | 	 3.43  2.96 
| MD10.reftest.20130715.c1f38       | z.00000 |  2291355 |       85 |	     18 | 	 3.15  2.98 
| MD10.reftest.20130715.c1f40       | z.00000 |  2291356 |      118 |	     23 | 	 3.22  3.04 
| MD10.reftest.20130715.all.38      | y.00000 |  2291329 |       82 |	     20 |	 3.37  3.07 
| MD10.reftest.20130715.2012.38     | y.00000 |  2291331 |       28 |	      4 | 	 3.37  2.89 
| MD10.reftest.20130715.c1f35       | y.00000 |  2291349 |       27 |	      5 | 	 3.29  2.85 
| MD10.reftest.20130715.c1f38       | y.00000 |  2291350 |       59 |	     10 | 	 3.38  3.06 
| MD10.reftest.20130715.c1f41       | y.00000 |  2291351 |       88 |	     17 | 	 3.41  3.15 
| MD10.reftest.20130715.c1f46       | y.00000 |  2291352 |      118 |	     18 | 	 3.50  3.26 
	
--> previous 2012 refstack				
| MD10.refstack.20120804            | g.00000 |  1108704 |      103 |	     36 | 	 5.08  4.10 
| MD10.refstack.20120804            | r.00000 |  1112755 |       98 |	     21 | 	 4.25  3.86 
| MD10.refstack.20120804            | i.00000 |  1112815 |       89 |	     13 | 	 3.50  3.15 
| MD10.refstack.20120804            | z.00000 |  1112695 |      100 |	     19 | 	 3.41  3.12 
| MD10.refstack.20120804            | y.00000 |  1108632 |       71 |	     11 | 	 3.64  3.32 

As before, the FWHM_MAJOR with the central skycell can be variable and more so with largely different input sets. Comparison with the psf_major/minor maps are more useful between refstack versions.

Problems

• as with all past refstacks, generally edge skycells have issues -- need better containment of those still. (see core coverage below).
• like others, greater % of inputs rejected (20-30%) so per pixel input is less, but still of order 30 inputs per pixel
• some convolved stacks have poorer FWHM PSF but not significant enough to be likely improved by reprocessing using DEEP_STACK config as in earlier MD fields
• skycell notes:
  • y: s025,026,036,037,047,057 bad cell bands still.. s032 odd patch
  • z: s012 quality fault, rerun with same MD_REF_STACK but change elong cut <1.25 from 1.3 to remove odd exposures messing up Target PSF; s046,063,065,066 odd,raised background (bad cell?)
  • i: s065 like some other cells, some odd raised background (bad cell? glint?)
  • r: s014,015,048,064,065,066,084 poor ghost mask/mess
  • g: s014,015,022,023,027,032,037,045,048,058,064,065,067,085 poor ghost mask/mess; s038 edge really ratty; s044,054 poor cell arc (wonder if could/should mask cell as seen in MD10.refstack.20120705)

Ghost mask example -- skycell.045 g-band has a cluster of donuts, sample of example input warps

neb://any/gpc1/MD10.refstack.20130715/MD10.V3/skycell.045/MD10.V3.skycell.045.stk.2293997

--> one is missing 
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5416g0465o.204618/o5416g0465o.204618.wrp.814962.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5776g0435o.372304/o5776g0435o.372304.wrp.815009.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5779g0510o.373485/o5779g0510o.373485.wrp.815020.skycell.045
... likely others

--> slightly missed position, ~40 pixels
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5425g0536o.208404/o5425g0536o.208404.wrp.814971.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5443g0459o.219428/o5443g0459o.219428.wrp.814976.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5443g0461o.219430/o5443g0461o.219430.wrp.814978.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5443g0456o.219426/o5443g0456o.219426.wrp.814979.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5443g0458o.219427/o5443g0458o.219427.wrp.814981.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5767g0498o.368083/o5767g0498o.368083.wrp.815004.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5776g0433o.372302/o5776g0433o.372302.wrp.815008.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5776g0437o.372306/o5776g0437o.372306.wrp.815010.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5776g0438o.372307/o5776g0438o.372307.wrp.815012.skycell.045
neb://stsci05.0/gpc1/MD10.refstack.20130715/o5779g0505o.373480/o5779g0505o.373480.wrp.815013.skycell.045
... and others

Core coverage -- one of the shifted pointings so different from normal distribution of skycells (also see detection efficiency maps below)

082-087
071-078
061-068
051-058
041-048
031-038
021-028
012-017

• skycells with fewer inputs (typical for edges): 012,017,021,028,071,078,082,087
• skycells with <50% or more of typical input number is filter dependent - usually s017,028,078,087; sometimes none except s078

Stackphot (staticsky) oddity/problem -- skycell.024 z-band main example (other hot region/skycells in plots below) in the single filter photometry cases (1x). The 5 filter (Nx) ones are a little better and without an odd deteff in z-band, but still show hot spots more than previous tag.

-- log snippet from ipp-20130715 -- neb://any/gpc1/MD10.refstack.20130715.1x20130719/MD10.V3/skycell.024/MD10.V3.skycell.024.sky.446833.log
        fwhm (psf): 6.367312,2.439518 (moments): 3.011427,3.011427
        PSF extra parameter 1: -0.232193 +/- 0.461459
    select psf model: 1.384724 sec
      psf model PS_MODEL_PS1_V1, ApResid: 0.141451 +/- 0.008141
    built models for 4767 objects: 0.210036 sec (12358026 pixels)


-- log snippet from previous ipp-20130307 -- neb://any/gpc1/MD10.refstack.20130715.1x20130724.redoipp130307/MD10.V3/skycell.024/MD10.V3.skycell.024.sky.446953.log
        fwhm (psf): 3.516740,3.209410 (moments): 3.032322,3.032322
        PSF extra parameter 1: -0.052530 +/- 0.030366
    select psf model: 4.576834 sec
      psf model PS_MODEL_PS1_V1, ApResid: 0.121756 +/- 0.009789
    built models for 4905 objects: 0.252904 sec

Refstack Summary

Stack phomtometry on the refstacks done through the staticsky and skycal stages using the unconvolved images

• single filter mode so basic psphot with deteff measurements that has the zero-point recalculated for the test plots below
  • label: MD10.refstack.20130715, MD10.refstack.20130715
  • data_group (single filter): MD10.refstack.20130715.1x20130719, MD10.refstack.20130715.1x20130719.cal20130719
• multi-filter (Nx) also run
  • label: MD10.refstack.20130715.5x20130725, MD10.refstack.20130715.5x20130725.cal20130725
  • data_group: MD10.refstack.20130715.5x20130725, MD10.refstack.20130715.5x20130725.cal20130725 (all skycells have 5 filters matched)
  • on datastore under ps1-md-staticsky, but only the staticsky version, the skycal product still has no method for distribution

Source photometry psf_major,_minor maps (grizy): all new plots here scaled by root(2) to match past scaling (but not on the plot label yet)

Magnitude of sources@5-sigma maps (grizy):

Detection efficiency magnitude @50% recovery (500 fakes, grizy): 0 rej for all now that the SCL_ keyword has its true value other than 1. when rejected in stacking

---

1D convolution mode PV2

Same naming format as generally above, adding _1dg to the data_group (as will be seen on the datastore)

• label: MD10.reftest.20130715
• data_group:
  • MD10.reftest.20130715.c#f<FWHM cut>_1dg where #=comparison run, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
  • MD10.reftest.20130715.all.<FWHM cut>_1dg where all means 05/2010+2011+2012 data possible, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
  • MD10.reftest.20130715.2012.<FWHM cut>_1dg where 2012 means 2012 data only for season stack example case, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
• c1 cuts (same as above)

Full refstack:

• label: MD10.refstack.20130715_1dg
• data_group: MD10.refstack.20130715_1dg

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055) after running psphot on the stacks.

| data_group                        | filter  | stack_id | warps_in | warps_rej |  FWHM_MAJ   IQ_FW1 (pixels)
+-----------------------------------+---------+----------+----------+-----------+
| MD10.refstack.20130715_1dg        | y.00000 |  2294065 |       87 |        15 | 
| MD10.refstack.20130715_1dg        | z.00000 |  2294125 |       85 |        32 | 
| MD10.refstack.20130715_1dg        | i.00000 |  2294185 |       86 |        14 | 
| MD10.refstack.20130715_1dg        | r.00000 |  2294245 |       84 |        24 | 
| MD10.refstack.20130715_1dg        | g.00000 |  2294305 |       94 |        27 | 

| MD10.reftest.20130715.all_1dg.38  | y.00000 |  2291330 |       82 |        14 | 
| MD10.reftest.20130715.2012_1dg.38 | y.00000 |  2291340 |       28 |         3 | 
| MD10.reftest.20130715.all_1dg.37  | z.00000 |  2291341 |       85 |        30 | 
| MD10.reftest.20130715.2012_1dg.37 | z.00000 |  2291342 |       18 |         2 | 
| MD10.reftest.20130715.all_1dg.40  | i.00000 |  2291343 |       90 |        17 | 
| MD10.reftest.20130715.2012_1dg.40 | i.00000 |  2291344 |       13 |         1 | 
| MD10.reftest.20130715.all_1dg.44  | r.00000 |  2291345 |       90 |        15 | 
| MD10.reftest.20130715.2012_1dg.44 | r.00000 |  2291346 |       17 |         3 | 
| MD10.reftest.20130715.all_1dg.49  | g.00000 |  2291347 |       95 |        23 | 
| MD10.reftest.20130715.2012_1dg.49 | g.00000 |  2291348 |       13 |         5 | 
| MD10.reftest.20130715.c1f35_1dg   | y.00000 |  2291369 |       27 |         5 | 
| MD10.reftest.20130715.c1f38_1dg   | y.00000 |  2291370 |       59 |         9 | 
| MD10.reftest.20130715.c1f41_1dg   | y.00000 |  2291371 |       88 |        17 | 
| MD10.reftest.20130715.c1f46_1dg   | y.00000 |  2291372 |      118 |        34 | 
| MD10.reftest.20130715.c1f36_1dg   | z.00000 |  2291373 |       46 |        17 | 
| MD10.reftest.20130715.c1f37_1dg   | z.00000 |  2291374 |       66 |        23 | 
| MD10.reftest.20130715.c1f38_1dg   | z.00000 |  2291375 |       85 |        17 | 
| MD10.reftest.20130715.c1f40_1dg   | z.00000 |  2291376 |      118 |        25 | 
| MD10.reftest.20130715.c1f37_1dg   | i.00000 |  2291377 |       30 |         7 | 
| MD10.reftest.20130715.c1f39_1dg   | i.00000 |  2291378 |       54 |        18 | 
| MD10.reftest.20130715.c1f41_1dg   | i.00000 |  2291379 |       87 |        17 | 
| MD10.reftest.20130715.c1f43_1dg   | i.00000 |  2291380 |      124 |        31 | 
| MD10.reftest.20130715.c1f42_1dg   | r.00000 |  2291381 |       31 |         8 | 
| MD10.reftest.20130715.c1f44_1dg   | r.00000 |  2291382 |       65 |        17 | 
| MD10.reftest.20130715.c1f45_1dg   | r.00000 |  2291383 |       84 |        26 | 
| MD10.reftest.20130715.c1f47_1dg   | r.00000 |  2291384 |      121 |        31 | 
| MD10.reftest.20130715.c1f46_1dg   | g.00000 |  2291385 |       25 |         9 | 
| MD10.reftest.20130715.c1f48_1dg   | g.00000 |  2291386 |       56 |         9 | 
| MD10.reftest.20130715.c1f50_1dg   | g.00000 |  2291387 |       95 |        26 | 
| MD10.reftest.20130715.c1f52_1dg   | g.00000 |  2291388 |      124 |        35 | 

PROBLEM -- new 1D convolution code produced unexpected excess rejection and cannot be used for the refstack -- investigate further below

1D Excess Rejection

data_group/log:

MD10.reftest.20130715.c1f46 -- 18 rej, all cut at start
neb://any/gpc1/MD10.reftest.20130715/MD10.V3/skycell.055/MD10.V3.skycell.055.stk.2291352.log

MD10.reftest.20130715.c1f46_1dg -- 34 rej, so 14 additionally rej by chi2 
neb://any/gpc1/MD10.reftest.20130715/MD10.V3/skycell.055/MD10.V3.skycell.055.stk.2291372.log

Target PSF ~4.67 pixels

input index 16,37,38,39,45,46,49,50,62,65,70,71,74,93,96,111 -- many close to the 4.67 target (or even over at 4.7 for 50)

range 4.5-4.7 pixels

possible tests:

• manually run with manually set target psf 4.8 to see if all pass
• increase MATCH.REJ from 4 to >10 -- would be necessary to include many rej other than #50

Stack Photometry 1D version

Like the default refstack, also run for the 1D convolution version and the multi-filter version is on the datastore (only staticsky product as the skycal has no method for distribution)

• label (staticsky, skycal): MD10.refstack.20130715_1dg.5x20130726, MD10.refstack.20130715_1dg.5x20130726.cal20130726
• data_group: MD10.refstack.20130715_1dg.5x20130726, MD10.refstack.20130715_1dg.5x20130726.cal20130726

Odd Features in 1D version

input warp

1D conv stack

1D conv mask

unconv .num.

1D .num.

unconv .exp.

1D .exp

Appears that the .exp. files for the >30 input stacks are mostly NAN. Suspect the compression had a problem. Has been happening for over a year. Added a .EXP.NOCOMP option (one already existed for EXPWT) and if a stack is made uncompressed (as is with deep stacks) then this is used.

• filesize change 933 KB -->163 MB, a large change without compression, but only for deep stacks (MD ref and deep types only) --

1D convolution v3 OK

r35925 appears to produce similar rejection to the original ISIS method, and includes a fix the the other rejection issue found in the night stack sample below. Target PSF for convolved stack is now set to the max FWHM of the input warps that pass all input cuts with an additional epsilon of 0.1 pixels.

    PSF.TARGET.AS.MAX  BOOL  TRUE
    PSF.TARGET.AS.MAX.EPSILON F32 0.1

• label and data_group -- MD10.refstack.20130715_1dg_v3

| data_group                    | filter  | stack_id | skycell_id  | warps_in | warps_rej |
+-------------------------------+---------+----------+-------------+----------+-----------+

| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455233 | skycell.051 |       87 |        16 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455234 | skycell.052 |       87 |        13 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455235 | skycell.053 |       87 |        14 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455236 | skycell.054 |       87 |        11 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455237 | skycell.055 |       87 |        15 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455238 | skycell.056 |       87 |        16 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455239 | skycell.057 |       87 |        14 | 
| MD10.refstack.20130715_1dg_v3 | y.00000 |  2455240 | skycell.058 |       87 |        10 | 

| MD10.refstack.20130715 | y.00000 |  2293761 | skycell.051 |       87 |        19 | 
| MD10.refstack.20130715 | y.00000 |  2293762 | skycell.052 |       87 |        13 | 
| MD10.refstack.20130715 | y.00000 |  2293763 | skycell.053 |       87 |        15 | 
| MD10.refstack.20130715 | y.00000 |  2293764 | skycell.054 |       87 |        11 | 
| MD10.refstack.20130715 | y.00000 |  2293765 | skycell.055 |       87 |        15 | 
| MD10.refstack.20130715 | y.00000 |  2293766 | skycell.056 |       87 |        16 | 
| MD10.refstack.20130715 | y.00000 |  2293767 | skycell.057 |       87 |        15 | 
| MD10.refstack.20130715 | y.00000 |  2293768 | skycell.058 |       87 |        12 |

| MD10.refstack.20130715 | y.00000 |  2293763 | skycell.053 |       87 |        15 | 
| MD10.refstack.20130715_1dg | y.00000 |  2294063 | skycell.053 |       87 |        29 | 
| MD10.refstack.20130715_1dg_v2 | y.00000 |  2450747 | skycell.053 |       87 |        29 | 

---

Night Stacks and PV2

Configuration modifications for PV2 and 1D convolution should probably be applied to nightly stacks as well as other additions

• SAFE false --
• 1D convolution --
• MATCH.REJ 4.0 --
• PSF.INPUT.MAX 12.0 -- normally limited by camera stage of only processing exposures with FWHM <12 pixels, but there can be cases of edge skycells being just poor or mis-measured and probably do not want to include those
• N input warp minimun 2 -- was originally set to 4 for magic restrictions, then on partial field nights will end up missing stacks and a stack of two warps is better than none. For nights of 1 warp, there are WS diffims. All reprocessed night stacks use 2 as the minimum.

Test 1D Night Stacks

Night 20130707 g and z -- using ipp-trunk build r35868 with OPTIMUM.TOL setup to relax rejection for FWHM near target PSF and config modifications above

• label: meh_md10nstest20130707, meh_md10nstest20130707_1dg
• data_group: meh_md10nstest20130707<f>, meh_md10nstest20130707<f>_1dg -- <f> is g or z filter

Another 1D Rejection Problem

Another rejection problem -- excess rejection even with trunk modification to use OPTIMUM.TOL, entire images (not convolved because > target PSF) are being fully rejected in pixel rejection stage. Also from example log notice Target seeing FWHM appears not well matched and probably the actual trigger for this (manually setting larger and is fine). Wonder if is related to the diffim PSF problems with recent psphot PSF fixes.

| data_group                  | filter  | stack_id | warps_in | warps_rej |
+-----------------------------+---------+----------+----------+-----------+
| meh_md10nstest20130707g     | g.00000 |  2355589 |        8 |         0 | 
| meh_md10nstest20130707z     | z.00000 |  2355649 |        8 |         0 | 
| meh_md10nstest20130707g_1dg | g.00000 |  2355709 |        8 |         0 | 
| meh_md10nstest20130707z_1dg | z.00000 |  2355769 |        8 |         5 | 

-- not just central poor PSF skycells either, see at various % over FPA in both filters

| meh_md10nstest20130707g_1dg | g.00000 |  2355692 | skycell.034 |        8 |         4 | 

neb://any/meh/meh_md10nstest20130707z_1dg/MD10.V3/skycell.055/MD10.V3.skycell.055.stk.2355769.log

      Input seeing FWHMs:
      Input 0: 5.394163
      Input 1: 5.390012
      Input 2: 5.393055
      Input 3: 5.100585
      Input 4: 5.308995
      Input 5: 4.463570
      Input 6: 4.224805
      Input 7: 3.980880
      Input FWHMs : 4.907008 +/- 0.588663
      PSF FWHM distribution: Used simple clip method.  Limit set to 12.000000.

      Target seeing FWHM: 4.742117

      Stage 2: Generate Convolutions and Save: 161.477153 sec
      Summary of images for initial combination:
      Image 0:  Good.
      Image 1:  Good.
      Image 2:  Good.
      Image 3:  Good.
      Image 4:  Good.
      Image 5:  Good.
      Image 6:  Good.
      Image 7:  Good.
      Stage 3: Make Initial Stack: 290.986779 sec
    Rejection on image 0 didn't work --- reject entire image.
      Time to perform rejection on image 0: 0.304300 sec
    Rejection on image 1 didn't work --- reject entire image.
      Time to perform rejection on image 1: 0.272519 sec
    Rejection on image 2 didn't work --- reject entire image.
      Time to perform rejection on image 2: 0.274936 sec
    Rejection on image 3 didn't work --- reject entire image.
      Time to perform rejection on image 3: 0.274820 sec
    Rejection on image 4 didn't work --- reject entire image.
      Time to perform rejection on image 4: 0.275285 sec
      Convolve image: 5.637316 sec
      150010 pixels rejected from image 5 (0.4%)
      Time to perform rejection on image 5: 6.448552 sec
      Convolve image: 5.874424 sec
      102450 pixels rejected from image 6 (0.3%)
      Time to perform rejection on image 6: 6.665843 sec
      Convolve image: 5.654609 sec
      93433 pixels rejected from image 7 (0.2%)
      Time to perform rejection on image 7: 6.459099 sec

      Stage 4: Pixel Rejection: 21.048442 sec
      Summary of images for final combination:
      Image 0:  Rejection exceeded threshold.
      Image 1:  Rejection exceeded threshold.
      Image 2:  Rejection exceeded threshold.
      Image 3:  Rejection exceeded threshold.
      Image 4:  Rejection exceeded threshold.
      Image 5:  Good.
      Image 6:  Good.
      Image 7:  Good.
      Stage 5: Final Stack: 12.869377 sec

Odd Core NAN Masking Oddity in 1D version

If the target PSF is << input PSF then even though image is triggered to not be convolved it is rejected in the pixel rejection stage. The resulting images have many sources with NAN cores.

• unknown if this is due to the odd rejection case due to PSF > target PSF or if due to bug when image rejection done in pixel rejection stage

Masking of poor and suspect pixels may have issues

PV2 Update Test

Part of PV2 is the inclusion of the dynamic mask during chip->warp updates by also updating the camera stage SMF.

Sample --

PV2 Update Masking Losses

Odd mask lost in PV2 updates -- seems to be limited to

    PM_SOURCE_MODE_CR_LIMIT
    PM_SOURCE_MODE_BLEND

Was suggested that possibly from photometry not being run on update -- since masks being remade in PV2 updates then lost, while a normal update uses the already defined mask? Since PV2 is mainly for improved inputs to stacks, should be minor and just rejected? (except for low N..)

Example -- one masked CR is lost (full reprocessing retains it while the PV2 update method loses it) and one is gained (as with the full reprocessing and should be)

exposure.exp_id: o5503g0240o.250240
new chip/cam/warp_id = 883673 / 854492 / 835018
old/pv1/up chip/cam/warp_id = 810251 / 786993 / 766825
skycell.045

Image panels left to right -- new reprocessing, pv1 normal update, pv2 w/ camera update, clean and normal update of pv2

• Mask panels left to right -- (same sequence)

PV2 Update Night Stack

PV2 Update Night Stack SSdiff

MD+3PI stack