Summary of MD01.refstack.20130731

(back to MediumDeepFields summary)

Similar to the revised MD01.refstack.20120803 (MD01.refstack.20120803) refstacks for the 2012 season, these include recent improvements for the PV2. Overall fewer inputs to the refstack were done this round and may not be as deep as last season.

• PV2 1D convolution not done since problematic with MD fields
• appear to still be psphot problems in poor PSF solution for skycell/regions that may not be realistic (i.e., not in center).

MD01 observations:

• starting late July 2013 with pre-observations, regular observations early Aug.
• 08/03/2012--02/12/2013
• 08/04/2011--01/15/2012
• 07/28/2010--02/05/2011
• 08/29/2009--01/26/2010
• (some 2008 but pre-survey)

Filter	N total	N FWHM<8 pixels	N (FWHM<8 pixels for 2012	N FWHM<6 pixels	N (FWHM<6 pixels for 2012
g	604	560	85	444	65
r	574	540	86	458	71
i	725	673	146	546	99
z	943	900	281	774	237
y	741	643	298	552	261

• 2012 wasn't a great season for MD01 gr, but better than MD09

Status:

• 07/31/2013 selected exposure reprocessing chip->warp for refstack started
• 08/02 exposures finished (processing issues).
• 08/05 reftests started, only ISIS since 1D rejections still not working properly
• 08/07 refstack finished, checked and working on problem skycells found/fixed if possible.
• 08/08 staticsky single filter only finished and mappings started
• 08/09 mappings finished, field pre-obs exposures setup for diffims and finished, reftest+refstack to distribution
• 08/12 updating wiki

MD01.V3 Processing (chip->warp + refstack)

Exposure reprocessing was done for only a limited number (~150) of the best images from 2010-2012/2013 to provide a balance of N input warps for some depth minimum and assist poor pixel rejection versus best seeing, excluding exposures with large background, low zeropoints and large elongation.

Processing chip-warp and the refstack itself has the same label, data_group

• label: MD01.refstack.20130731 --> once set as active refstack for diffims, stack label will change to MD01.refstack, chip/cam/warp label will remain the same
• data_group: MD01.refstack.20130731

The ipp-20130712 tag used

• GPC1_Detrend_Documentation -- new video darks as started with MD10.refstack.20130731
• dynamic masks now propagating through to warps -- ghost mask is slightly offset or missing at times -- will not be fixed for PV2 and will have ghost/junk clusters still (worse with fewer inputs)
• 1-D convolution PV2 -- more rejection than expected and not in use yet for MD
• changes to psphot, PSF_MAJOR,MINOR root(2) different but appropriate now and 2*root(2) for FWHM; may be stability issue in PSF fitting
• SCL_ keyword fixed back to if rejected the scaling value changed -- now 0 instead of previous value of 1 (as it has been before the bug) -- but not fixed in time for MD01.refstack.20130731, will be for 1D
• new reduction class with clear name MD_REF_STACK
• SAFE config now set to false -- has helped to fill in ratty edges

Histograms and scatter plots of N exposures and their FWHM major, elongation, zeropoint and background by MJD from the camera stage (i.e., full exposure stats) for the sample of reprocessed exposures (red) and used for refstack.20130731 (top blue) and used for refstack.20120803 (bottom blue) in grizy order. This replaces the simple histograms of the FWHM major, minor and elongation.

Test stack (reftest) sample also made again for skycell.055 with different number of input warps.

• label: MD01.reftest.20130731
• data_group:
  • MD01.reftest.20130715.c#f<FWHM cut> where #=comparison run, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
  • MD01.reftest.20130715.all.<FWHM cut> where all means 05/2010+2011+2012/2013 data possible, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
  • MD01.reftest.20130715.2012.<FWHM cut> where 2012 means 2012/2013 data only for season stack example case, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
• excluding 2009 since usually not very good, or uncharacteristically too good (avoiding problems basically)
• c1 cuts

  filter: (FWHM set), elongation, zeropoint, background limits
  y: (3.6, 3.7, 3.8, 4.0), 1.3, 23.0, 4000 -- 
  z: (3.6, 3.8, 4.0, 4.2), 1.3, 24.0, 3000 -- 
  i: (3.8, 4.0, 4.1, 4.3), 1.25, 24.3, 2000 --
  r: (4.1, 4.4, 4.6, 4.8), 1.25, 24.3, 600 -- 
  g: (4.8, 5.0, 5.2, 5.5max), 1.25, 24.3, 400 -- 
  --> refstack starts with  pixels for upper cut in grizy (may go smaller based on MEAN+1.0*STDDEV)
  

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055) after running psphot on the stacks.

| data_group                        | filter  | stack_id | warps_in | warps_rej |  FWHM_MAJ   IQ_FW1 (pixels)
+-----------------------------------+---------+----------+----------+-----------+
--> 2013 refstack
| MD01.refstack.20130731            | y.00000 |  2392474 |       82 |        14 | 	 3.43  3.25
| MD01.refstack.20130731            | z.00000 |  2392534 |       89 |        17 | 	 3.37  3.12
| MD01.refstack.20130731            | i.00000 |  2392594 |       79 |        15 | 	 4.02  3.14
| MD01.refstack.20130731            | r.00000 |  2392654 |       85 |        16 | 	 4.46  3.80
| MD01.refstack.20130731            | g.00000 |  2392714 |       82 |        13 | 	 4.79  4.26

--> reftest comparisons							
| MD01.reftest.20130731.all.37      | y.00000 |  2383927 |       79 |        20 | 	 3.36  3.18
| MD01.reftest.20130731.2012.37     | y.00000 |  2383928 |       28 |         9 |	 3.38  3.10
| MD01.reftest.20130731.c1f36       | y.00000 |  2383947 |       36 |         6 | 	 3.41  3.12
| MD01.reftest.20130731.c1f37       | y.00000 |  2383948 |       59 |        11 | 	 3.40  3.17
| MD01.reftest.20130731.c1f38       | y.00000 |  2383949 |       82 |        14 | 	 3.43  3.27
| MD01.reftest.20130731.c1f40       | y.00000 |  2383950 |      108 |        18 |  	 3.45  3.29
| MD01.reftest.20130731.all.38      | z.00000 |  2383929 |       81 |	     18 | 	 3.39  3.01
| MD01.reftest.20130731.2012.38     | z.00000 |  2383930 |       19 |	      3 | 	 3.67  3.21
| MD01.reftest.20130731.c1f36       | z.00000 |  2383951 |       34 |	      8 | 	 3.29  3.05
| MD01.reftest.20130731.c1f38       | z.00000 |  2383952 |       60 |	     11 | 	 3.38  3.06
| MD01.reftest.20130731.c1f40       | z.00000 |  2383953 |       89 |	     19 | 	 3.45  3.11
| MD01.reftest.20130731.c1f41       | z.00000 |  2383954 |      122 |	     20 | 	 3.46  3.19
| MD01.reftest.20130731.all.41      | i.00000 |  2383931 |       90 |	     21 | 	 5.85  3.12
| MD01.reftest.20130731.2012.41     | i.00000 |  2383932 |       17 |	      4 | 	 3.65  3.14
| MD01.reftest.20130731.c1f38       | i.00000 |  2383955 |       34 |	      6 | 	 3.58  3.04
| MD01.reftest.20130731.c1f40       | i.00000 |  2383956 |       66 |	     13 | 	 3.66  3.10
| MD01.reftest.20130731.c1f41       | i.00000 |  2383957 |       79 |	     15 | 	 3.72  3.14
| MD01.reftest.20130731.c1f43       | i.00000 |  2383958 |      115 |	     23 | 	 3.48  3.21
| MD01.reftest.20130731.all.45      | r.00000 |  2383933 |       86 |	     18 | 	 4.46  3.77
| MD01.reftest.20130731.2012.45     | r.00000 |  2383934 |        9 |	      2 | 	 4.51  3.35
| MD01.reftest.20130731.c1f41       | r.00000 |  2383959 |       29 |	      4 | 	 4.42  3.68
| MD01.reftest.20130731.c1f44       | r.00000 |  2383960 |       61 |	      8 | 	 4.47  3.79
| MD01.reftest.20130731.c1f46       | r.00000 |  2383961 |       85 |	     16 | 	 4.46  3.79
| MD01.reftest.20130731.c1f48       | r.00000 |  2383962 |      113 |	     23 | 	 4.40  3.82
| MD01.reftest.20130731.all.50      | g.00000 |  2383935 |       77 |	     15 | 	 5.00  4.06
| MD01.reftest.20130731.2012.50     | g.00000 |  2383936 |       11 |	      2 | 	 4.78  3.64
| MD01.reftest.20130731.c1f48       | g.00000 |  2383963 |       31 |	      5 | 	 5.09  4.04
| MD01.reftest.20130731.c1f50       | g.00000 |  2383964 |       49 |	      7 | 	 5.09  4.12
| MD01.reftest.20130731.c1f52       | g.00000 |  2383965 |       82 |	     13 | 	 4.83  4.26
| MD01.reftest.20130731.c1f55       | g.00000 |  2383966 |      107 |	     19 | 	 4.91  4.34

--> previous 2012 refstack					
| MD01.refstack.20120803            | y.00000 |  1112395 |       90 |        28 | 	 3.53  3.28 
| MD01.refstack.20120803            | z.00000 |  1112455 |       88 | 	     18 | 	 3.59  3.16 
| MD01.refstack.20120803            | i.00000 |  1112515 |       88 |	     15 | 	 3.58  3.31 
| MD01.refstack.20120803            | r.00000 |  1112575 |       84 |	     20 | 	 4.60  3.84 
| MD01.refstack.20120803            | g.00000 |  1112635 |       96 |	     29 | 	 5.28  4.30 

As before, the FWHM_MAJOR with the central skycell can be variable and more so with largely different input sets. Comparison with the psf_major/minor maps are more useful between refstack versions.

Problems

• as with all past refstacks, generally edge skycells have issues -- now more "bright" chunks but ratty edges filled in now.
• like others, greater % of inputs rejected (~20%) so per pixel input is less, but still of order 30 inputs per pixel minimum in core skycells
• skycell notes:
  • y: s012,013 and others with bright edges; s014 odd underflux print through; s024,025,026,036,047 poor cell bands (need to be masked in y); s066,076,077 super ghost..; s087 extra over rejection?
  • z: s012,013 again bright edge but almost really good; s021 extra over rejection?; s075 odd junk; s087 same; super ghost same
  • i: s012,013 again bright chunk edges; s016 underflux print through; s047 underflux by bright source; s071 large psf; super ghost; s087 extra over rejection?
  • r: s012,013 etc again; s015,025,026,041,051,061, ghosts ; s027 under flux bands (seen in others too); s047 underflux area and other junk/cells; s087 extra over rej? or high sky?
  • g: s013 bright chunk edges (not really s012 from over rej?); s013,015,023,024,025,026,027,034,041,042,043!! ghosts s047,061 junk; s016 under flux chunks/strips; s054 odd cell print through; s044,054 poor cell arc/scatter (wonder if could/should mask cell as seen in MD01.refstack.20120803)
• some convolved stacks have poorer FWHM PSF but not significant enough to be likely improved by reprocessing using MD_DEEP_STACK config as in earlier MD fields (input cut <10 rather than <6 and FWHM+100*STDEV rather than FWHM+1.0*STDEV)
  • g: s071g -- -- no real improvement found since edge skycell and kept original
  • g: s082 -- very few inputs (4) 30/34 rej -- edge skycell and g-band so most are cut >6, MD_DEEP_STACK config has 4/19 rej
  • i: s071 -- very few inputs (4), 27/31 rej and bad target PSF (conv image will be mostly useless) -- use MD_DEEP_STACK and 1/12 rej with slightly better target PSF (but still poor)
  • y: s016 -- quality fault with PSF problem -- reduced elong cut 1.3->1.25 and increased FWHM input cut from 3.8->3.9 pixels, still used MD_REF_STACK reduction config
  • re-processing tests (have not written this up before and should note):
    • data_group -- MD01.refstack.20130731redo
    • workdir base -- MD01.refstack.20130731redo
  • unused redo processing
    • label -- MD01.refstack.20130731skip
    • data_group -- MD01.refstack.20130731redo
    • workdir base -- MD01.refstack.20130731redo
  • replaced original run
    • label -- MD01.refstack.20130731poor
    • data_group -- MD01.refstack.20130731poor
    • workdir base -- original name MD01.refstack.20130731
  • redo processing replacing original processing
    • label -- normal refstack label
    • data_group -- normal refstack data_group
    • workdir base -- MD01.refstack.20130731redo

Core coverage -- not a shifted pointing

082-087
071-078
061-068
051-058
041-048
031-038
021-028
012-017

• skycells with fewer inputs (typical for edges): 012, 017, 021, 028, 071, 078, 082, 087
• skycells with <50% or more of typical input number is filter dependent - usually 028, 078, 082

Refstack Summary

Stack phomtometry on the refstacks done through the staticsky and skycal stages using the unconvolved images

• single filter mode so basic psphot with deteff measurements that has the zero-point recalculated for the test plots below
  • label: MD01.refstack.20130731, MD01.refstack.20130731
  • data_group (single filter): MD01.refstack.20130731.1x20130805, MD01.refstack.20130731.1x20130805.cal20130805
• multi-filter (Nx) not run yet
  • label:
  • data_group: (all skycells have 5 filters matched)
• on datastore under ps1-md-staticsky, but only the staticsky version, the skycal product still has no method for distribution

Source photometry psf_major,_minor maps (grizy): all new plots here scaled by root(2) to match past scaling (plots now labelled accordingly) and so values larger by root(2) to what should actually be

Magnitude of sources@5-sigma maps (grizy):

Detection efficiency magnitude @50% recovery (500 fakes, grizy): 0 rej for all now that the SCL_ keyword has its true value other than 1. when rejected in stacking

---

1D convolution mode PV2

NOT started (yet) since 1D convolution not working properly for MD

Same naming format as generally above, adding _1dg to the data_group (as will be seen on the datastore)

• label: MD01.reftest.20130731
• data_group:
  • MD01.reftest.20130731.c#f<FWHM cut>_1dg where #=comparison run, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
  • MD01.reftest.20130731.all.<FWHM cut>_1dg where all means 05/2010+2011+2012/2013 data possible, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
  • MD01.reftest.20130731.2012.<FWHM cut>_1dg where 2012 means 2012/2013 data only for season stack example case, <FWHM cut>=cut <fwhm_major from camProcessedExp x10>
• c1 cuts (same as above)

Full refstack:

• label: MD01.refstack.20130731_1dg
• data_group: MD01.refstack.20130731_1dg

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055) after running psphot on the stacks.

| data_group                        | filter  | stack_id | warps_in | warps_rej |  FWHM_MAJ   IQ_FW1 (pixels)
+-----------------------------------+---------+----------+----------+-----------+

---