Summary of MD09.PV1_deepstack_tests

(MediumDeepFields summary page)

Recent deep stacks test include (label/data_group)

  • MD09.deeptest.20120727: best attempt to make a 3x input stack over the refstacks with constrained PSF on the convolved stacks for staticsky stack photometry and to avoid critical failures in order to produce a full field stack in each filter. As part of the initial MD.GR0 reprocessing, details can be found at MD.GR0#MD09deepstacktest20120727, and only uses available exposures from 2009-2011 reprocessing (also PV1 class).
  • stack-o-stacks: an idea to try stacking night stacks in order to reduce the large processing time required in deepstacks. Local tests were run -- while the code seems to make them and in significantly less time, it is unclear how good the result is and does break some of the extra output products such as the .num.fits and .exp.fits without a good amount of development work.
  • MD09.deeptest.20130330: mostly failed attempt at increasing the input number from 20120727 due to bugs in ppStack
  • MD09.alldeeptest.20130507: a sub-field set of skycells with everything going in, only failed PSF matching rejected. (using improved ppStack version, see stacking_coverage.20130307)
  • MD09.cutdeeptest.20130508: a sub-field set of skycells using exposures from 2010-2012 with modest FWHM, background, ZPT cuts (see section below)
  • MD09.deeptest.20130509: full field set of all 2009-2012 exposures with modest FWHM, background, ZPT cuts.

Exposures from 2012 by default are PV1, 2009-2011 are GR0 reprocessing that are also PV1 class.

MD09.alldeeptest.20130507

Include all MD09 data from 2009-04 through end of last season Dec. 2012.

Test skycells:

  • 027 (past problem edge mix of old 2009 and newer dither data, 1 galaxy from Thilker list)
  • 045 (2 galaxies)
  • 055 (common test skycell, center with PSF issues, 1 galaxy)
  • 068 (another near edge, 2 galaxies)

N input and rejected warps:

| stack_id | label                     | skycell_id  | filter  | fault | quality | reject_images | dtime_script | count(warp_id) |
+----------+---------------------------+-------------+---------+-------+---------+---------------+--------------+----------------+
|  2237787 | MD09.alldeeptest.20130507 | skycell.027 | g.00000 |     0 |       0 |            35 |        44117 |            577 | 
|  2237788 | MD09.alldeeptest.20130507 | skycell.027 | i.00000 |     0 |       0 |            36 |        44957 |            583 | 
|  2237789 | MD09.alldeeptest.20130507 | skycell.027 | r.00000 |     0 |       0 |           259 |        40616 |            567 | 
|  2237790 | MD09.alldeeptest.20130507 | skycell.027 | y.00000 |     0 |       0 |           388 |        33102 |            562 | 
|  2237791 | MD09.alldeeptest.20130507 | skycell.027 | z.00000 |     0 |       0 |           317 |        51544 |            654 | 
|  2237792 | MD09.alldeeptest.20130507 | skycell.045 | g.00000 |     0 |       0 |            22 |        82983 |            578 | 
|  2237793 | MD09.alldeeptest.20130507 | skycell.045 | i.00000 |     0 |       0 |            38 |        83354 |            584 | 
|  2237794 | MD09.alldeeptest.20130507 | skycell.045 | r.00000 |     0 |       0 |            24 |        75814 |            567 | 
|  2237795 | MD09.alldeeptest.20130507 | skycell.045 | y.00000 |     0 |       0 |           249 |        66896 |            568 | 
|  2237796 | MD09.alldeeptest.20130507 | skycell.045 | z.00000 |  NULL |    NULL |          NULL |         NULL |            657 | 
|  2237797 | MD09.alldeeptest.20130507 | skycell.055 | g.00000 |     0 |       0 |            14 |        78568 |            578 | 
|  2237798 | MD09.alldeeptest.20130507 | skycell.055 | i.00000 |     0 |       0 |            33 |        83524 |            584 | 
|  2237799 | MD09.alldeeptest.20130507 | skycell.055 | r.00000 |     0 |       0 |            17 |        76823 |            567 | 
|  2237800 | MD09.alldeeptest.20130507 | skycell.055 | y.00000 |     0 |       0 |            96 |        82326 |            566 | 
|  2237801 | MD09.alldeeptest.20130507 | skycell.055 | z.00000 |  NULL |    NULL |          NULL |         NULL |            657 | 
|  2237802 | MD09.alldeeptest.20130507 | skycell.068 | g.00000 |     0 |       0 |            67 |        18716 |            338 | 
|  2237803 | MD09.alldeeptest.20130507 | skycell.068 | i.00000 |     0 |       0 |            54 |        20887 |            328 | 
|  2237804 | MD09.alldeeptest.20130507 | skycell.068 | r.00000 |     0 |       0 |            84 |        20893 |            327 | 
|  2237805 | MD09.alldeeptest.20130507 | skycell.068 | y.00000 |     0 |       0 |           288 |         6864 |            301 | 
|  2237806 | MD09.alldeeptest.20130507 | skycell.068 | z.00000 |     0 |       0 |           115 |        19176 |            353 | 
  • label=data_group on datastore and postage stamp server. If dtime_script NULL, then the stack is still running.
  • stack pixels to the datastore under ps1-deeptest
  • even though all-in stacks, even with recent improvements there are still cases when large % are rejected by chi2 matching failures.. at times making the inputs fewer than with more strict cuts..

MD09.cutdeeptest.20130508

Include all MD09 data from April 2009 through end of last season Dec. 2012, but impose exposure level cuts as suggested by Ken Chambers:

  • FWHM<=10 pixels (2.5") is ~9.77 pixel in exposure stats (table numbers using iq_fwhm_major)
  • elong<=1.5
  • dZPT<=0.33 mag, so using mean ZPT grizy >= 24.09, 24.32, 24.23, 23.85, 22.89
  • background grizy <= 600, 800, 2500, 4000, 6000
  • drop 2009 data as may have a variety of issues
  • using 2010+cuts in table below for cutdeeptest.20130508. 2010+ is all data 2010-2012, cuts is all data 2009-2012 with the input cuts applied (numbers for edge skycells can be greatly less).
filter all 2010+ cuts 2010+cuts comment
g 584 545 500 464 lose ~120
r 571 530 498 457 lose ~110
i 599 525 526 462 lose ~140, driven by what appears evenly by all cuts..
z 660 599 598 539 lose ~120 ok
y 575 552 421 401 lose ~170, seem a problem mainly with many of poorer ZPT from marginal weather observations

Test skycells (same set):

  • 027 (past problem edge mix of old 2009 and newer dither data, 1 galaxy)
  • 045 (2 galaxies)
  • 055 (common test skycell, center with PSF issues, 1 galaxy)
  • 068 (another near edge, 2 galaxies)
| stack_id | label                     | skycell_id  | filter  | fault | quality | reject_images | dtime_script | count(warp_id) |
+----------+---------------------------+-------------+---------+-------+---------+---------------+--------------+----------------+
|  2237807 | MD09.cutdeeptest.20130508 | skycell.027 | g.00000 |     0 |       0 |            71 |        26885 |            472 | 
|  2237808 | MD09.cutdeeptest.20130508 | skycell.027 | r.00000 |     0 |       0 |           132 |        35485 |            462 | 
|  2237809 | MD09.cutdeeptest.20130508 | skycell.027 | i.00000 |     0 |       0 |            43 |        31509 |            470 | 
|  2237810 | MD09.cutdeeptest.20130508 | skycell.027 | z.00000 |     0 |       0 |            56 |        34626 |            531 | 
|  2237811 | MD09.cutdeeptest.20130508 | skycell.027 | y.00000 |     0 |       0 |           124 |        22651 |            405 | 

|  2237812 | MD09.cutdeeptest.20130508 | skycell.045 | g.00000 |     0 |       0 |            45 |        55381 |            473 | 
|  2237813 | MD09.cutdeeptest.20130508 | skycell.045 | r.00000 |     0 |       0 |            35 |        63818 |            462 | 
|  2237814 | MD09.cutdeeptest.20130508 | skycell.045 | i.00000 |     0 |       0 |            30 |        61355 |            471 | 
|  2237815 | MD09.cutdeeptest.20130508 | skycell.045 | z.00000 |     0 |       0 |            39 |        73388 |            532 | 
|  2237816 | MD09.cutdeeptest.20130508 | skycell.045 | y.00000 |     0 |       0 |            63 |       169391 |            405 | 

|  2237817 | MD09.cutdeeptest.20130508 | skycell.055 | g.00000 |     0 |       0 |            37 |        54054 |            473 | 
|  2237818 | MD09.cutdeeptest.20130508 | skycell.055 | r.00000 |     0 |       0 |            44 |        65216 |            462 | 
|  2237819 | MD09.cutdeeptest.20130508 | skycell.055 | i.00000 |     0 |       0 |            22 |        63907 |            471 | 
|  2237820 | MD09.cutdeeptest.20130508 | skycell.055 | z.00000 |     0 |       0 |            38 |        72188 |            532 | 
|  2237821 | MD09.cutdeeptest.20130508 | skycell.055 | y.00000 |     0 |       0 |            44 |        50172 |            405 | 

|  2237822 | MD09.cutdeeptest.20130508 | skycell.068 | g.00000 |     0 |       0 |            50 |        11495 |            262 | 
|  2237823 | MD09.cutdeeptest.20130508 | skycell.068 | r.00000 |     0 |       0 |            50 |        10491 |            247 | 
|  2237824 | MD09.cutdeeptest.20130508 | skycell.068 | i.00000 |     0 |       0 |            47 |         9472 |            234 | 
|  2237825 | MD09.cutdeeptest.20130508 | skycell.068 | z.00000 |     0 |       0 |            74 |         9465 |            261 | 
|  2237826 | MD09.cutdeeptest.20130508 | skycell.068 | y.00000 |     0 |       0 |           121 |         5533 |            205 | 
  • label=data_group on datastore and postage stamp server
  • stack pixels to the datastore under ps1-deeptest

MD09.deeptest.20130509

Setup with same cuts as above in MD09.cutdeeptest.20130508, but includes 2009-2012 (cuts in table).

-- status -- finished as of 5/19 (fewer faults so ran much faster)

| label                  | filter  | avg(reject_images) | avg(dtime_script) | count(stack_id) |
+------------------------+---------+--------------------+-------------------+-----------------+
| MD09.deeptest.20130509 | g.00000 |            54.9861 |   45340.069444233 |              72 | /73 -- 1 permanent fault, done, out to datastore 5/13
| MD09.deeptest.20130509 | r.00000 |            39.4028 |   42004.361111111 |              73 | /73 --  done, out to datastore 5/15
| MD09.deeptest.20130509 | i.00000 |            35.7067 |   43037.719999695 |              75 | /75 -- done, out to datastore 5/16 (fewer faults so much faster...)
| MD09.deeptest.20130509 | z.00000 |            51.1081 |   48250.175676088 |              74 | /74 -- done, out to datastore 5/20
| MD09.deeptest.20130509 | y.00000 |            70.8714 |   34822.771428571 |              70 | /70 - done, out to datastore 5/20

Quality faults:

| label                  | filter  | skycell_id  | avg(reject_images) | avg(dtime_script) | quality | count(warp_id) |
+------------------------+---------+-------------+--------------------+-------------------+---------+----------------+
| MD09.deeptest.20130509 | i.00000 | skycell.016 |             0.0000 |             16902 |   13008 |            487 | 
| MD09.deeptest.20130509 | z.00000 | skycell.012 |             0.0000 |             17640 |   13008 |            513 | 

N input and rejected warps (of similar skycell sub-samples to cases above):

| stack_id | label                  | skycell_id  | filter  | fault | quality | reject_images | dtime_script | count(warp_id) |
+----------+------------------------+-------------+---------+-------+---------+---------------+--------------+----------------+
|  2239309 | MD09.deeptest.20130509 | skycell.027 | g.00000 |     0 |       0 |            72 |        30308 |            502 | 
|  2239951 | MD09.deeptest.20130509 | skycell.027 | r.00000 |     0 |       0 |           110 |        39792 |            486 | 
|  2240258 | MD09.deeptest.20130509 | skycell.027 | i.00000 |     0 |       0 |            53 |        37778 |            516 | 
|  2240796 | MD09.deeptest.20130509 | skycell.027 | z.00000 |     0 |       0 |            58 |        40298 |            571 | 
|  2241436 | MD09.deeptest.20130509 | skycell.027 | y.00000 |     0 |       0 |           128 |        23774 |            421 | 

|  2239327 | MD09.deeptest.20130509 | skycell.045 | g.00000 |     0 |       0 |            49 |        59529 |            503 | 
|  2239969 | MD09.deeptest.20130509 | skycell.045 | r.00000 |     0 |       0 |            42 |        64725 |            486 | 
|  2240276 | MD09.deeptest.20130509 | skycell.045 | i.00000 |     0 |       0 |            34 |        69330 |            517 | 
|  2240813 | MD09.deeptest.20130509 | skycell.045 | z.00000 |     0 |       0 |            34 |        78785 |            572 | 
|  2241453 | MD09.deeptest.20130509 | skycell.045 | y.00000 |     0 |       0 |            65 |        54503 |            421 | 

|  2239337 | MD09.deeptest.20130509 | skycell.055 | g.00000 |     0 |       0 |            57 |        58442 |            503 | 
|  2239979 | MD09.deeptest.20130509 | skycell.055 | r.00000 |     0 |       0 |            41 |        63812 |            486 | 
|  2240286 | MD09.deeptest.20130509 | skycell.055 | i.00000 |     0 |       0 |            33 |        68963 |            517 | 
|  2240823 | MD09.deeptest.20130509 | skycell.055 | z.00000 |     0 |       0 |            40 |        78463 |            572 | 
|  2241463 | MD09.deeptest.20130509 | skycell.055 | y.00000 |     0 |       0 |            43 |        53591 |            421 | 

|  2239350 | MD09.deeptest.20130509 | skycell.068 | g.00000 |     0 |       0 |            76 |        13755 |            292 | 
|  2239992 | MD09.deeptest.20130509 | skycell.068 | r.00000 |     0 |       0 |            54 |        13431 |            271 | 
|  2240299 | MD09.deeptest.20130509 | skycell.068 | i.00000 |     0 |       0 |            44 |        15991 |            280 | 
|  2240836 | MD09.deeptest.20130509 | skycell.068 | z.00000 |     0 |       0 |            77 |        13871 |            301 | 
|  2241476 | MD09.deeptest.20130509 | skycell.068 | y.00000 |     0 |       0 |           114 |         7682 |            221 | 

Even with cuts on the initial input exposure stats, there appear to be significant additional cuts w/in ppStack for some. A break-down of some of the rejections are as follows:

  • skycell.027-r -- mix of >10 pixel for the warps, bad PSF info for warps, NAN for chi2, but also unclear why ~1/3 rejected are from failed PSF match
  • skycell.027-y -- mix of >10 pixel for the warps, bad PSF info for warps, but also unclear why ~1/3 rejected are from failed PSF match
  • skycell.045-y -- mostly >10 pixel for the warps
  • skycell.055-g -- mostly >10 pixel for the warps
  • skycell.068-g -- mostly >10 pixel for the warps, some bad PSF info for warps and NAN for chi2
  • skycell.068-y -- mix of >10 pixel for the warps, bad PSF info for warps, NAN for chi2