Summary for MD_stack_types (updates in progress)

The start of a summary/requirements document for the different types of stacks for the Medium Deep Fields (MD). Other summaries from past data processing (and folded in here) have included

Back to MediumDeepFields

As of 2/2014, planned stack processing for PV3 diffims are as follows:

  • nightly stack -- will the new configuration be used below? Will these exclude 2009+early 2010 exposures for better quality?
  • deep stack -- undetermined what parameters this has. Will these exclude 2009+early 2010 exposures for better quality?
  • best IQ deep stack -- undetermined what parameters in terms of FWHM and depth this has. Will these exclude 2009+early 2010 exposures for better quality?
  • out-of-season refstack -- similar to what has been done previously, except now have after-season exposures to draw from. NOTE this means any long duration events will have difference reference/template stacks in the diffims. Will these exclude 2009+early 2010 exposures for better quality?

Night Stacks (nightstack): actively made -- all warps from a night (re)processed and stacked into a night stack as part of the nightlyscience tasks

  • GR0 reprocessed warps into night stacks through the 2012 season can be a mix of synthetic/ubercal catalog calibrations for the input warp pixels. However, the warps going into the stack should be of the same type of calibration.
  • PV1 sub-reprocessing of synthetic catalog based night stacks to be ubercal catalog based calibation of the input warp pixels. There is no simple, unique method for selecting other than manually creating a release listing or using the gpc1 DB stackInputSkyfile table for input warp_id.

Current configuration for nightly science (GR0/PV1 had Nin>=2 warps but with SAFE stacking true, pixels only kept if both were good):

  • Nin>=4
  • SAFE stacking
  • ISIS convolution
  • all exposures used in stack (even on marginal nights)

Proposed configuation (align with PV2)-- setting up test with MD10 stacks MD10.refstack.20130715#Test1DNightStacks

  • Nin>=2
  • SAFE stacking false
  • 1D convolution -- excess rejections present so not possible until fixed (should be okay now as seen with MD02.refstack.20130815)
  • selection cuts for marginal exposures?

Reference Stacks (refstack): actively made each season -- primarily as template/reference for difference imaging and transient detections, was historically necessary for magic destreaking. The best exposure reprocessing prior to the start of the MD field observations for the season to support the difference image products. The focus of MD.GR0 was on the 2011/2012 season fields starting with MD10, then 01, 02 ... through MD09. The 2012/2013 season refstacks replace the 2011/2012 set and have their own page linked from MediumDeepFields. The 2013 season will start using PV2.

  • Magic destreaking: requires a fairly clean reference stack (uses the convolved stack) with as much spatial area coverage as possible to enable release of non-stack data products. Particularly important is inclusion of older data on different pointings and of course a fairly balanced dither set.
  • Transient detection: a balance slanted towards best FWHM but with some depth (~1 magnitude over night stacks) and also a fairly balanced dither set. Better with more recent data to limit AGN/variable star variations.
  • how much of an IQ variation can go into a stack before ppSub has difficulty modeling the variations within a skycell?
  • for cases when a stack does not have a good sample of exposures with a reasonably small PSF FWHM, it should be possible to generate a reference stack with the fewer 'best' exposures for just SSdiffs and transient detection. MD02-g may be such a case, but could leave large regions of order a night stack number of inputs.

See summary for each MD field reference stack for details on specifics for that field. Due to processing+time constraints/early mistakes, an attempt has been made to merge the similar criteria under the refstack label -- full field coverage, best FWHM, 1-1.5 mag and >10ks depth over central FPA as possible

  • based on the steep edge on the FWHM distribution of input warps, a large number of inputs could be used without degrading the final FWHM by much (~0.2 pixels, ~0.05") even upwards of 150 input warps
  • of order 100 inputs found necessary to keep a minimal depth per pixel in many central regions with the dither and masking (see MD04 deep stack tests).

Out-of-season and yearly refstack: on hold for next reprocessing round (would need to update/reprocess large numbers of exposures to do) -- will try a test with MD04.deepstacks

  • these will be the end of survey refstacks -- need to decide if out-of-season will include post-season data as SN can be long-tailed, or if both type will be done

PV1 stack_id>=923376 and/or label MD##.refstack should be valid, data_group of form MD##.refstack.YYYYMMDD

Deep Stacks (deepstack): not a full production product yet -- for deep, static sky science

  • As deep as possible. A maximum FWHM allowance? A minimum exposure time?
  • A specific request for at least ~10ks has been made, with best FWHM possible.
  • A fairly balanced dither set is almost by default with a large number of inputs. Issue really comes up in fields with shifted positions from older observations.
  • Test version exists for MD04.deeptest.20120727 (synthetic catalog, pre-PV1 warps only), MD09.deeptest.20120727 (ubercal catalog 20120524v0, PV1 warps only from pre-2012 season).
  • Yearly and/or out-of-season refstacks, would required much update processing to make for other fields. Looking at updating MD04 deep stack and adding MD07 to the test set however.

Best IQ/PSF Stack (iqstack): currently on hold -- the best seeing/FWHM images possible of order night stack depth? Could be one of two flavors:

  • only the best FWHM warps <X pixels used, so not a fully covered FPA
  • best FWHM skycell-by-skycell, meeting some Ninput and dither filling factor criteria (would probably be best)


  • should be made with ref/deep stacks to avoid excess processing/updating.
  • test version exists for MD02.iqtest.20120927, MD10.psfrefstack.20110814, but both are with pre-PV1 warps. A sample could be done for MD09 done but requires development/code/verification time.

Test Stacks (all sorts): if confidence in a stack is not to a production level, it is made with test in the name (i.e., teststack, deeptest, reftest, etc).

  • reftest: often made before refstacks as a preparation test for FWHM and other input cuts
  • deeptest: made for deep stacks until the content is finalized
  • teststack: general stack of various forms, and example is MD10+3PI exposures into a stack

Additional considerations:

  • since magic destreaking uses the convolved stack, a quickstack version of the unconvolved stacks for the transients reference and deep stack could be faster to make with a larger number of inputs but requires additional development for regular processing.
  • fields with shifted centers with offsets to include early exposures, the inclusion of enough exposures to provide a reference stack for magic is necessary in order for data from those associated chips/skycells to be distributed (being older data, may not have the optimum quality). Mixing older and newer data may also have other problems.
  • 2009 data may be suspect and is often excluded for general refstacks and possibly deepstacks
  • stack-of-stacks idea possible but would require additional development work for all output products to support properly
  • using 3PI data to fill in the MD holes and edges MD10.refstack.20130715