Summary for MD.PV1

  • MediumDeepFields summary page -- quick links to all up to date refstacks
  • MD.GR0 summary page -- summary of all reprocessing details and 2011/2012 refstacks and some deepstack samples

PV1 processing and (re)reprocessing is defined for MD based on the start date of 2012-05-16 to include improved and new detrend types (GPC1_Detrend_Documentation) and corrections for the faint-end bias photometry (ipp-20120404,r33642 tag -- details related to psphot elsewhere..). While detrends have been updated since that date, the changes are minor and should not impact the MD (re)processing. Also, even though the latest date for the new VideoMask? detrend is 2012-05-03, the date of 2012-05-16 was chosen as the day after the summary of the detrend updates had been established.

Many of the details here are also taken from the MD.GR0 page, see that page for more information. In general, all the MD fields required at least partial re-reprocessing of chip to warp except MD09 for which MD.GR0 reprocessing started at the beginning of June 2012 (including the refstack and night stacks). All MD fields now have refstacks created after the 2012-05-16 date. Only MD09 has a complete set of PV1 night stacks, all other fields will have a mixture of pre and post-PV1 night stacks (post-PV1 will generally be the normal nightly science processing of night stacks and not from MD.GR0).

Reprocessed Data Products

Each MD field has a variety of issues and details are summarized in each MD section of MD.GR0 or later in the relevant refstack wiki page (see links from MediumDeepFields). Label/data_group names have also slowly simplified over time and the appropriate MD section should be referred to if those must be be used. It is recommended to add the use of the <stage>_id to select appropriate data products and each section below will provide the number after which the PV1 definition should be valid for.

It is important to make clear that the x.PV1.x label/data_group/workdir for MD was a sub-re-reprocessing of exposures that were reprocessed or just processed before date X (now set at 5/16/2012). There have been various other re-reprocessing and reprocessing of exposures (for recent refstacks, GR0 late 2012 night stack completion, etc ***) that have happened after that date. This means there can also be duplicate processing of exposures with valid <stage>_id. Duplicates were allowed because initially the exact date for PV1 was unclear and can also provide an old/new valid reprocessed exposure comparison (i.e., has a change in the ipp-X tag caused problems). When PV1 re-reprocessing was done, MD fields that ended their observing season around this time (i.e., MD05,06,07) were reprocessed to finish out their observing season for some fields since the date was initially uncertain for PV1. Again, either should be acceptable to use, and a comparison of original and newer would probably be a helpful.

*** also be aware of -- reprocessing of exposures for old format V2 exist, those were ignored and re-reprocessed for V3 warps and stacks later on. If interested in the warp and later products, verification that the data product is of the V3 format should be done.

Chip, Warp

Fully reprocessed but not distributed due to disk space load. These can be obtained via the Postage Stamp Server

  • GR0 was a mix of synthetic/ubercal catalog calibrations except MD08,09
  • PV1 and GR0 with chip_id, warp_id >= 457521, 416815 respectively, all processing using the 20120524v0 ubercal catalog, valid detrends and ipp-tag

Camera/SMFs

Fully reprocessed and distributed

  • GR0 was a mix of synthetic/ubercal catalog calibrations except MD08,09
  • PV1 and GR0 with cam_id>=434615 respectively, all processing using the 20120524v0 ubercal catalog and valid detrends

Stacks

Each type of stack has its own sub-section below -- generally, if a stack_id>=923376 then it was likely made from exposures processed with PV1 requirements using the 20120524v0 ubercal catalog, detrends and ipp-tag (except some more recent night stacks were made using complete sets of old warps such as in the initially incomplete sets of MD01,02,03,05).

Night Stacks (nightstack): All warps from a night (re)processed and stacked into a night stack

  • the GR0 sample of stacks is complete but can be a mix of synthetic/ubercal catalog calibrations for the warp pixels. The warps going into the stack should be of the same type of calibration.
  • for PV1 there is no simple, unique method for selecting other than manually creating a release listing or using the gpc1 DB stackInputSkyfile table for input warp_id

Reference Stacks (refstack): best exposure reprocessing prior to the start of the MD field observations for the season to support the difference image products. The focus of MD.GR0 was on the 2011/2012 season fields starting with MD10, then 01, 02 ... through MD09. The 2012/2013 season refstacks replace the 2011/2012 set and have their own page linked from MediumDeepFields.

  • out-of-season/anti-yearly refstacks, yearly refstack: on hold for next reprocessing round (would need to update/reprocess large numbers of exposures to do)
  • PV1 stack_id>=923376 and/or label MD##.refstack should be valid, data_group of form MD##.refstack.YYYYMMDD

Deep Stacks (deeptest): Not a full production product yet. Test version exists for MD04.deeptest.20120727 (synthetic catalog, pre-PV1 warps only), MD09.deeptest.20120727 (ubercal catalog 20120524v0, PV1 warps only from pre-2012 season). Like yearly or out-of-season stacks, would required much update processing to make for other fields. Looking at updating MD04 deep stack and adding MD07 to the test set however.

best IQ/PSF Stack (none): on hold until next reprocessing round (should be done with refstacks to avoid excess processing). Test version exists for MD02.iqtest.20120927, MD10.psfrefstack.20110814, but neither are made from PV1 warps. A sample could be done for MD09 done but requires development/code time.

Difference Images

Originally a possible reprocessing product, but was decided not useful at this time (for future reprocessing will need to clearly outline interests for these, if any). Open items to address still -- finish the convolution choice (input stack or reference stack) and improvements to convolution.

  • diff_id>=240039 will be valid PV1 from nightly science processing

Stack Photometry (static sky + skycal)

Stack photometry (staticsky) and recalibration (skycal) is run on refstacks, deepstacks and night stacks (currently on a partial set of MD09.GR0 only). These often get reprocessed uniformly on new stacks or when/if bugs found and fix, latest one was January 2013, and so label/data_group is more direct for identification.

  • recalibration of the stack photometry catalogs has no method for distribution -- email for access

Re-revised planned loading into PSPS (March 2013) -- do for all the newer refstacks, MD09 deeptest, MD09.GR0 night stacks

  • ref stack label, data_groups: includes 5-1 filter sets now under same label, staticksky data_group will indicate <N> for # filters attempted, skycal data_group will include all under one data_group
    label staticsky:  MD09.refstack.20120831.Nx20130115 -- run on refstack data_group MD09.refstack.20120831, label for active refstacks is changed to MD<##>.refstack generically
    data_group staticsky:  MD09.refstack.20120831.<N>x20130115 -- <N> = 1,2,3,4,5 and number of filters available for the set, so 5 different data_group per MD
    label=data_group skycal:  MD09.refstack.20120831.Nx20130115.cal20130128 -- all N filter set is run under one data_group now per MD
    
    label staticsky:  MD01.refstack.20120803.Nx20130115 -- skycal label=data_group, add .cal20130128 
    label staticsky:  MD02.refstack.20120927.Nx20130115 -- skycal label=data_group, add .cal20130128
    label staticsky:  MD03.refstack.20121101.Nx20130325 -- skycal label=data_group, add .cal20130326
    label staticsky:  MD04.refstack.20121125.Nx20130325 -- skycal label=data_group, add .cal20130326
    label staticsky:  MD05.refstack.20121202.Nx20130325 -- skycal label=data_group, add .cal20130326
    label staticsky:  MD06.refstack.20121221.Nx20130325 -- skycal label=data_group, add .cal20130326
    label staticsky:  MD07.refstack.20130102.Nx20130325 -- skycal label=data_group, add .cal20130326
    label staticsky:  MD08.refstack.20130401.Nx20130407 -- skycal label=data_group, add .cal20130408
    label staticsky:  MD09.refstack.20120831.Nx20130115 -- skycal label=data_group, add .cal20130128
    
  • deep stack label, data_groups: includes 5-1 filter sets now under same label, staticsky data_group will indicate N-filter, skycal data_group all under one name -- only for MD09 at this time
    label=data_group skycal:  MD09.deeptest.20120727.Nx20130114.cal20130128 -- all N filter set is run under one data_group now per MD
    
  • night stack label, data_group:
    staticsky label = data_group:  MD09.GR0.20120602.nightstack.1x20120926rawtest
    skycal label = data_group:  MD09.GR0.20120602.nightstack.1x20120926rawtest.cal20130128 
    

PV1 Summary Table by MD Field

(revised Apr. 21, 2013)

For all fields, the <stage>_id cut necessary for a processing to be PV1

  • chip_id>=457521
  • cam_id>=434615
  • warp_id>=416815
  • diff_id>=240039 (not really reprocessed, so most are just recent nightly processing data products)
  • stack_id>=923376 (not uniquely PV1 for the night stacks and warp_id from stackInputSkyfile should be checked)
MD Reference Stack Deep Stack Night Stack Refstack Phot Deep Phot Night Phot for PSPS
01 20120803 x (mixed) .Nx20130115.cal20130128 x x SMF+refstack phot
02 20120927 x (mixed) .Nx20130115.cal20130128 x x SMF+refstack phot
03 20121101 x (mixed) .Nx20130325.cal20130326 x x SMF+refstack phot
04 20121125 deepstack.YYMMDD TBD (mixed) .Nx20130325.cal20130326 TBD x SMF+ref+deep phot (secondary)
05 20121202 x (mixed) .Nx20130325.cal20130326 x x SMF+refstack phot
06 20121221 x (mixed) .Nx20130325.cal20130326 x x SMF+refstack phot
07 20130102 deepstack.YYMMDD TBD (mixed) .Nx20130325.cal20130326 TBD x SMF+ref+deep phot (secondary)
08 20130401 x (mixed) .Nx20130408.cal20130408 x x SMF+refstack phot
09 20120831 deeptest.20120727 (TBRD) (PV1) .Nx20130115.cal20130128 .Nx20130114.cal20130128 .1x20120926rawtest.cal20130128 SMF+ref+deep+night phot (primary)
10 20120821 x (mixed) .Nx20130115.cal20130128 x x SMF+refstack phot
11 x x x x x x x

Notes:

  • TBD=To Be Decided/Done by priority, X=not planned, ?=unknown at this time, TBRD=To Be ReDone?
  • 20YYMMDD date entries are part of the data_group name to access via the PSS or Datastore (note some may also have MD<#>.refstack.<date>redo for replaced skycells, see specific MD field for details)
    • generally for stacks -- i.e., refstack for MD01 refstack.20120803, deepstack for MD09 deeptest.20120727
    • refstacks -- as these are the active refstacks, they will have label MD<#>.refstack
    • night stacks -- a mix of label, data_group formats and cannot be grouped simply. See specific GR0 section, but trying to move to data_group format MD<#>.GR<N>.<process date>.nightstack.<obs date> for reprocessed. Normal nightly processing is of form MD<#>.<date obs>. Stack photometry was only run on reprocessed night stacks, so stacks from 2012 season will not be included.
    • stack photometry -- originally the same date, but revised runs now have dates for each in the data_group form (see section above) of adding to MD<#>.<stack type>.<stack date> for staticsky .<#>x20YYMMMDD and full skycal recalibration of the staticsky photometry then adding .cal20YYMMDD. So the skycal data_group will be something of the form like MD09.refstack.20120831.Nx20130115.cal20130128 in the Refstack Phot column.
  • (mixed) for Night Stacks -- most fields have a mix of night stack based on input warp types (not a mix going into a particular night stack) of synthetic catalog and ubercal based catalog for both the GR0 and/or the following nightly science processing night stacks. Only MD09 have a PV1 full set of night stacks currently.
  • Deep stack exist as PV1 only for MD09 from MD.GR0 only, MD04 needs to be redone and MD07 added (TBD) -- likely all 3 will be made with same processing tag and criteria in near future and MD09 to be replaced.
  • MD11 not part of the MD.GR0, PV1 reprocessing
  • if you find any details missing or incorrect, please let us know.

Questions:

  • SMF loaded into PSPS limited to just the ones used in the refstack (or deepstack if available) or all available exposures Camera/SMFs?