Summary of MD.GR0

Summary page of information for the MD GR0 (grand reprocessing version 0) including the new reference stack in the V3 tessellation with the start of new observations of MD10 in OC142. This is version 0 as the test start of full reprocessing of the MD fields to be balanced along with other reprocessing, 3PI stacks and nightly science, generation of new reference stacks and as of yet does not include a revised reference calibration catalog (currently for MD10, 01, 02).

Creation of the new reference stacks need to be at the very least made prior to the return to the particular MD field given in the current schedule ​http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Medium_Deep_Schedule. After the July, August, September time crunch of a new field a month there should be time to continue reprocessing and creation of new reference stacks in advance.

• 8/15/11 LAP has priority, current mode of only minimum processing for reference stacks

• Feedback is important: We would not want to continue full reprocessing of the MD fields if there has been a gross oversight or problem for a particular science interest. Please let us know of any issues you discover with example details.

• tessellation and/or offsets issues in any of the fields
• any issues with masking, mask values - been report of possible ones in early MD10 but need details
• depth/quality of the stacks, reprocessed images in the new format
• any missing skycells, SMFs, other data products
• MD10.refstack.20110722 replacement with proper uncompressed MD10.refstack.20110821 for diffims

Reference versus Deep Stack

A requirements description for the needs in the different types of stacks would be useful to summarize.

Reference stacks -- primarily for difference imaging for transient detection and necessary for magic destreaking.

• Magic destreaking: a fairly clean reference stack (uses convolved version) with as much area coverage as possible to enable release of not-stack data products. Particularly important is inclusion of older data on different pointings and of course a fairly balanced dither set.
• Transient detection: a balance slanted towards best PSF but with some depth (1-1.5 magnitudes over nightly stacks).

Deep stacks -- for deep static sky science.

• As deep as possible.. maximum FWHM allowance? minimum exposure time? at least ~10ks has been requested, with best FWHM possible initially.

See summary for each MD field reference stack for details on specifics for that field. Due to processing+time constraints/early mistakes, an attempted has been made to merge the similar criteria.

MD.V3 Tessellation

The new V3 tessellations are defined at PS1_Tessellations with the following general properties:

• 0.25"/pix (from natural ~0.258"/pixel, previously warped to 0.2"/pix up through V2)
• 10 x 10 skycell grid (previously 12 x 12)
• +x is West (flipped from before +x was East), +y is still North
• 320 pixel (80") overlap (from 300 pixels, 60", in V2)
• revised centers (if necessary) -- see relevant MD section for more details and the MDS field table (​http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Medium_Deep_Survey#Actual_Field_Centeres_-_2011-02-21) but in summary
  • MD10.GR0: from (351.979 -0.433) to (352.312, -0.433) however an additional offset looks like it should have been applied to cover more of the earlier observation dithers.
  • MD07.GR0: from (213.146, +53.417) to (213.454, +53.243) to best cover as much of the earlier observation dithers as possible.
  • MD08.GR0: very slight offset in RA to the W to just align with the MDS field pointing.
  • MD09.GR0: from (333.688, +0.283) closer to the current pointing of (334.188, +0.283) however an addition offset was added to move the center about half a skycell to the E to (333.980, +0.283) in order to cover as much of the earlier observation dithers as possible.
• All fields should be looked at to see if there are any problems with the new tessellation and dither pattern.
• MD07, MD09 will have a preliminary nightly stack made for testing the revised center as well as MD06 for NGC4258.

---

MD10.GR0

Started for OC142 (July 15, 2011). Start with current statuswith a new reference stack and full sample of reprocessed exposures as an initial reprocessing version.

• z,y-band data processed prior to full Dark model improvements (higher background makes effect insignificant). gri-band processed after Dark model improvements.
• same photometry reference catalog as MD have previously been using.
• stack-stack diffims still using forced convolution of reference stack to input images

MD10.V3 center shifted to the current observation pointing even though there are earlier data that has offset dithered observations that populate the 0X8-0X9 skycells. Not as bad as MD07, but the revised reference stack may not provide for the outer skycells very well. Afterwards it was discovered the full extent of the previous observations wasn't quite accounted for, shown by these exposure plots with the MD10.V3 overlay and compared to a recent nightly stack set of exposures. This may need to be shifted in the next iteration.

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter	N total	N (FWHM<upper limit GPC1 pixels)
y	136	all
z	246	136 (<5.0 pixels), then finished reprocessing all
i	256	149 (<7.0 pixels), then finished reprocessing all
r	253	159 (<7.0 pixels), then finished reprocessing all
g	242	118 (<7.0 pixels), then finished reprocessing all

• also initially attempted to include some rough cuts by zeropoint and background (in 20110722) but after reprocessing all exposures wouldn't be included in the stacktool selection based solely on basically FWHM major. (improvements to selection and rejection for a future test)

MD10.V3 reference stack

Goal to maximize depth while minimizing the PSF FWHM with enough inputs (4 or more) to produce a stack for all skycells covered.

Compared to the cumulative metrics at ​http://ipp0022.ifa.hawaii.edu/ps1sc/metrics/range_2010-05-13_current/

In tracking down an odd skycell (048, y-band) highlighted an odd rejection case and prompted a modification to the criteria based on the RMS (see r32170). This may help reduce other cases of large number of rejections but likely needs some tuning (currently too restrictive?). Using the PSF FWHM cuts in the DEEP_STACK reduction also helps reject strange input warps as well.

Initially the stack processing was ran without a reduction type and ran sample of best PSF and test deep stack (version 20110722gri, 20110715y, 20110718z) along with a test sample of a best PSF focused set MD10.psfrefstack.20110814) and a test best deep stack (just skycell.055, i-band, not distributed). After discovery of this error, re-ran as DEEP_STACK reduction type (version 20110821) and should be the one used. (need to change in SSdiffs soon)

Reduction type description: NULL (none/default values used from recipes/ppStack.config unless defined in gpc1/ppStack.config):

• PSF.INPUT.MAX F32 NAN
• PSF.INPUT.CLIP.NSIGMA F32 NAN
• OUTPUT.NOCOMP BOOL FALSE
• OUTPUT.LOGFLUX BOOL FALSE
• OUTPUT.REPLICATE BOOL TRUE

DEEP_STACK:

• OUTPUT.NOCOMP BOOL TRUE
• OUTPUT.LOGFLUX BOOL FALSE
• PSF.INPUT.MAX F32 7.5 (should add to PS1_REFERENCE as the reduction to use for refstacks)
• PSF.INPUT.CLIP.NSIGMA F32 5.0 (should add to PS1_REFERENCE as the reduction to use for refstacks)

PS1_REFERENCE:

• OUTPUT.NOCOMP BOOL TRUE
• OUTPUT.LOGFLUX BOOL FALSE
• OUTPUT.REPLICATE BOOL TRUE

NIGHTLY_STACK:

• OUTPUT.NOCOMP BOOL FALSE
• OUTPUT.LOGFLUX BOOL FALSE

THREEPI_STACK: (for comparison)

• PSF.INPUT.MAX F32 12.0
• OUTPUT.NOCOMP BOOL FALSE
• OUTPUT.LOGFLUX BOOL TRUE

QUICKSTACK_DEEP METADATA (testing unconvolved stacks version only, similar to QUICKSTACK)

• CONVOLVE BOOL FALSE
• COMBINE.REJ F32 3.5 # Rejection threshold in combination (sigma). tune?
• COMBINE.DISCARD F32 0.2 # Discard fraction for Olympic weighted mean. tune?
• OUTPUT.NOCOMP BOOL TRUE
• OUTPUT.LOGFLUX BOOL FALSE
• PSF.INPUT.MAX F32 8.0 # tune?
• PSF.INPUT.CLIP.NSIGMA F32 5.0

END

version label	filter	center skycell input warps (rejected)	center skycell FWHM (major)
MD10.refstack.20110821 V3	y	63(11)	0.96"
MD10.refstack.20110715 V3 error	y	93(6)	0.99"
MD10.psfrefstack.20110814 V3 error	y	41(3)	0.9"
MD10.refstack.20100721 V2	y	28(1)	1.06"
MD10.refstack.20110821 V3	z	72(0)	1.02"
MD10.refstack.20110718 V3 error	z	174(6)	1.09"
MD10.psfrefstack.20110814 V3 error	z	44(4)	0.98"
MD10.refstack.20100721 V2	z	23(0)	1.09"
MD10.refstack.20110821 V3	i	75(1)	0.99"
MD10.refstack.20110722 V3 error	i	132(6)	1.07"
MD10.psfrefstack.20110814 V3 error	i	46(2)	1.00"
MD10.deeprefstack.20110814 V3 error	i	217(11)	1.08"
MD10.refstack.20100721 V2	i	18(1)	1.32"
MD10.refstack.20110821 V3	r	76(0)	1.28"
MD10.refstack.20110722 V3 error	r	134(2)	1.33"
MD10.psfrefstack.20110814 V3 error	r	47(3)	1.28"
MD10.refstack.20100721 V2	r	19(1)	1.46"
MD10.refstack.20110821 V3	g	79(3)	1.4"
MD10.refstack.20110722 V3 error	g	123(0)	1.41"
MD10.psfrefstack.20110814 V3 error	g	45(0)	1.31"
MD10.refstack.20100721 V2	g	22(0)	1.57"

*center skycell for V3=skycell.055, V2=skycell.078

Using the MD10.refstack.20110722 and related set, histograms and mag-magerr plots for the reference stack, deeprefstack test, psfrefstack, typical nightly stack and V2 refstack are compared. The re-run MD10.refstack.20110821 will go a little deeper than the psfrefstack version.

A representative skycell mapping of the new tessellation and good_fraction coverage. Nightly stacks will have similar, if not more, trouble with covering the corner edge skycells: 012, 017, 021, 028, 071, 078, 082, 087.

Other plots that may be good to see

• Mask fractions vs V2
• mapping of all exposures WRT new tessellation to check maximum coverage given some fields offset and others have had extra dither patterns (i.e., like MD10)

Troubled skycells for MD10.refstack.20110821:

• z: 012, 018,021,029,071,079,082 quality issue with edge skycells
• y: 012, 018,021,029,071,079,082 quality issue with edge skycells
• r: 012, 017,021,028,041,051,071,078,082,087 quality issue with edge skycells except 041, 051 should be recovered
• i: 012,017,021,028,071,078,082,087 quality issue with edge skycells
• g: 012, 017, 021, 071,078, 082 quality issue with edge skycells

MD10.GR0 reprocessed exposures

The full set of exposures since April 2009 were reprocessed until the following tags.

• label: MD10.GR0
• data_group: MD10.GR0.20110715 = y
• data_group: MD10.GR0.20110718 = z
• data_group: MD10.GR0.20110720 = g
• data_group: MD10.GR0.20110721 = r
• data_group: MD10.GR0.20110722 = i
• dist_group: MD10.GR

Distribution

Planned distribution of data products:

• ​http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR (full): warp, nightly stack, warp-stack and stack-stack diffims
• ps1-md-GR-cat (cleaned): camera, warp-stack and stack-stack diffims

NOTE: full chip and camera products will not be included as datastore release bundles unless there is interest

(the stack-stack diffims are still awaiting the ppSub final version of the auto convolution direction choice addition)

Cleanup time-scale TBD see below

Data bundles will unpack in the following ways warp-stack diffims: all under gpc1/MD10.GR0/2011/08/09/MD!0.V3/skycell.NNN -- very large directory

Postage-stamp Server

Access to be tested still.

Missing/Failed Data

Summary of any failed data products during processing.

• two skycell.021(and?) failed to produce a nightly stack (date?)

Cleanup Cycle

IPP:

stage	time
chip	ASAP
camera
warps	(V0-V2)
stacks	like (V0-V2) after JT has sync'd remaining ones?
WSdiff	must wait for MOPS postage stamp requests to finish
SSdiff
refstack	not cleaned

Datastore:

stage	time
chips	none distributed
camera	none distributed
warps
stacks
WSdiff
SSdiff
refstack

Reprocessed Status

---

MD01.GR0

Status: Refstack and sample nightly stacks finished and distributed.

Started for OC143 (August 13, 2011) with a new reference stack and small sample of reprocessed exposures released as nightly stacks. Due to a need of rebalancing the reprocessing load, the reprocessed sample includes only those exposures that may be used in the reference stack.

MD03 does have some offset dither exposures illustrated in the sample g-band V3 plot.

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter	N total	N (FWHM<upper limit GPC1 pixels)'"
y	178	106 (<5.0 pixels)
z	377	214 (<5.0 pixels)
i	351	172 (<5.0 pixels)
r	330	120 (<5.0 pixels)
g	340	124 (<5.5 pixels)

MD01.V3 reference stack

• label: MD01.refstack.20110820
• data_group: MD01.refstack.20110820

version label	filter	center skycell input warps (rejected)	center skycell FWHM (major)
MD01.refstack.20110820 V3	y	52(1)	0.84"
MD01.refstack.20110813 V3 error	y	106(1)
MD01.refstack.20100812 V2	y	21(3)	1.04"
MD01.refstack.20110820 V3	z	58(3)	0.87"
MD01.refstack.20110813 V3 error	z	115(2)
MD01.refstack.20100812 V2	z	60(2)	1.12"
MD01.refstack.20110820 V3	i	74(3)	1.00"
MD01.refstack.20100812 V2	i	50(0)	1.13"
MD01.refstack.20110820 V3	r	54(7)	1.10"
MD01.refstack.20100812 V2	r	56(1)	1.24"
MD01.refstack.20110820 V3	g	65(7)	1.20"
MD01.refstack.20100812 V2	g	54(5)	1.24"

with the approximate FWHM (major) as the PSF FWHM from a center field skycell after running psphot on the stacks. The MD01.refstack.20110813 V3 error was a run without the DEEP_STACK reduction

Distributed under ps1-md-refstack -- stack images only.

Troubled skycells: (none for proper run?)

• 087-y,z (MD01.refstack.20110813 V3 error): quality issue rejected, edge skycell.
• 086-z (MD01.refstack.20110813 V3 error): quality issue rejected, edge skycell.

Data volume on disk:

Processing time:

MD01.GR0 reprocessed sample

• label: MD01.GR0
• data_group: MD01.GR0.20110812 = y
• data_group: MD01.GR0.20110815 = z
• data_group: MD01.GR0.20110816 = i
• data_group: MD01.GR0.20110817 = r
• data_group: MD01.GR0.20110818 = g
• dist_group: MD02.GR

Distributed under ps1-md-GR -- currently only a small sub-sample of nightly stack images only. SSdiff sample to be processed still.

Data volume on disk:

Processing time:

---

MD02.GR0

Status: Refstack and sample nightly stacks finished and distributed.

Started for OC144 (September 12, 2011) with a new reference stack and a small sample of reprocessed exposures released as nightly stacks. Due to a need of rebalancing the reprocessing load, only a minimal amount of processing required to make the reference stack was done.

MD02 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many are single exposure coverage and won't typically be enough for a reference stack.

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter	N total	N (FWHM<upper limit GPC1 pixels)
y	147	85 (<6.0 pixels)
z	305	101 (<4.5 pixels)
i	266	117 (<5.5 pixels)
r	215	118 (<6.5 pixels)
g	218	121 (<7.0 pixels)

MD02.V3 reference stack

• label: MD02.refstack.20110907
• data_group: MD02.refstack.20110907

version label	filter	center skycell input warps	center skycell FWHM (major)
MD02.refstack.20110907 V3	y	58(3)	1.01"
MD02.refstack.20101126 V2	y	47(3)	1.02"
MD02.refstack.20110907 V3	z	67(5)	0.90"
MD02.refstack.20101126 V2	z	84(10)	1.03"
MD02.refstack.20110907 V3	i	61(2)	1.00"
MD02.refstack.20101126 V2	i	102(15)	1.16"
MD02.refstack.20110907 V3	r	63(6)	1.15"
MD02.refstack.20101126 V2	r	67(15)	1.10"
MD02.refstack.20110907 V3	g	64(6)	1.32"
MD02.refstack.20101126 V2	g	71(24)	1.26"

with the approximate FWHM (major) as the PSF FWHM from a center field skycell after running psphot on the stacks.

Troubled skycells: none really? 040 in g,r oddly seems to have enough coverage to make a stack.

Distributed under ps1-md-refstack -- stack images only.

Data volume on disk:

Processing time:

MD02.GR0 reprocessed sample

• label: MD02.GR0
• data_group: MD02.GR0.20110902 = y
• data_group: MD02.GR0.20110903 = z
• data_group: MD02.GR0.20110906 = g
• data_group: MD02.GR0.20110905 = r
• data_group: MD02.GR0.20110904 = i
• dist_group: MD02.GR

Distributed under ps1-md-GR -- sample on nightly stack images only. SSdiff sample still to be run.

A typo in the loading script set the workdir to be MD01 rather than MD02 for all filter data groups (see string below) but will not cause any trouble other than the directory name in mirroring data

/gpc1/MD02.GR0/MD01.20110906/MD02.V3/skycell.055/*unconv.fits

Data volume on disk:

Processing time:

---

MD03.GR0

Status: In progress. Sub-sample of y and z-band exposures through warp, sample nightly stacks made.

MD03.V3 refstack needed before new observations in OC146 (November 10).

MD03 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many of the outer skycells are only covered by single to a few exposure without the 4 or more for a stack.

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter	N total	N (FWHM<upper limit GPC1 pixels)'"
y	224	97 (<5.0 pixels)
z	526	150 (<4.2 pixels)
i	488	?
r	377	?
g	388	?

MD03.V3 reference stack

• label: MD03.refstack.20111010
• data_group: MD03.refstack.20111010

version label	filter	center skycell input warps	center skycell FWHM (major)
MD03.refstack.20111010 V3	y
MD03.refstack.20101202 V2	y	75(12)
MD03.refstack.20111010 V3	z
MD03.refstack.20101202 V2	z	101(27)
MD03.refstack.20111010 V3	i
MD03.refstack.20101202 V2	i	108(17)
MD03.refstack.20111010 V3	r
MD03.refstack.20101202 V2	r	115(44)
MD03.refstack.20111010 V3	g
MD03.refstack.20101202 V2	g	90(29)

with the approximate FWHM (major) as the PSF FWHM from a center field skycell after running psphot on the stacks.

Troubled skycells:

Distributed under ps1-md-refstack -- stack images only soon

Data volume on disk:

Processing times:

MD03.GR0 reprocessed sample

• label: MD03.GR0
• data_group: MD03.GR0.20111001 = y
• data_group: MD03.GR0.20111002 = z

• dist_group: MD03.GR

---

MD04.GR0

Status: In progress. Starting with i-band rather than y for the sub-sample reprocessing exposures through warp.

New observations aren't planned until OC147 (December 9), but need it for comparison analysis ASAP.

Retired MD04.altstack in a pre-V3 tessellation from last November (20101102). Pre-V3 MD04.altstack warps have been purged (no longer updatable) but the stacks are still available for any comparison.

All past g-band exposures in the V3 tessellation (other filters nearly identical).

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter	N total	N (FWHM<upper limit GPC1 pixels)'"
y	209	?
z	343	?
i	428	208 (<5.0 pixels)
r	357	?
g	364	?

• this field had been the focus of deep stack tests, so the initial reprocessed set will attempt to be double the number from past versions.

MD04.V3 reference stack

• label: MD04.refstack.201110xx
• data_group: MD04.refstack.201110xx

version label	filter	center skycell input warps	center skycell FWHM (major)
MD04.refstack.201110xx V3	y
MD04.refstack.20101102 V2	y	100(14)
MD04.refstack.201110xx V3	z
MD04.refstack.20101102 V2	z	105(8)
MD04.refstack.201110xx V3	i
MD04.refstack.20101102 V2	i	108(15)
MD04.refstack.201110xx V3	r
MD04.refstack.20101102 V2	r	92(8)
MD04.refstack.201110xx V3	g
MD04.refstack.20101102 V2	g	101(13)

MD04.GR0 reprocessed sample

• label: MD04.GR0
• data_group: MD03.GR0.20111002 = i
• dist_group: MD03.GR

---

MD05.GR0

Coming soon. New observations aren't planned until OC147 (December 9).

All past g-band exposures in the V3 tessellation (other filters nearly identical).

MD05.V3 reference stack

MD05.GR0 reprocessed sample

---

MD06.GR0

Coming soon. New observations aren't planned until OC148 (January 8, 2012).

All past g-band exposures in the V3 tessellation (other filters nearly identical).

MD06.V3 reference stack

MD06.GR0 reprocessed sample

---

MD07.GR0

Coming soon. New observations aren't planned until OC148 (January 8, 2012).

All MD07 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD07 z-band nightly stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible.

MD07.V3 reference stack

MD07.GR0 reprocessed sample

---

MD08.GR0

Coming soon. New observations aren't planned until OC152 (May 5).

All past g-band exposures in the V3 tessellation (other filters nearly identical).

MD08.V3 reference stack

MD08.GR0 reprocessed sample

---

MD09.GR0

Coming soon.

All MD09 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD09 z-band nightly stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible.

MD09.V3 reference stack

MD09.GR0 reprocessed sample

---