Summary of MD.GR0

Summary page of information for the MD GR0 (grand reprocessing version 0) including the new reference stack in the V3 tessellation with the start of new observations of MD10 in OC142. This is version 0 as the test start of full reprocessing of the MD fields to be balanced along with 3PI stacks and nightly science, generation of new reference stacks and not yet including a revised reference calibration catalog (currently for MD10, 01, 02).

Creation of the new reference stacks need to be at the very least made prior to the return to the particular MD field given in the current schedule ​http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Medium_Deep_Schedule. After the July, August, September crunch of a new field a month there should be time to continue reprocessing and creation of new reference stacks in advance.

• Feedback is critical at this point. We would not want to continue full reprocessing of the MD fields if there has been a gross oversight or problem for a particular science interest. Please let us know of any issues you discover with example details.

Reference versus Deep Stack

A requirements description for the needs in the different types of stacks would be useful to summarize.

Reference stacks -- primarily for difference imaging for transient detection and necessary for magic destreaking.

• Magic destreaking: a fairly clean reference stack (uses convolved version) with as much area coverage as possible to enable release of not-stack data products.
• Transient detection: a balance slanted towards best PSF but with some depth (1-1.5 magnitudes over nightly stacks).

Deep stacks -- for deep static sky science.

• As deep as possible.. maximum FWHM allowance? minimum exposure time? at least ~10ks has been requested, with best FWHM possible.

See summary for each MD field reference stack for details on specifics for that field. Due to processing+time constraints/early mistakes, an attempted has been made to merge the similar requests.

MD.V3 Tessellation

The new V3 tessellations are defined at PS1_Tessellations with the following properties:

• 0.25"/pix (previously 0.2"/pix up through V2, and from natural ~0.258"/pixel)
• 10 x 10 skycell grid (previously 12 x 12)
• +x is West (flipped from before +x was East), +y is still North
• 320 pixel (80") overlap (from 300 pixels, 60", in V2)
• revised centers if necessary -- was necessary for MD10 (will be for MD07, MD09 and slight RA shift in MD08). Should check all fields work well with new tessellation centers, particularly MD06/NGC4258.

MD10.GR0

Started for OC142 (July 15, 2011) with a new reference stack and full sample of reprocessed exposures as an initial reprocessing version.

• z,y-band data processed prior to full Dark model improvements (higher background makes effect insignificant). gri-band processed after Dark model improvements.

• same photometry reference catalog as before

• ...

• stack-stack diffims still using forced convolution of reference stack to input images

• ...

• center based on current observation pointing even though there are earlier data that has offset dithered observations that populate the 0X8-0X9 skycells. not as bad as MD07, but the revised reference stack may not provide for the outer skycells very well.

MD10.V3 reference stack

Goal to maximize depth while minimizing the PSF FWHM.

Compared to the cumulative metrics at ​http://ipp0022.ifa.hawaii.edu/ps1sc/metrics/range_2010-05-13_current/

Initially ran without reduction type and ran sample of best PSF and test deep stack (version 20110722gri, 20110715y, 20110718z) along with a test sample of a best PSF focused set MD10.psfrefstack.20110814) and a test best deep stack (just skycell.055, i-band, not distributed). After discovery of this error, re-ran as DEEP_STACK reduction type (version 20110821) and should be the one used. (need to change in SSdiffs soon)

Side note: running w/o the DEEP_STACK and the PSF restrictions highlighted a strange behavior in the image rejection in ppStack. (add detail and modification description)

Reduction type description: NULL (none/default values used from recipes/ppStack.config unless defined in gpc1/ppStack.config):

• PSF.INPUT.MAX F32 NAN
• PSF.INPUT.CLIP.NSIGMA F32 NAN
• OUTPUT.NOCOMP BOOL FALSE
• OUTPUT.LOGFLUX BOOL FALSE
• OUTPUT.REPLICATE BOOL TRUE

DEEP_STACK:

• OUTPUT.NOCOMP BOOL TRUE
• OUTPUT.LOGFLUX BOOL FALSE
• PSF.INPUT.MAX F32 7.5 (should add to PS1_REFERENCE as the reduction to use for refstacks)
• PSF.INPUT.CLIP.NSIGMA F32 5.0 (should add to PS1_REFERENCE as the reduction to use for refstacks)

PS1_REFERENCE:

• OUTPUT.NOCOMP BOOL TRUE
• OUTPUT.LOGFLUX BOOL FALSE
• OUTPUT.REPLICATE BOOL TRUE

NIGHTLY_STACK:

• OUTPUT.NOCOMP BOOL FALSE
• OUTPUT.LOGFLUX BOOL FALSE

THREEPI_STACK: (for comparison)

• PSF.INPUT.MAX F32 12.0
• OUTPUT.NOCOMP BOOL FALSE
• OUTPUT.LOGFLUX BOOL TRUE

QUICKSTACK_DEEP METADATA (testing unconvolved stacks version only, similar to QUICKSTACK)

• CONVOLVE BOOL FALSE
• COMBINE.REJ F32 3.5 # Rejection threshold in combination (sigma). tune?
• COMBINE.DISCARD F32 0.2 # Discard fraction for Olympic weighted mean. tune?
• OUTPUT.NOCOMP BOOL TRUE
• OUTPUT.LOGFLUX BOOL FALSE
• PSF.INPUT.MAX F32 8.0 # tune?
• PSF.INPUT.CLIP.NSIGMA F32 5.0

END

version label	filter	center skycell input warps (rejected)	center skycell FWHM (major)
MD10.refstack.20110821 V3	y	63(11)	0.96"
MD10.refstack.20110715 V3 error	y	93(6)	0.99"
MD10.psfrefstack.20110814 V3 error	y	41(3)	0.9"
MD10.refstack.20100721 V2	y	28(1)	1.06"
MD10.refstack.20110821 V3	z	72(0)	1.02"
MD10.refstack.20110718 V3 error	z	174(6)	1.09"
MD10.psfrefstack.20110814 V3 error	z	44(4)	0.98"
MD10.refstack.20100721 V2	z	23(0)	1.09"
MD10.refstack.20110821 V3	i	75(1)	0.99"
MD10.refstack.20110722 V3 error	i	132(6)	1.07"
MD10.psfrefstack.20110814 V3 error	i	46(2)	1.00"
MD10.deeprefstack.20110814 V3 error	i	217(11)	1.08"
MD10.refstack.20100721 V2	i	18(1)	1.32"
MD10.refstack.20110821 V3	r	76(0)	1.28"
MD10.refstack.20110722 V3 error	r	134(2)	1.33"
MD10.psfrefstack.20110814 V3 error	r	47(3)	1.28"
MD10.refstack.20100721 V2	r	19(1)	1.46"
MD10.refstack.20110821 V3	g	79(3)	1.4"
MD10.refstack.20110722 V3 error	g	123(0)	1.41"
MD10.psfrefstack.20110814 V3 error	g	45(0)	1.31"
MD10.refstack.20100721 V2	g	22(0)	1.57"

*center skycell for V3=skycell.055, V2=skycell.078

• FWHM mean,histograms vs V2
• Mask fractions vs V2
• photometry vs V2
• reference stack vs typical nightly stack skycell comparison

Troubled skycells:

• z: 012, 018,021,029,071,079,082 quality issue with edge skycells
• y: 012, 018,021,029,071,079,082 quality issue with edge skycells
• r: 012, 017,021,028,041,051,071,078,082,087 quality issue with edge skycells except 041, 051 should be recovered
• i: 012,017,021,028,071,078,082,087 quality issue with edge skycells
• g: 012, 017, 021, 071,078, 082 quality issue with edge skycells

MD10.GR0 reprocessed exposures

The full set of exposures from April 2009 were run.

• label: MD10.GR0
• data_group: MD10.GR0.20110715 = y
• data_group: MD10.GR0.20110718 = z
• data_group: MD10.GR0.20110720 = g
• data_group: MD10.GR0.20110721 = r
• data_group: MD10.GR0.20110722 = i
• dist_group: MD10.GR

Distribution

• ​http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR (full): warp, nightly stack, warp-stack and stack-stack diffims
• ps1-md-GR-cat (cleaned): camera, warp-stack and stack-stack diffims

NOTE: full chip and camera products will not be included as datastore release bundles unless there is interest

Cleanup time-scale TBD see below

Data bundles will unpack in the following ways warp-stack diffims: all under gpc1/MD10.GR0/2011/08/09/MD!0.V3/skycell.NNN -- very large directory

Postage-stamp Server

Access to be tested still.

Missing/Failed Data

Summary of any failed data products during processing.

• two skycell.021(and?) failed to produce a nightly stack (date?)

Cleanup Cycle

IPP:

stage	time
chip	ASAP
camera
warps	(V0-V2)
stacks	like (V0-V2) after JT has sync'd remaining ones?
WSdiff	must wait for MOPS postage stamp requests to finish
SSdiff
refstack	not cleaned

Datastore:

stage	time
chips	none distributed
camera	none distributed
warps
stacks
WSdiff
SSdiff
refstack

MD01.GR0

Started for OC143 (August 13, 2011) with a new reference stack and small sample of reprocessed exposures released as nightly stacks. Due to a need of rebalancing the reprocessing load, only a minimal amount of processing required to make the reference stack was done.

Reprocessed sample includes only those exposures that may be used in the reference stack.

MD01.V3 reference stack

• label: MD01.refstack.20110820
• data_group: MD01.refstack.20110820

version label	filter	center skycell input warps (rejected)	center skycell FWHM (major)
MD01.refstack.20110820 V3	y	52(1)	0.84"
MD01.refstack.20110813 V3 error	y	106(1)
MD01.refstack.20100812 V2	y	21(3)	1.04"
MD01.refstack.20110820 V3	z	58(3)	0.87"
MD01.refstack.20110813 V3 error	z	115(2)
MD01.refstack.20100812 V2	z	60(2)	1.12"
MD01.refstack.20110820 V3	i	74(3)	1.00"
MD01.refstack.20100812 V2	i	50(0)	1.13"
MD01.refstack.20110820 V3	r	54(7)	1.10"
MD01.refstack.20100812 V2	r	56(1)	1.24"
MD01.refstack.20110820 V3	g	65(7)	1.20"
MD01.refstack.20100812 V2	g	54(5)	1.24"

with the approximate FWHM (major) as the PSF FWHM from a center field skycell after running psphot on the stacks. The MD01.refstack.20110813 V3 error was a run without the DEEP_STACK reduction

Distributed under ps1-md-refstack -- stack images only.

Troubled skycells: (none for proper run?)

• 087-y,z (MD01.refstack.20110813 V3 error): quality issue rejected, edge skycell.
• 086-z (MD01.refstack.20110813 V3 error): quality issue rejected, edge skycell.

Data volume on disk:

Processing time:

MD01.GR0 reprocessed sample

• label: MD01.GR0
• data_group: MD01.GR0.20110812 = y
• data_group: MD01.GR0.20110815 = z
• data_group: MD01.GR0.20110816 = i
• data_group: MD01.GR0.20110817 = r
• data_group: MD01.GR0.20110818 = g
• dist_group: MD02.GR

Distributed under ps1-md-GR -- sample on nightly stack images only. SSdiff sample to be processed still.

Data volume on disk:

Processing time:

MD02.GR0

Started for OC144 (September 12, 2011) with a new reference stack and a small sample of reprocessed exposures released as nightly stacks. Due to a need of rebalancing the reprocessing load, only a minimal amount of processing required to make the reference stack was done.

MD02.V3 reference stack

• label: MD02.refstack.20110907
• data_group: MD02.refstack.20110907

version label	filter	center skycell input warps	center skycell FWHM (major)
MD02.refstack.20110907 V3	y	58(3)	1.01"
MD02.refstack.20101126 V2	y	47(3)	1.02"
MD02.refstack.20110907 V3	z	67(5)	0.90"
MD02.refstack.20101126 V2	z	84(10)	1.03"
MD02.refstack.20110907 V3	i	61(2)	1.00"
MD02.refstack.20101126 V2	i	102(15)	1.16"
MD02.refstack.20110907 V3	r	63(6)	1.15"
MD02.refstack.20101126 V2	r	67(15)	1.10"
MD02.refstack.20110907 V3	g	64(6)	1.32"
MD02.refstack.20101126 V2	g	71(24)	1.26"

with the approximate FWHM (major) as the PSF FWHM from a center field skycell after running psphot on the stacks.

Troubled skycells: none really? 040 in g,r oddly seems to have enough coverage to make a stack.

Distributed under ps1-md-refstack -- stack images only.

Data volume on disk:

Processing time:

MD02.GR0 reprocessed sample

• label: MD02.GR0
• data_group: MD02.GR0.20110902 = y
• data_group: MD02.GR0.20110903 = z
• data_group: MD02.GR0.20110906 = g
• data_group: MD02.GR0.20110905 = r
• data_group: MD02.GR0.20110904 = i
• dist_group: MD02.GR

Distributed under ps1-md-GR -- sample on nightly stack images only. SSdiff sample still to be run.

A typo in the loading script set the workdir to be MD01 rather than MD02 for all filter data groups (see string below) but will not cause any trouble other than the directory name in mirroring data

/gpc1/MD02.GR0/MD01.20110906/MD02.V3/skycell.055/*unconv.fits

Data volume on disk:

Processing time:

MD03.GR0

Coming soon. MD03.V3 needed before OC146 (November 10)

MD03.V3 reference stack

MD03.GR0 reprocessed sample