Summary of MD.GR0

Summary page of information for the MD GR0 (grand reprocessing version 0) including the new reference stack in the V3 tessellation with the start of new observations of MD10 in OC142. This is version 0 as the test start of full reprocessing of the MD fields to be balanced along with other reprocessing, 3PI stacks and nightly science, generation of new reference stacks and as of yet chip/camera processing does not include a revised reference calibration catalog (currently applies to MD10, 01, 02, 03, 04, 05, 07, 08).

Creation of the new reference stacks need to be, at the very least, made prior to the return to the particular MD field given in the current schedule ​http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Medium_Deep_Schedule. After the July, August, September time crunch of a new field a month, there should be time to continue reprocessing and creation of new reference stacks in advance.

• 8/15/11 minimum mode re-processing for reference stacks after MD10 for evaluation and get next MD field ready (MD01)
• 9/30/11 more success weaving in reprocessing with LAP, started reprocessing for refstacks MD03,04 (with more reprocessed exposures)
• 10/09/11 MD03 refstack and sample stacks available, testing MD04 ~200 very deep stack, MD05 running
• 10/17/11 MD07 chip-->warp processing started with condor, using any extra cpu cycles.
• 11/06/11 MD07 refstacks being made with order ~120 inputs. MD04 deepstack testing continued, gri sample released, zy on hold until fixes made.
• 11/18/11 MD06 chip->warp processing running with condor, refstack preparation (using new ipp-20111110 ops tag now)
• 11/25/11 MD06 reftests made and y-band refstack started, MD08 chip->warp processing via condor, refstack preparation
• 12/02/11" MD06/08 on hold, data uncertain with ipp064 down.
• "12/20/11" MD06 processing recoverable, refstacks proceeding. MD08 a mess and will need to be re-run.

• Feedback is important: We would not want to continue full reprocessing of the MD fields if there has been a gross oversight or problem for a particular science interest. Please let us know of any issues you discover with example details. For example

• tessellation and/or offsets issues in any of the fields
  • corner of skycell.055 seems to be regularly masked (and in neighbors with the overlap)
  • fields with offsets need to include sufficient images in old and new pointings (i.e., MD10), for the magic reference stack at least.
• any issues with masking, mask values
  • been report of possible ones in early MD10 but still need details
• depth/quality of the stacks, reprocessed images in the new format
• any missing skycells, SMFs, other data products -- in particular MD10 which has been the only MD field full reprocessed and magicked for release of SMFs, warps.
  • MD10.refstack.20110722, MD10.refstack.20110821 r-band has two edge skycells (041,051) with poor quality and failed
  • MD10.refstack.20110722 y-band skycell.048 had rejection problem, fixed with MD10.refstack.20110821
• MD10.refstack.20110722 replacement with proper uncompressed MD10.refstack.20110821 for diffims

Where to find the data:

• ​http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR (full, pixel images)
• ​http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR-cat (cleaned, catalogs)
• ​http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack (full, pixel images for updated refstacks)
• ​http://datastore.ipp.ifa.hawaii.edu/ps1-md-staticsky (full and cleaned? new deepest stacks or just catalogs?)

Reference versus Deep Stack

A requirements description for the needs in the different types of stacks would be useful to summarize.

Reference stacks -- primarily for difference imaging for transient detection and necessary for magic destreaking.

• Magic destreaking: a fairly clean reference stack (uses the convolved stack) with as much spatial area coverage as possible to enable release of non-stack data products. Particularly important is inclusion of older data on different pointings and of course a fairly balanced dither set.
• Transient detection: a balance slanted towards best PSF but with some depth (1-1.5 magnitudes over nightly stacks) and also a fairly balanced dither set.

Deep stacks -- for deep static sky science.

• As deep as possible.. maximum FWHM allowance? minimum exposure time? specific request for at least ~10ks has been requested, with best FWHM possible and also a fairly balanced dither set..

See summary for each MD field reference stack for details on specifics for that field. Due to processing+time constraints/early mistakes, an attempt has been made to merge the similar criteria under the refstack label.

Since magic destreaking uses the convolved stack, looking into a quickstack version of the unconvolved stacks for the transients reference and deep stack that will be faster to make with more inputs.

For cases when a stack does not have a good sample of exposures with a reasonably small PSF FWHM, it should be possible to generate a reference stack with the fewer 'best' exposures for just SSdiffs and transient detection. MD02-g may be such a case.

For fields with shifted centers from offset earlier exposures, inclusion of enough exposures to provide a reference stack for magic is necessary in order for data from those associated chips/skycells to be distributed (being older data, may not have the optimum quality). Mixing older and newer data may also be a detriment.

Related to deep stacks, there needs to be enough inputs to obtain minimum depth that is as pixel contiguous as possible (see MD04 deep stack tests).

How much of a varying IQ can go into a stack before ppSub has difficulty modeling the variations within a skycell?

MD.V3 Tessellation

The new V3 tessellations are defined at PS1_Tessellations with the following general properties:

• 0.25"/pix (from natural ~0.258"/pixel, previously warped to 0.2"/pix up through V2)
• 10 x 10 skycell grid (previously 12 x 12)
• +x is West (flipped from before +x was East), +y is still North
• 320 pixel (80") overlap (from 300 pixels, 60", in V2)
• revised centers (if necessary) -- see relevant MD section for more details and the MDS field table (​http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Medium_Deep_Survey#Actual_Field_Centeres_-_2011-02-21) but in summary
  • MD10.GR0: from (351.979 -0.433) to (352.312, -0.433) however an additional offset looks like it should have been applied to cover more of the earlier observation dithers.
  • MD07.GR0: from (213.146, +53.417) to (213.454, +53.243) to best cover as much of the earlier observation dithers as possible.
  • MD08.GR0: very slight offset in RA to the W to just align with the MDS field pointing.
  • MD09.GR0: from (333.688, +0.283) closer to the current pointing of (334.188, +0.283) however an addition offset was added to move the center about half a skycell to the E to (333.980, +0.283) in order to cover as much of the earlier observation dithers as possible.
• All fields should be looked at to see if there are any problems with the new tessellation and dither pattern.
• MD07, MD09 will have a preliminary nightly stack made for testing the revised center as well as MD06 for NGC4258.

---

MD10.GR0

Status: reference stack (revised) processed and distributed. All exposures since April 2009 reprocessed and distributed except for SSdiffs. SSdiffs being held back until ppSub auto convolution choice is tested on.

Started for OC142 (July 15, 2011). Start with current statuswith a new reference stack and full sample of reprocessed exposures as an initial reprocessing version.

• z,y-band data processed prior to full Dark model improvements (higher background makes effect insignificant). gri-band processed after Dark model improvements.
• same photometry reference catalog as MD have previously been using.
• stack-stack diffims still using forced convolution of reference stack to input images

MD10.V3 center shifted to the current observation pointing even though there are earlier data that has offset dithered observations that populate the 0X8-0X9 skycells. Not as bad as MD07, but the revised reference stack may not provide for the outer skycells very well. Afterwards it was discovered the full extent of the previous observations wasn't quite accounted for, shown by these exposure plots with the MD10.V3 overlay and compared to a recent nightly stack set of exposures. This may need to be shifted in the next iteration.

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter	N total	N (FWHM<upper limit GPC1 pixels)
y	136	all
z	246	136 (<5.0 pixels), then finished reprocessing all
i	256	149 (<7.0 pixels), then finished reprocessing all
r	253	159 (<7.0 pixels), then finished reprocessing all
g	242	118 (<7.0 pixels), then finished reprocessing all

• also initially attempted to include some rough cuts by zeropoint and background (in 20110722) but after reprocessing all exposures wouldn't be included in the stacktool selection based solely on basically FWHM major. (improvements to selection and rejection for a future test)

MD10.V3 reference stack

• label: MD01.refstack.20110820
• data_group: MD01.refstack.20110820

Initial goal was to maximize depth while minimizing the PSF FWHM with enough inputs (4 or more) to produce a stack for all skycells covered, in particular

Compared to the cumulative metrics at ​http://ipp0022.ifa.hawaii.edu/ps1sc/metrics/range_2010-05-13_current/

Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20110722 (top blue), psfrefstack.20110814 (middle blue) and refstack.20100721 V2 (bottom blue). The unmatched blue/red overlap for refstack.20110722 FWHM major<6.5 pixels for exposures in the refstack is from using also low zeropoint and bright background cuts. Psfrefstack used <5.1 pixels.

The same refstacks with the selected exposures over time

Sample skycell coverage map by good_frac with N input warps, N rejected warps, dtime_script for making stack, and if quality error that rejects entire skycell. The ~130 input warps provided enough of a sample for the outer edge skycells from earlier exposures.

In tracking down an odd skycell (048, y-band), an odd rejection case was highlighted and prompted a modification to the rejection criteria based on the RMS of the stamps (see r32170). This may help reduce large number of rejections in future refstacks but likely needs some tuning (currently still rejecting too many?). Using the PSF FWHM cuts in the DEEP_STACK reduction also helps reject strange input warps as well.

Initially the stack processing was run without a reduction type and thus with just defaults that compresses the images (version 20110722gri, 20110715y, 20110718z) along with a test sample to produce the best PSF focused set (MD10.psfrefstack.20110814) and a test best deep stack (just skycell.055, i-band, not distributed). This is not optimal and a re-run was done with the DEEP_STACK reduction type (version 20110821) and should be the one used. The different reduction types include:

NULL (none/default values used from recipes/ppStack.config unless defined in gpc1/ppStack.config):

• PSF.INPUT.MAX F32 NAN
• PSF.INPUT.CLIP.NSIGMA F32 NAN
• OUTPUT.NOCOMP BOOL FALSE
• OUTPUT.LOGFLUX BOOL FALSE
• OUTPUT.REPLICATE BOOL TRUE

DEEP_STACK:

• OUTPUT.NOCOMP BOOL TRUE
• OUTPUT.LOGFLUX BOOL FALSE
• PSF.INPUT.MAX F32 7.5 (should add to PS1_REFERENCE as the reduction to use for refstacks)
• PSF.INPUT.CLIP.NSIGMA F32 5.0 (should add to PS1_REFERENCE as the reduction to use for refstacks)

PS1_REFERENCE:

• OUTPUT.NOCOMP BOOL TRUE
• OUTPUT.LOGFLUX BOOL FALSE
• OUTPUT.REPLICATE BOOL TRUE

NIGHTLY_STACK:

• OUTPUT.NOCOMP BOOL FALSE
• OUTPUT.LOGFLUX BOOL FALSE

THREEPI_STACK: (for comparison)

• PSF.INPUT.MAX F32 12.0
• OUTPUT.NOCOMP BOOL FALSE
• OUTPUT.LOGFLUX BOOL TRUE

QUICKSTACK_DEEP METADATA (testing unconvolved stacks version only, similar to QUICKSTACK)

• CONVOLVE BOOL FALSE
• COMBINE.REJ F32 3.5 # Rejection threshold in combination (sigma). tune?
• COMBINE.DISCARD F32 0.2 # Discard fraction for Olympic weighted mean. tune?
• OUTPUT.NOCOMP BOOL TRUE
• OUTPUT.LOGFLUX BOOL FALSE
• PSF.INPUT.MAX F32 8.0 # tune?
• PSF.INPUT.CLIP.NSIGMA F32 5.0

END

A summary table of the reprocessed reference stacks and the former V2 where center skycell for V3=skycell.055, V2=skycell.078. (need to add FWHM upper limit used)

version label	filter	center skycell input warps (rejected)	center skycell FWHM (major)
MD10.refstack.20110821 V3	y	63(11)	0.96"
MD10.refstack.20110715 V3 error	y	93(6)	0.99"
MD10.psfrefstack.20110814 V3 error	y	41(3)	0.9"
MD10.refstack.20100721 V2	y	28(1)	1.06"
MD10.refstack.20110821 V3	z	72(0)	1.02"
MD10.refstack.20110718 V3 error	z	174(6)	1.09"
MD10.psfrefstack.20110814 V3 error	z	44(4)	0.98"
MD10.refstack.20100721 V2	z	23(0)	1.09"
MD10.refstack.20110821 V3	i	75(1)	0.99"
MD10.refstack.20110722 V3 error	i	132(6)	1.07"
MD10.psfrefstack.20110814 V3 error	i	46(2)	1.00"
MD10.deeprefstack.20110814 V3 error	i	217(11)	1.08"
MD10.refstack.20100721 V2	i	18(1)	1.32"
MD10.refstack.20110821 V3	r	76(0)	1.28"
MD10.refstack.20110722 V3 error	r	134(2)	1.33"
MD10.psfrefstack.20110814 V3 error	r	47(3)	1.28"
MD10.refstack.20100721 V2	r	19(1)	1.46"
MD10.refstack.20110821 V3	g	79(3)	1.4"
MD10.refstack.20110722 V3 error	g	123(0)	1.41"
MD10.psfrefstack.20110814 V3 error	g	45(0)	1.31"
MD10.refstack.20100721 V2	g	22(0)	1.57"

Using the MD10.refstack.20110722 and related set, histograms and mag-magerr plots for the reference stack, deeprefstack test, psfrefstack, typical nightly stack and V2 refstack are compared. The re-run MD10.refstack.20110821 will go a little deeper than the psfrefstack version.

A representative skycell mapping of the new tessellation and good_fraction coverage. Nightly stacks will have similar, if not more, trouble with covering the corner edge skycells: 012, 017, 021, 028, 071, 078, 082, 087.

Other plots that may be good to see

• Mask fractions vs V2
• mapping of all exposures WRT new tessellation to check maximum coverage given some fields offset and others have had extra dither patterns (i.e., like MD10)

Troubled skycells for MD10.refstack.20110821:

• z: 012, 018,021,029,071,079,082 quality issue with edge skycells
• y: 012, 018,021,029,071,079,082 quality issue with edge skycells
• r: 012, 017,021,028,041,051,071,078,082,087 quality issue with edge skycells except 041, 051 should be recovered
• i: 012,017,021,028,071,078,082,087 quality issue with edge skycells
• g: 012, 017, 021, 071,078, 082 quality issue with edge skycells

MD10.GR0 reprocessed exposures

The full set of exposures since April 2009 were reprocessed until the following tags.

• label: MD10.GR0
• data_group: MD10.GR0.20110715 = y
• data_group: MD10.GR0.20110718 = z
• data_group: MD10.GR0.20110720 = g
• data_group: MD10.GR0.20110721 = r
• data_group: MD10.GR0.20110722 = i
• dist_group: MD10.GR

Distribution

Planned distribution of data products:

• ​http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR (full): warp, nightly stack, warp-stack and stack-stack diffims
• ​http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR-cat (cleaned): camera, warp-stack and stack-stack diffims

NOTE: full chip and camera products will not be included as datastore release bundles unless there is interest

Available data products on the datastore:

• warp (full): done
• nightly stack (full): done
• warp-stack diffim (full, cleaned): done, done
• camera/SMF: done
• stack-stack diffims: TBD - are still awaiting the ppSub final version of the auto convolution direction choice addition

Data bundles will unpack in the following ways

• warp-stack diffims: all under gpc1/MD10.GR0/2011/08/09/MD10.V3/skycell.0NN -- very large directory *

Postage-stamp Server

Access to be tested still.

Missing/Failed Data

Summary of any failed data products during processing.

• two skycell.021(and?) failed to produce a nightly stack (date?)

Cleanup Cycle

IPP (for faster postage stamp requests):

stage	time-frame
chip	similar timescale (V0-V2)
camera	not cleaned
warps	similar timescale (V0-V2)
stacks	similar timescale (V0-V2)
WSdiff	similar timescale (V0-V2), must wait for MOPS postage stamp requests to finish, but not this round
SSdiff	similar timescale (V0-V2)
refstack	not cleaned

Datastore:

stage	time-frame
chips	none distributed
camera	none distributed
warps	TBD
stacks	TBD
WSdiff	TBD
SSdiff	TBD
refstack	TBD

Reprocessed Status

---

MD01.GR0

Status: Refstack and sample nightly stacks finished and distributed.

Started for OC143 (August 13, 2011) with a new reference stack and small sample of reprocessed exposures released as nightly stacks. Due to a need of rebalancing the reprocessing load, the reprocessed sample includes only those exposures that may be used in the reference stack.

MD03 does have some offset dither exposures illustrated in the sample g-band V3 plot.

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter	N total	N (FWHM<upper limit GPC1 pixels)'"
y	178	106 (<5.0 pixels)
z	377	214 (<5.0 pixels)
i	351	172 (<5.0 pixels)
r	330	120 (<5.0 pixels)
g	340	124 (<5.5 pixels)

MD01.V3 reference stack

• label: MD01.refstack.20110820
• data_group: MD01.refstack.20110820

	Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20110820 (top blue) and refstack.2010812 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label	filter	center skycell input warps (rejected)	center skycell FWHM (major)
MD01.refstack.20110820 V3	y	52(1)	0.84"
MD01.refstack.20110813 V3 error	y	106(1)
MD01.refstack.20100812 V2	y	21(3)	1.04"
MD01.refstack.20110820 V3	z	58(3)	0.87"
MD01.refstack.20110813 V3 error	z	115(2)
MD01.refstack.20100812 V2	z	60(2)	1.12"
MD01.refstack.20110820 V3	i	74(3)	1.00"
MD01.refstack.20100812 V2	i	50(0)	1.13"
MD01.refstack.20110820 V3	r	54(7)	1.10"
MD01.refstack.20100812 V2	r	56(1)	1.24"
MD01.refstack.20110820 V3	g	65(7)	1.20"
MD01.refstack.20100812 V2	g	54(5)	1.24"

with the approximate FWHM (major) as the PSF FWHM from a center field skycell after running psphot on the stacks. The MD01.refstack.20110813 V3 error was a run without the DEEP_STACK reduction

Distributed under ps1-md-refstack -- stack images only.

Troubled skycells: none on initial inspection for refstack.20110820

• 087-y,z (MD01.refstack.20110813 V3 error): quality issue rejected, edge skycell.
• 086-z (MD01.refstack.20110813 V3 error): quality issue rejected, edge skycell.

Data volume on disk:

Processing time:

MD01.GR0 reprocessed sample

• label: MD01.GR0
• data_group: MD01.GR0.20110812 = y
• data_group: MD01.GR0.20110815 = z
• data_group: MD01.GR0.20110816 = i
• data_group: MD01.GR0.20110817 = r
• data_group: MD01.GR0.20110818 = g
• dist_group: MD02.GR

Distributed under ps1-md-GR -- currently only a small sub-sample of nightly stack images only. SSdiff sample to be processed still.

Data volume on disk:

Processing time:

---

MD02.GR0

Status: Refstack and sample nightly stacks finished and distributed.

Started for OC144 (September 12, 2011) with a new reference stack and a small sample of reprocessed exposures released as nightly stacks. Due to a need of rebalancing the reprocessing load, only a minimal amount of processing required to make the reference stack was done.

MD02 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many are single exposure coverage and won't typically be enough for a reference stack skycell.

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter	N total	N (FWHM<upper limit GPC1 pixels)
y	147	85 (<6.0 pixels)
z	305	101 (<4.5 pixels)
i	266	117 (<5.5 pixels)
r	215	118 (<6.5 pixels)
g	218	121 (<7.0 pixels)

MD02.V3 reference stack

• label: MD02.refstack.20110907
• data_group: MD02.refstack.20110907

	Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20110907 (top blue) and refstack.20101126 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label	filter	center skycell input warps(rejected)	center skycell FWHM (major)
MD02.refstack.20110907 V3	y	58(3)	1.01"
MD02.refstack.20101126 V2	y	47(3)	1.02"
MD02.refstack.20110907 V3	z	67(5)	0.90"
MD02.refstack.20101126 V2	z	84(10)	1.03"
MD02.refstack.20110907 V3	i	61(2)	1.00"
MD02.refstack.20101126 V2	i	102(15)	1.16"
MD02.refstack.20110907 V3	r	63(6)	1.15"
MD02.refstack.20101126 V2	r	67(15)	1.10"
MD02.refstack.20110907 V3	g	64(6)	1.32"
MD02.refstack.20101126 V2	g	71(24)	1.26"

with the approximate FWHM (major) as the PSF FWHM from a center field skycell after running psphot on the stacks.

Troubled skycells: none on initial inspection. 040 in g,r oddly seems to have enough coverage to make a stack.

Distributed under ps1-md-refstack -- stack images only.

Data volume on disk:

Processing time:

MD02.GR0 reprocessed sample

• label: MD02.GR0
• data_group: MD02.GR0.20110902 = y
• data_group: MD02.GR0.20110903 = z
• data_group: MD02.GR0.20110906 = g
• data_group: MD02.GR0.20110905 = r
• data_group: MD02.GR0.20110904 = i
• dist_group: MD02.GR

Distributed under ps1-md-GR -- sample on nightly stack images only. SSdiff sample still to be run.

A typo in the loading script set the workdir to be MD01 rather than MD02 for all filter data groups (see string below) but will not cause any trouble other than the directory name in mirroring data

/gpc1/MD02.GR0/MD01.2011090*/MD02.V3/skycell.*/*unconv.fits

Data volume on disk:

Processing time:

• chip-warp:
• refstack: 8-9000s/skycell for the ~60 input warps, on averages looks longer than the previous refstack MD02.refstack.20101126 (to be investigated)
• night stack: order 800-1200s/skycell

---

MD03.GR0

Status: refstacks and sample nightly stacks finished and distributed

• 10/3: Sub-sample of y and z-band exposures through warp, sample nightly stacks made. y-band refstack in progress.
• 10/4: r,i through warp. z-band refstack in progress
• 10/6: g through warp, ri refstack in progress
• 10/8: queued for distribution

MD03.V3 refstack needed before new observations in OC146 (November 10).

MD03 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many of the outer skycells are only covered by single to a few exposure without the 4 or more for a stack.

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter	N total	N (FWHM<upper limit GPC1 pixels)'"
y	224	97 (<5.0 pixels)
z	526	150 (<4.2 pixels)
i	488	163 (<4.8 pixels)
r	377	160 (<4.9 pixels)
g	388	147 (5.5 pixels)

• the number of exposures being reprocessed have been increased from that in MD01,02 in order to at least attempt to match the input warp numbers in the previous MD03.refstack (~100).

MD03.V3 reference stack

• label: MD03.refstack.20111010
• data_group: MD03.refstack.20111010

	Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20111010 (top blue) and refstack.20101202 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label	filter	center skycell input warps(rejected)	center skycell FWHM (major)
MD03.refstack.20111010 V3	y	108(7)	0.91"
MD03.refstack.20101202 V2	y	75(12)	1.13"
MD03.reftest.20111011 V3	y	57(2)	0.87"
MD03.reftest.20111011 V3	y	26(2)	0.83"
MD03.refstack.20111010 V3	z	125(18)	0.85"
MD03.refstack.20101202 V2	z	101(27)	1.04"
MD03.reftest.20111011 V3	z	60(2)	0.84"
MD03.reftest.20111011 V3	z	19(1)	0.84"
MD03.refstack.20111010 V3	i	116(7)	0.97"
MD03.refstack.20101202 V2	i	108(17)	1.12"
MD03.reftest.20111011 V3	i	67(5)	0.97"
MD03.reftest.20111011 V3	i	24(0)	0.98"
MD03.refstack.20111010 V3	r	123(4)	1.05"
MD03.refstack.20101202 V2	r	115(44)	1.10"
MD03.reftest.20111011 V3	r	64(3)	0.97"
MD03.reftest.20111011 V3	r	24(1)	1.02"
MD03.refstack.20111010 V3	g	114(5)	~1.08" (looked at 054,053 as well)
MD03.refstack.20101202 V2	g	90(29)	1.13"
MD03.reftest.20111011 V3	g	67(2)	1.08"
MD03.reftest.20111011 V3	g	17(2)	1.03"

• overshot the number of input warps compared to V2. The average PSF FWHM_major seems much better than expected from the histograms (above) of input exposures.

Troubled skycells: none on initial inspection. Far outer edges with few dithers may/may not have successful skycells. Sample skycell coverage maps for V2 and V3 with good_frac with N input warps, N rejected warps, dtime_script for making stack, and flag of any quality error that rejects an entire skycell (ERR). The ~110-125 input warps provided a sample for some of the far outer skycells from earlier dithers to be covered still.

Distributed under ps1-md-refstack

Data volume on disk: ~330GB

Processing time summary: 12-18000 sec for 100-125 input warps. ~6 days to reprocess a ~50% of exposures through warp, create reference stack, and distribute (along with LAP and normal nightly science).

MD03.GR0 reprocessed sample

• label: MD03.GR0
• data_group: MD03.GR0.20111001 = y
• data_group: MD03.GR0.20111002 = z
• data_group: MD03.GR0.20111003 = i
• data_group: MD03.GR0.20111004 = r
• data_group: MD03.GR0.20111005 = g

• dist_group: MD03.GR

Distributed under ps1-md-GR -- sample of nightly stack images only. SSdiff sample still to be run. Magicked camera SMFs to be done.

Data volume on disk:

Processing times:

---

MD04.GR0

Status: Refstacks and sample nightly stacks finished and distributed. Very deep stack tests in progress.

• 10/2 Starting with i-band rather than y for the sub-sample reprocessing exposures through warp.
• 10/4 z-band through warp done. testing very deep >150 stack on i-band. r-band running through warp
• 10/5 gy running through wrap
• 10/6: izrgy refstacks ready to queue
• 10/8: i-band refstack finished, z-band running
• 10/9: r-band running, g-band running
• 10/10: remaining y refstack finishing up
• 10/16: refstack and sample nightly stack set finished and distributed

New observations aren't planned until OC147 (December 9), but need it for comparison analysis ASAP.

Retired MD04.altstack in a pre-V3 tessellation from last November (20101102). Pre-V3 MD04.altstack warps have been purged (no longer updatable) but the stacks are still available for any comparison.

All past g-band exposures in the V3 tessellation (other filters nearly identical).

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter	N total	N (FWHM<8 pixels)	N (FWHM<upper limit GPC1 pixels)'"
y	209	177	146 (<5.5 pixels)
z	343	321	196 (<5.2 pixels)
i	428	392	208 (<5.0 pixels)
r	357	327	203 (<5.6 pixels)
g	364	313	206 (<6 pixels)

• the number of exposures being reprocessed have been increased from that in MD01,02 in order to at least attempt to match the input warp numbers in the previous MD03.refstack (~100). This field has also been the focus of deep stack tests, so the initial reprocessed set will attempt to be double the number from past versions (~200).

MD04.V3 reference stack

• label: MD04.refstack.20111012 (~100)
• data_group: MD04.refstack.20111012 (~100)

	Histograms of the FWHM major and minor from camera stage for reprocessed grizy exposures (red) and refstack.20111012 (top blue) and refstack.20101102 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label	filter	center skycell input warps	center skycell FWHM (major)
MD04.refstack.20111012 V3	y	107(0)	0.90"
MD04.reftest.20111011 V3	y	58(5)	0.87"
MD04.reftest.20111011 V3	y	20(0)	0.88"?
MD04.refstack.20101102 V2	y	100(14)	1.13"
MD04.refstack.20111012 V3	z	109(2)	0.90"
MD04.reftest.20111011 V3	z	52(1)	0.8"
MD04.reftest.20111011 V3	z	21(1)	0.92"?
MD04.refstack.20101102 V2	z	105(8)	1.14"
MD04.refstack.20111012 V3	i	92(11)	0.88"
MD04.reftest.20111011 V3	i	55(5)	0.98"
MD04.reftest.20111011 V3	i	26(2)	0.93"?
MD04.refstack.20111011 V3 (deep stack test, bad label)	i	206(16)	1.08"
MD04.deeptest.20111012 V3	i	308(17)	1.16"
MD04.refstack.20101102 V2	i	108(15)	1.22"
MD04.refstack.20111012 V3	r	128(9)	1.12"
MD04.reftest.20111011 V3	r	57(7)	1.06"
MD04.reftest.20111011 V3	r	26(0)	1.13"?
MD04.refstack.20101102 V2	r	92(8)	1.24"
MD04.refstack.20111012 V3	g	113(8)	1.21"
MD04.reftest.20111011 V3	g	60(0)	1.34"?
MD04.reftest.20111011 V3	g	22(0)	1.29"?
MD04.refstack.20101102 V2	g	101(13)	1.21"

• was an minor error in the cutoff for i,r: a fewer number in i-band and few more in r-band compared to others. Should be okay for basic refstack, and planned deeper stack should offset any deepstack concerns.

Troubled skycells: none on initial inspection. Far outer edges with few dithers may/may not have successful skycells. Sample skycell coverage maps for V2 and V3 with good_frac with N input warps, N rejected warps, dtime_script for making stack, and flag of any quality error that rejects an entire skycell (ERR). The ~110 input warps provided a sample for some of the far outer skycells from earlier dithers to be covered still.

MD04.V3 deep stack

• label/data_group:
  • MD04.refstack.20111011 (~200) for skycell.055, i-band
  • MD04.deeptest.20111012 (~300) for skycell.055, i-band
  • MD04.deeptest.20111015 (~300) full FPA, i-band
  • MD04.deeptest.20111017 (~300) full FPA, g-band

Histograms of the FWHM major and minor from camera stage for reprocessed exposures (red) and deeptest.20111012 (top blue, ~300 inputs) and refstack.20101102 V2 (bottom blue, ~100 inputs). Any unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Distribution:

• ​http://datastore.ipp.ifa.hawaii.edu/ps1-deeptest (MD04.deeptest.20111015)
• ​http://datastore.ipp.ifa.hawaii.edu/ps1-deeptest (MD04.deeptest.20111017)

Test skycell.055 distributed for

• ​http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.435977.819286.33 (MD04.refstack.20111011)
• ​http://datastore.ipp.ifa.hawaii.edu/ps1-deeptest/stack.462048.825873.46 (MD04.deeptest.20111012)

MD04.GR0 reprocessed sample

• label: MD04.GR0
• data_group: MD04.GR0.20111002 = i
• data_group: MD04.GR0.20111003 = z
• data_group: MD04.GR0.20111004 = r
• data_group: MD04.GR0.20111005 = g
• data_group: MD04.GR0.20111006 = y
• dist_group: MD04.GR

Distributed under ps1-md-GR -- sample of nightly stack images only. SSdiff sample still to be run. Magicked camera SMFs to be done.

Heather/Serge successful reprocessing of chip-warp under condor:

• label: condor_MD04.V3_01
• data_group: condor_MD04.V3_01_dg
• dist_group: NULL

MD04 data was magicked for PSPS loading 11/29.

---

MD05.GR0

Status: In progress

• 10/7: y-band chip-warp setup.
• 10/8: yzi chip-warp run
• 10/9: rg chip-warp run, yzi sample night stacks setup
• 10/10: sample nightly stacks yzi finished, gr started, y refstack started
• 10/16: refstack and sample nightly stack set finished and distributed

Coming soon. New observations aren't planned until OC147 (December 9).

All past g-band exposures in the V3 tessellation (other filters nearly identical).

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter	N total	N (FWHM<8 pixels)	N (FWHM<upper limit GPC1 pixels)'"
y	338	224	174 (<5.0 pixels)
z	383	325	164 (<4.5 pixels)
i	406	343	185 (<5.0 pixels)
r	416	386	168 (<5.2 pixels)
g	426	379	163 (<5.7 pixels)

MD05.V3 reference stack

• label: MD05.refstack.20111014 -- try to get out by 10/14
• data_group: MD05.refstack.20111014

	Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20111014 (top blue) and refstack.20110121 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label	filter	center skycell input warps	center skycell FWHM (major)
MD05.refstack.20111014 V3	y	133(4)	0.90"
MD05.reftest.20111011 V3	y	69(3)	0.85"
MD05.reftest.20111011 V3	y	22(0)	0.86"?
MD05.refstack.20110121 V2	y	100(25)	1.16"
MD05.refstack.20111014 V3	z	132(4)	0.84"
MD05.reftest.20111011 V3	z	70(9)	0.83"
MD05.reftest.20111011 V3	z	25(0)	0.88"?
MD05.refstack.20110121 V2	z	100(28)	1.01"
MD05.refstack.20111014 V3	i	147(17)	1.03"?
MD05.reftest.20111011 V3	i	63(8)	1.11"?
MD05.reftest.20111011 V3	i	19(0)	0.99"?
MD05.refstack.20110121 V2	i	100(14)	1.24"
MD05.refstack.20111014 V3	r	156(15)	1.15"
MD05.reftest.20111011 V3	r	61(1)	1.12"
MD05.reftest.20111011 V3	r	23(1)	1.23"?
MD05.refstack.20110121 V2	r	95(37)	1.31"
MD05.refstack.20111014 V3	g	154(8)	1.24"
MD05.reftest.20111011 V3	g	70(5)	1.21"
MD05.reftest.20111011 V3	g	21(5)	?1.28"?
MD05.refstack.20110121 V2	g	95(10)	1.38"

MD05.GR0 reprocessed sample

• label: MD05.GR0
• data_group: MD05.GR0.20111007 = y
• data_group: MD05.GR0.20111008 = z
• data_group: MD05.GR0.20111009 = i
• data_group: MD05.GR0.20111010 = r
• data_group: MD05.GR0.20111011 = g

• dist_group: MD05.GR

---

MD06.GR0

Status: In progress

• 11/18: all filters all exposures running chip->warp
• 11/23: reftest samples running

New observations aren't planned until OC148 (January 8, 2012).

All past g-band exposures in the V3 tessellation (other filters nearly identical).

Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks.

Filter	N total	N (FWHM<8 pixels)
y	298	268
z	471	446
i	564	513
r	436	391
g	407	341

MD06.V3 reference stack

• label: MD06.refstack.201111
• data_group: MD06.refstack.201111

Test stack (reftest) sample also made again for skycell.055 with different number of input warps.

• label: MD06.reftest.201111
• data_group: MD06.reftest.201111

	Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.201111xx (top blue) and refstack.2010xxxx (bottom blue). Any unmatched blue/red overlap for (V0) are due to ones included/rejected in the selected reprocessing. The previous (V0) refstack was only 8 inputs so should be much improvement!

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, (V0)=078) after running psphot on the stacks.

version label	filter	center skycell input warps	center skycell FWHM (major)
MD06.refstack.20111122 V3	y	107 (8)	0.92"
MD06.refstack.20110323 (V0)	y	87 (1)	1.25"
MD06.reftest.201111121 V3	y	33 (4)	0.83"
MD06.reftest.201111121 V3	y	69 (10)	0.89"
MD06.reftest.201111121 V3	y	107 (11)	0.92"
MD06.reftest.201111121 V3	y	129 (13)	0.94"
MD06.refstack.20111122 V3	z	110 (2)	0.93"
MD06.refstack.20110323 (V0)	z	16 (0)	1.19"
MD06.reftest.201111121 V3	z	30 (6)	0.88"
MD06.reftest.201111121 V3	z	70 (9)	0.89"
MD06.reftest.201111121 V3	z	102 (6)	0.95"
MD06.reftest.201111121 V3	z	142 (5)	0.94"
MD06.refstack.20111122 V3	i	108 (1)	0.98"
MD06.refstack.20110323 (V0)	i	8 (1)	1.17"
MD06.reftest.201111121 V3	i	27 (2)	0.86"
MD06.reftest.201111121 V3	i	62 (2)	0.92"
MD06.reftest.201111121 V3	i	108 (3)	0.97"
MD06.reftest.201111121 V3	i	128 (6)	0.99"
MD06.refstack.20111122 V3	r	114 (7)	1.10"
MD06.refstack.20110323 (V0)	r	12 (1)	1.13"
MD06.reftest.201111121 V3	r	31 (2)	1.15"
MD06.reftest.201111121 V3	r	60 (4)	1.06"
MD06.reftest.201111121 V3	r	114 (8)	1.10"
MD06.reftest.201111121 V3	r	145 (9)	1.11"
MD06.refstack.20111122 V3	g	113 (15)	1.17"
MD06.refstack.20110323 (V0)	g	12 (3)	1.36"
MD06.reftest.20111121 V3	g	25 (0)	1.20"
MD06.reftest.20111121 V3	g	64 (8)	1.14"
MD06.reftest.20111121 V3	g	113 (12)	1.31"
MD06.reftest.20111121 V3	g	114 (25)	1.17"

Troubled skycells?

• skycell.087 i-band had quality failure, rerun after excluding some earlier warps and ok?

MD06.GR0 reprocessed sample

• label: condor_MD06.V3_01
• data_group: condor_MD06.V3_01_dg
• dist_group: NULL

Reprocessed nightly stack sample under the following label and data_group.

• label: MD06.GR0
• data_group: MD06.GR0.20111122 = y
• data_group: MD06.GR0.20111123 = z
• data_group: MD06.GR0.20111124 = i
• data_group: MD06.GR0.20111125 = r
• data_group: MD06.GR0.20111126 = g

• dist_group: MD06.GR

Distributed under ps1-md-GR -- sample of nightly stack images only. SSdiff sample still to be run. Magicked camera SMFs to be done.

---

MD07.GR0

Status: In progress

• 10/12: all filters all exposures running chip->warp
• 11/6: all warps finished, y-band refstack started.
• 11/07: y-band refstack finished, z-band started, reftest samples run
• 11/08: zirg refstack running, nightly stack samples spanning the period since 4/2009 running.
• 11/12: full refstack set, nightly stack samples finished and distributed

Coming soon. New observations aren't planned until OC148 (January 8, 2012).

All MD07 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD07 z-band nightly stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible.

Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks (more important for this field given the two pointings since 4/2009).

Filter	N total	N (FWHM<8 pixels)
y	391	302
z	481	422
i	561	458
r	423	370
g	423	343

MD07.V3 reference stack

• label: MD07.refstack.20111106
• data_group: MD07.refstack.20111106

Test stack (reftest) sample also made again for skycell.055 with different number of input warps.

• label: MD07.reftest.20111107
• data_group: MD07.reftest.20111107

	Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20111106 (top blue) and refstack.20100330 (V0) (bottom blue). Any unmatched blue/red overlap for (V0) are due to ones included/rejected in the selected reprocessing. The previous (V0) refstack was only 8 inputs so should be much improvement!

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, (V0)=078) after running psphot on the stacks.

version label	filter	center skycell input warps	center skycell FWHM (major)
MD07.refstack.20111106 V3	y	116(8)	0.81"
MD07.refstack.20100330 (V0)	y	8(0)	0.91"
MD07.reftest.201111107 V3	y	31(1)	0.74"
MD07.reftest.201111107 V3	y	66(9)	0.77"
MD07.reftest.201111107 V3	y	143(10)	0.85"
MD07.refstack.20111106 V3	z	126(10)	0.96"
MD07.refstack.20100330 (V0)	z	8(0)	1.27"
MD07.reftest.201111107 V3	z	29(2)	0.89"
MD07.reftest.201111107 V3	z	59(4)	0.91"
MD07.reftest.201111107 V3	z	126(10)	0.96"
MD07.reftest.201111107 V3	z	154(14)	0.98"
MD07.refstack.20111106 V3	i	126(19)	1.07"
MD07.refstack.20100330 (V0)	i	8(0)	1.28"
MD07.reftest.201111107 V3	i	29(0)	1.00"
MD07.reftest.201111107 V3	i	68(9)	1.03"
MD07.reftest.201111107 V3	i	98(5)	1.05"
MD07.reftest.201111107 V3	i	126(19)	1.06"
MD07.reftest.201111107 V3	i	146(16)	1.07"
MD07.refstack.20111106 V3	r	138(6)	1.11"
MD07.refstack.20110121 (V0)	r	8(0)	1.46"
MD07.reftest.201111107 V3	r	25(2)	1.02"
MD07.reftest.201111107 V3	r	60(7)	1.08"
MD07.reftest.201111107 V3	r	115(18)	1.10"
MD07.reftest.201111107 V3	r	138(7)	1.10"
MD07.refstack.20111106 V3	g	152(11)	1.15"
MD07.refstack.20100330 (V0)	g	8(0)	1.67"
MD07.reftest.20111107 V3	g	29(2)	1.08"
MD07.reftest.20111107 V3	g	59(4)	1.11"
MD07.reftest.20111107 V3	g	126(10)	1.13"
MD07.reftest.20111107 V3	g	154(14)	1.14"

• seems to be some strange rejection behavior in at least the i-band.

How well does the "best" selected inputs cover the two pointings?

• note (V0) y-band refstack was at newer pointing, how many y-bands in old pointing?

Troubled skycells?

• 030-z quality fail
• first pointing (older data) skycell coverage needs further looking into. generally includes 093,094,095,096, 086,087,088, 078,079, 068,069, 059, 049, 039
• 077 large rej %, looks like for older data coverage

MD07.GR0 reprocessed sample

Reprocessing of chip-->warp done mostly with condor. Hiccup with loss of tmp files stalled things in the middle of warp, but pantasks picked up the processing fine using the label.

• label: condor_MD07.V3_01
• data_group: condor_MD07.V3_01_dg
• dist_group: NULL

Reprocessed nightly stack sample under the following label and data_group.

• label: MD07.GR0
• data_group: MD07.GR0.20111107 = y
• data_group: MD07.GR0.20111108 = z
• data_group: MD07.GR0.20111109 = i
• data_group: MD07.GR0.20111110 = r
• data_group: MD07.GR0.20111111 = g

• dist_group: MD07.GR

Distributed under ps1-md-GR -- sample of nightly stack images only. SSdiff sample still to be run. Magicked camera SMFs to be done.

---

MD08.GR0

Status: In progress

• 11/25: all filters all exposures running chip->warp

Coming soon. New observations aren't planned until OC152 (May 5).

All past g-band exposures in the V3 tessellation (other filters nearly identical).

Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks (more important for this field given the two pointings since 4/2009).

Filter	N total	N (FWHM<8 pixels)
y
z
i
r
g

MD08.V3 reference stack

• label: MD08.refstack.201111xx
• data_group: MD08.refstack.201111xx

Test stack (reftest) sample also made again for skycell.055 with different number of input warps.

• label: MD08.reftest.201111xx
• data_group: MD08.reftest.201111xx

	Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.201111xx (top blue) and refstack.2011xxxx (bottom blue). Any unmatched blue/red overlap for (V0) are due to ones included/rejected in the selected reprocessing. The previous (V0) refstack was only 8 inputs so should be much improvement!

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, (V0)=078) after running psphot on the stacks.

version label	filter	center skycell input warps	center skycell FWHM (major)
MD08.refstack.201111xx V3	y	()
MD08.refstack.2010xxxx	y	()
MD08.reftest.2011111xx V3	y	()
MD08.refstack.201111xx V3	z	()
MD08.refstack.2010xxxx	z	()
MD08.reftest.2011111xx V3	z	()
MD08.refstack.201111xx V3	i	()
MD08.refstack.2010xxxx	i	()
MD08.reftest.2011111xx V3	i	()
MD08.refstack.201111xx V3	r	()
MD08.refstack.2011xxxx	r	()
MD08.reftest.2011111xx V3	r	()
MD08.refstack.201111xx V3	g	()
MD08.refstack.2010xxxx	g	()
MD08.reftest.201111xx V3	g	()

Troubled skycells?

MD08.GR0 reprocessed sample

• label: condor_MD08.V3_01
• data_group: condor_MD08.V3_01_dg
• dist_group: NULL

Reprocessed nightly stack sample under the following label and data_group.

• label: MD08.GR0
• data_group: MD08.GR0.201111xx = y
• data_group: MD08.GR0.201111xx = z
• data_group: MD08.GR0.201111xx = i
• data_group: MD08.GR0.201111xx = r
• data_group: MD08.GR0.201111xx = g

• dist_group: MD08.GR

Distributed under ps1-md-GR -- sample of nightly stack images only. SSdiff sample still to be run. Magicked camera SMFs to be done.

---

MD09.GR0

Coming soon.

All MD09 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD09 z-band nightly stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible.

MD09.V3 reference stack

MD09.GR0 reprocessed sample

---