= Summary of MD.GR0 = [[PageOutline]] Summary page of information for the MD GR0 (grand reprocessing version 0) including the new reference stack in the V3 tessellation with the start of new observations of MD10 in OC142. This is version 0 as the test start of full reprocessing of the MD fields to be balanced along with other reprocessing, 3PI stacks and nightly science, generation of new reference stacks and as of yet does not include a revised reference calibration catalog (currently for MD10, 01, 02). Creation of the new reference stacks need to be at the very least made prior to the return to the particular MD field given in the current schedule http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Medium_Deep_Schedule. After the July, August, September time crunch of a new field a month there should be time to continue reprocessing and creation of new reference stacks in advance. * ''8/15/11'' LAP has priority, current mode of only minimum processing for reference stacks * '''Feedback is important:''' We would not want to continue full reprocessing of the MD fields if there has been a gross oversight or problem for a particular science interest. Please let us know of any issues you discover with example details. * tessellation and/or offsets issues in any of the fields * any issues with masking, mask values - been report of possible ones in early MD10 but need details * depth/quality of the stacks, reprocessed images in the new format * any missing skycells, SMFs, other data products * MD10.refstack.20110722 replacement with proper uncompressed MD10.refstack.20110821 for diffims * == Reference versus Deep Stack == A requirements description for the needs in the different types of stacks would be useful to summarize. Reference stacks -- primarily for difference imaging for transient detection and necessary for magic destreaking. * Magic destreaking: a fairly clean reference stack (uses convolved version) with as much area coverage as possible to enable release of not-stack data products. Particularly important is inclusion of older data on different pointings and of course a fairly balanced dither set. * Transient detection: a balance slanted towards best PSF but with some depth (1-1.5 magnitudes over nightly stacks). Deep stacks -- for deep static sky science. * As deep as possible.. maximum FWHM allowance? minimum exposure time? at least ~10ks has been requested, with best FWHM possible initially. See summary for each MD field reference stack for details on specifics for that field. Due to processing+time constraints/early mistakes, an attempted has been made to merge the similar criteria. = MD.V3 Tessellation = The new V3 tessellations are defined at [wiki:PS1_Tessellations] with the following general properties: * 0.25"/pix (from natural ~0.258"/pixel, previously warped to 0.2"/pix up through V2) * 10 x 10 skycell grid (previously 12 x 12) * +x is West (flipped from before +x was East), +y is still North * 320 pixel (80") overlap (from 300 pixels, 60", in V2) * revised centers (if necessary) -- see relevant MD section for more details and the MDS field table (http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Medium_Deep_Survey#Actual_Field_Centeres_-_2011-02-21) but in summary * [wiki:MD.GR0#MD10.GR0 MD10.GR0]: from (351.979 -0.433) to (352.312, -0.433) however an additional offset looks like it should have been applied to cover more of the earlier observation dithers. * [wiki:MD.GR0#MD07.GR0 MD07.GR0]: from (213.146, +53.417) to (213.454, +53.243) to best cover as much of the earlier observation dithers as possible. * [wiki:MD.GR0#MD08.GR0 MD08.GR0]: very slight offset in RA to the W to just align with the MDS field pointing. * [wiki:MD.GR0#MD09.GR0 MD09.GR0]: from (333.688, +0.283) closer to the current pointing of (334.188, +0.283) however an addition offset was added to move the center about half a skycell to the E to (333.980, +0.283) in order to cover as much of the earlier observation dithers as possible. * All fields should be looked at to see if there are any problems with the new tessellation and dither pattern. * [wiki:MD.GR0#MD07.GR0 MD07], [wiki:MD.GR0#MD09.GR0 MD09] will have a preliminary nightly stack made for testing the revised center as well as [wiki:MD.GR0#MD06.GR0 MD06] for NGC4258. ---- = MD10.GR0 = Started for OC142 (July 15, 2011). Start with current statuswith a new reference stack and full sample of reprocessed exposures as an initial reprocessing version. * z,y-band data processed prior to full Dark model improvements (higher background makes effect insignificant). gri-band processed after Dark model improvements. * same photometry reference catalog as MD have previously been using. * * stack-stack diffims still using forced convolution of reference stack to input images MD10.V3 center shifted to the current observation pointing even though there are earlier data that has offset dithered observations that populate the 0X8-0X9 skycells. Not as bad as MD07, but the revised reference stack may not provide for the outer skycells very well. Afterwards it was discovered the full extent of the previous observations was wrong, shown by these exposure plots with the MD10.V3 overlay and compared to a recent nightly stack set of exposures. This may need to be shifted in the next iteration. [[Image(MD10_g_smf.png,150px)]] [[Image(MD10_r_smf.png,150px)]] [[Image(MD10_i_smf.png,150px)]] [[Image(MD10_z_smf.png,150px)]] [[Image(MD10_y_smf.png,150px)]][[Image(MD10ns_i_smf.png,150px)]] == MD10.V3 reference stack == Goal to maximize depth while minimizing the PSF FWHM with enough inputs (4 or more) to produce a stack for all skycells covered. Compared to the cumulative metrics at http://ipp0022.ifa.hawaii.edu/ps1sc/metrics/range_2010-05-13_current/ [[Image(MD10.g.00000.expfwhm_stkrefpsfv2.png,300px)]] [[Image(MD10.g.00000.expfwhmzptbgmjd_stkrefpsfv2.png,400px)]] [[Image(MD10.refstack.20110722.NULL.g.00000.stksummap.png,500px)]] In tracking down an odd skycell (048, y-band) highlighted an odd rejection case and prompted a modification to the criteria based on the RMS (see r32170). This may help reduce other cases of large number of rejections but likely needs some tuning (currently too restrictive?). Using the PSF FWHM cuts in the DEEP_STACK reduction also helps reject strange input warps as well. Initially the stack processing was ran without a reduction type and ran sample of best PSF and test deep stack (version 20110722gri, 20110715y, 20110718z) along with a test sample of a best PSF focused set MD10.psfrefstack.20110814) and a test best deep stack (just skycell.055, i-band, not distributed). After discovery of this error, re-ran as DEEP_STACK reduction type (version 20110821) and should be the one used. (need to change in SSdiffs soon) Reduction type description: NULL (none/default values used from recipes/ppStack.config unless defined in gpc1/ppStack.config): * PSF.INPUT.MAX F32 NAN * PSF.INPUT.CLIP.NSIGMA F32 NAN * OUTPUT.NOCOMP BOOL FALSE * OUTPUT.LOGFLUX BOOL FALSE * OUTPUT.REPLICATE BOOL TRUE DEEP_STACK: * OUTPUT.NOCOMP BOOL TRUE * OUTPUT.LOGFLUX BOOL FALSE * PSF.INPUT.MAX F32 7.5 (should add to PS1_REFERENCE as the reduction to use for refstacks) * PSF.INPUT.CLIP.NSIGMA F32 5.0 (should add to PS1_REFERENCE as the reduction to use for refstacks) PS1_REFERENCE: * OUTPUT.NOCOMP BOOL TRUE * OUTPUT.LOGFLUX BOOL FALSE * OUTPUT.REPLICATE BOOL TRUE NIGHTLY_STACK: * OUTPUT.NOCOMP BOOL FALSE * OUTPUT.LOGFLUX BOOL FALSE THREEPI_STACK: (for comparison) * PSF.INPUT.MAX F32 12.0 * OUTPUT.NOCOMP BOOL FALSE * OUTPUT.LOGFLUX BOOL TRUE QUICKSTACK_DEEP METADATA (testing unconvolved stacks version only, similar to QUICKSTACK) * CONVOLVE BOOL FALSE * COMBINE.REJ F32 3.5 # Rejection threshold in combination (sigma). tune? * COMBINE.DISCARD F32 0.2 # Discard fraction for Olympic weighted mean. tune? * OUTPUT.NOCOMP BOOL TRUE * OUTPUT.LOGFLUX BOOL FALSE * PSF.INPUT.MAX F32 8.0 # tune? * PSF.INPUT.CLIP.NSIGMA F32 5.0 END || '''version label''' || '''filter''' || '''center skycell input warps (rejected)''' || '''center skycell FWHM (major)''' || || MD10.refstack.20110821 V3 || y || 63(11) || 0.96" || || MD10.refstack.20110715 V3 error || y || 93(6) || 0.99" || || MD10.psfrefstack.20110814 V3 error || y || 41(3) || 0.9" || || MD10.refstack.20100721 V2 || y || 28(1) || 1.06" || || || || || || || MD10.refstack.20110821 V3 || z || 72(0) || 1.02" || || MD10.refstack.20110718 V3 error || z || 174(6) || 1.09" || || MD10.psfrefstack.20110814 V3 error || z || 44(4) || 0.98" || || MD10.refstack.20100721 V2 || z || 23(0) || 1.09" || || || || || || || MD10.refstack.20110821 V3 || i || 75(1) || 0.99" || || MD10.refstack.20110722 V3 error || i || 132(6) || 1.07" || || MD10.psfrefstack.20110814 V3 error|| i || 46(2) || 1.00" || || MD10.deeprefstack.20110814 V3 error || i || 217(11) || 1.08" || || MD10.refstack.20100721 V2 || i || 18(1) || 1.32" || || || || || || || MD10.refstack.20110821 V3 || r || 76(0) || 1.28" || || MD10.refstack.20110722 V3 error || r || 134(2) || 1.33" || || MD10.psfrefstack.20110814 V3 error || r || 47(3) || 1.28" || || MD10.refstack.20100721 V2 || r || 19(1) || 1.46" || || || || || || || MD10.refstack.20110821 V3 || g || 79(3) || 1.4" || || MD10.refstack.20110722 V3 error || g || 123(0) || 1.41" || || MD10.psfrefstack.20110814 V3 error || g || 45(0) || 1.31" || || MD10.refstack.20100721 V2 || g || 22(0) || 1.57" || || || || || || *center skycell for V3=skycell.055, V2=skycell.078 Using the MD10.refstack.20110722 and related set, histograms and mag-magerr plots for the reference stack, deeprefstack test, psfrefstack, typical nightly stack and V2 refstack are compared. The re-run MD10.refstack.20110821 will go a little deeper than the psfrefstack version. [[Image(MD10.20110814_i_testdeep.png,700px)]] A representative skycell mapping of the new tessellation and good_fraction coverage. Nightly stacks will have similar, if not more, trouble with covering the corner edge skycells: 012, 017, 021, 028, 071, 078, 082, 087. [[Image(MD10.refstack.20110821.NULL.g.00000.stksummap.png,700px)]] Other plots that may be good to see * Mask fractions vs V2 * mapping of all exposures WRT new tessellation to check maximum coverage given some fields offset and others have had extra dither patterns (i.e., like MD10) Troubled skycells for MD10.refstack.20110821: * z: 012, 018,021,029,071,079,082 quality issue with edge skycells * y: 012, 018,021,029,071,079,082 quality issue with edge skycells * r: 012, 017,021,028,041,051,071,078,082,087 quality issue with edge skycells except 041, 051 should be recovered * i: 012,017,021,028,071,078,082,087 quality issue with edge skycells * g: 012, 017, 021, 071,078, 082 quality issue with edge skycells == MD10.GR0 reprocessed exposures == The full set of exposures since April 2009 were reprocessed until the following tags. * label: MD10.GR0 * data_group: MD10.GR0.20110715 = y * data_group: MD10.GR0.20110718 = z * data_group: MD10.GR0.20110720 = g * data_group: MD10.GR0.20110721 = r * data_group: MD10.GR0.20110722 = i * dist_group: MD10.GR == Distribution == Planned distribution of data products: * http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR (full): warp, nightly stack, warp-stack and stack-stack diffims * ps1-md-GR-cat (cleaned): camera, warp-stack and stack-stack diffims '''NOTE: full chip and camera products will not be included as datastore release bundles unless there is interest''' (the stack-stack diffims are still awaiting the ppSub final version of the auto convolution direction choice addition) Cleanup time-scale TBD see below Data bundles will unpack in the following ways warp-stack diffims: all under gpc1/MD10.GR0/2011/08/09/MD!0.V3/skycell.NNN -- very large directory == Postage-stamp Server == Access to be tested still. == Missing/Failed Data == Summary of any failed data products during processing. * two skycell.021(and?) failed to produce a nightly stack (date?) == Cleanup Cycle == IPP: || stage || time || || chip || ASAP || || camera || || || warps || (V0-V2) || || stacks || like (V0-V2) after JT has sync'd remaining ones? || || WSdiff || must wait for MOPS postage stamp requests to finish || || SSdiff || || || refstack || not cleaned || Datastore: || stage || time || || chips || none distributed || || camera || none distributed || || warps || || || stacks || || || WSdiff || || || SSdiff || || || refstack || || == Reprocessed Status == ---- = MD01.GR0 = Started for OC143 (August 13, 2011) with a new reference stack and small sample of reprocessed exposures released as nightly stacks. Due to a need of rebalancing the reprocessing load, only a minimal amount of processing required to make the reference stack was done. Reprocessed sample includes only those exposures that may be used in the reference stack. [[Image(MD01_g_smf.png,200px)]] == MD01.V3 reference stack == * label: MD01.refstack.20110820 * data_group: MD01.refstack.20110820 || '''version label''' || '''filter''' || '''center skycell input warps (rejected)''' || '''center skycell FWHM (major)''' || || MD01.refstack.20110820 V3 || y || 52(1) || 0.84" || || MD01.refstack.20110813 V3 error || y || 106(1) || || || MD01.refstack.20100812 V2 || y || 21(3) || 1.04" || || || || || || || MD01.refstack.20110820 V3 || z || 58(3) || 0.87" || || MD01.refstack.20110813 V3 error || z || 115(2) || || || MD01.refstack.20100812 V2 || z || 60(2) || 1.12" || || || || || || || MD01.refstack.20110820 V3 || i || 74(3) || 1.00" || || MD01.refstack.20100812 V2 || i || 50(0) || 1.13" || || || || || || || MD01.refstack.20110820 V3 || r || 54(7) || 1.10" || || MD01.refstack.20100812 V2 || r || 56(1) || 1.24" || || || || || || || MD01.refstack.20110820 V3 || g || 65(7) || 1.20" || || MD01.refstack.20100812 V2 || g || 54(5) || 1.24" || || || || || || with the approximate FWHM (major) as the PSF FWHM from a center field skycell after running psphot on the stacks. The MD01.refstack.20110813 V3 error was a run without the DEEP_STACK reduction Distributed under ''ps1-md-refstack'' -- stack images only. Troubled skycells: (none for proper run?) * 087-y,z (MD01.refstack.20110813 V3 error): quality issue rejected, edge skycell. * 086-z (MD01.refstack.20110813 V3 error): quality issue rejected, edge skycell. Data volume on disk: Processing time: == MD01.GR0 reprocessed sample == * label: MD01.GR0 * data_group: MD01.GR0.20110812 = y * data_group: MD01.GR0.20110815 = z * data_group: MD01.GR0.20110816 = i * data_group: MD01.GR0.20110817 = r * data_group: MD01.GR0.20110818 = g * dist_group: MD02.GR Distributed under ''ps1-md-GR'' -- currently only a small sub-sample of nightly stack images only. SSdiff sample to be processed still. Data volume on disk: Processing time: ---- = MD02.GR0 = Started for OC144 (September 12, 2011) with a new reference stack and a small sample of reprocessed exposures released as nightly stacks. Due to a need of rebalancing the reprocessing load, only a minimal amount of processing required to make the reference stack was done. MD02 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many are single exposure coverage and won't be enough for a reference stack. [[Image(MD02_g_smf.png,200px)]] == MD02.V3 reference stack == * label: MD02.refstack.20110907 * data_group: MD02.refstack.20110907 || '''version label''' || '''filter''' || '''center skycell input warps''' || '''center skycell FWHM (major)''' || || MD02.refstack.20110907 V3 || y || 58(3) || 1.01" || || MD02.refstack.20101126 V2 || y || 47(3) || 1.02" || || || || || || || MD02.refstack.20110907 V3 || z || 67(5) || 0.90" || || MD02.refstack.20101126 V2 || z || 84(10) || 1.03" || || || || || || || MD02.refstack.20110907 V3 || i || 61(2) || 1.00" || || MD02.refstack.20101126 V2 || i || 102(15) || 1.16" || || || || || || || MD02.refstack.20110907 V3 || r || 63(6) || 1.15" || || MD02.refstack.20101126 V2 || r || 67(15) || 1.10" || || || || || || || MD02.refstack.20110907 V3 || g || 64(6) || 1.32" || || MD02.refstack.20101126 V2 || g || 71(24) || 1.26" || || || || || || with the approximate FWHM (major) as the PSF FWHM from a center field skycell after running psphot on the stacks. Troubled skycells: none really? 040 in g,r oddly seems to have enough coverage to make a stack. Distributed under ''ps1-md-refstack'' -- stack images only. Data volume on disk: Processing time: == MD02.GR0 reprocessed sample == * label: MD02.GR0 * data_group: MD02.GR0.20110902 = y * data_group: MD02.GR0.20110903 = z * data_group: MD02.GR0.20110906 = g * data_group: MD02.GR0.20110905 = r * data_group: MD02.GR0.20110904 = i * dist_group: MD02.GR Distributed under ''ps1-md-GR'' -- sample on nightly stack images only. SSdiff sample still to be run. A typo in the loading script set the workdir to be MD01 rather than MD02 for all filter data groups (see string below) but will not cause any trouble other than the directory name in mirroring data /gpc1/MD02.GR0/MD01.20110906/MD02.V3/skycell.055/*unconv.fits Data volume on disk: Processing time: ---- = MD03.GR0 = In progress. MD03.V3 refstack needed before new observations in OC146 (November 10) MD03 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many of the outer skycells are only covered by single to a few exposure without the 4 or more for a stack. [[Image(MD03_g_smf.png,200px)]] Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack: || ''Filter'' || N total || N (FWHM