= Summary of MD.GR0 = [[PageOutline]] ([wiki:MediumDeepFields] summary page) An overly long summary page of information for the MD GR0 (grand reprocessing version 0) including the new reference stacks in the V3 tessellation with the start of new MD10 observations in OC142 (August 2011). This is version 0, the first start of full reprocessing of the MD fields and must be balanced along with other reprocessing, 3PI stacks, nightly science, generation of new reference stacks etc. Processing has also extended over a long period of time with many starts and stops for various reasons, thus, has become a very inhomogeneous set with code and labeling improvements as it progressed. A best attempt at documenting this has been done field by field. The new ubercal reference catalog were available starting with MD08 (with PS1.REF.20120325) and MD09 (with a revised REF.20120524), but all others (MD10, 01, 02, 03, 04, 05, 06, 07) were ''originally'' processed using the original synthetic catalog. MD01,02,03,05 were not fully reprocessed originally due to various problems, so those fields will have a mix of exposures calibrated with different catalogs. There has continued to be limited reprocessing of MD exposures in order to create new refstacks for the season so there can be exposures with multiple reprocessed versions. Creation of the new reference stacks need to be, at the very least if possible, made a week prior to the return to the particular MD field given in the current schedule http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Medium_Deep_Schedule. After the July, August, September time crunch of a new field a month, there should be time to continue reprocessing and creation of new reference stacks in advance. Where to find the data: * http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR/ (full: pixel images of reprocessed nightly data products like night stacks) * http://datastore.ipp.ifa.hawaii.edu/ps1-md-GR-cat/ (cleaned: catalogs of reprocessed nightly data products like camera SMFs) * http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/ (full: pixel images for updated refstacks and reftest stacks) * http://datastore.ipp.ifa.hawaii.edu/ps1-md-staticsky/ (cleaned: stack photometry staticsky catalogs, run on the refstacks typically but may have others like from deep stacks) * http://datastore.ipp.ifa.hawaii.edu/ps1-deeptest/ (full: pixel images of deep stack tests) * Postage Stamp Server: http://pstamp.ipp.ifa.hawaii.edu/pshome.php == Status == * ''8/15/11'' minimum mode re-processing for reference stacks after MD10 for evaluation and get next MD field ready (MD01) * ''9/30/11'' more success weaving in reprocessing with LAP, started reprocessing for refstacks MD03,04 (with more reprocessed exposures) * ''10/09/11'' MD03 refstack and sample stacks available, testing MD04 ~200 very deep stack, MD05 running * ''10/17/11'' MD07 chip-->warp processing started with condor, using any extra cpu cycles. * ''11/06/11'' MD07 refstacks being made with order ~120 inputs. MD04 deepstack testing continued, gri sample released, zy on hold until fixes made. * ''11/18/11'' MD06 chip->warp processing running with condor, refstack preparation (using new ipp-20111110 ops tag now) * ''11/25/11'' MD06 reftests made and y-band refstack started, MD08 chip->warp processing via condor, refstack preparation * ''12/02/11'' MD06/08 on hold, data uncertain with ipp064 down. * ''12/20/11'' MD06 processing recoverable, refstacks proceeding. MD08 a mess and will need to be re-run. * ''12/31/11'' MD06 finally finished, distributed. * ''1/09/12'' MD08, 09 on hold while finishing up full reprocessing and release of V3 products now (i.e., night stacks, SMFs, SSdiffs). * ''1/11/12'' MD06,04 full camera SMFs on datastore, MD07 later today. MD04 set has some duplicates with both unmagicked SMFs (data_group=MD04.GR0, sub-sample of ~150/filter best seeing) and magicked SMFs (data_group=condor_MD04.V3_01_dg, full sample of ~400/filter loaded into PSPS Nov. 2011) * ''1/11'' full V3 night stack reprocessing with MD06 started. -- important note: these are now being run allowing 2 or more inputs to make a stack (down from 4 during the magic era) so outer skycells may have strange behavior that we need feedback on. * ''1/17'' majority of MD06 full reprocessing of night stacks on datastore. MD04 full V3 night stack reprocessing started. * ''1/23'' MD04.V3 full reprocessing of night stacks finished and pushed out to datastore. * ''4/17'' MD08.V3 restarted from scratch after raid crash in November/December with the new ipp-20120404 tag that will include new ubercal refcat (PS1.REF.20120325) and detrend improvements (dark modes, noise map, but not boost factors). * ''4/25'' MD08.GR0 reprocessed and distributed, refstacks made and distributed, staticsky run and distributed. * ''6/01'' MD09.GR0 chip->warp reprocessing started with new ops tag ipp-20120531 (improved REFCAT 20120524, ghost models, corner under-glow masking). seeing some trouble with new tag in z,i,r for XY67,71. * ''6/03'' MD09 g,y refstack started, night stack reprocessing running. * ''6/04'' MD09 z refstack started. g,y refstack and reftests and night stack pushed to distribution. * ''6/06'' MD09 r,i refstacks started. z refstack, 90% night stacks out to datastore. * ''6/07'' MD09 refstacks finished and distributed, staticsky on refstack finished, 90% of camera SMF and night stacks that are possible distributed. * ''6/18'' MD07 (pre-ubercal) staticsky run attempted, see notes below * "7/05" new MD10 refstack started, see [wiki:MD10.refstack.20120705] * "7/29" MD09.GR0 XY67,71 chip problem (~48 exposures) not fixable so set to poor quality, remaining camera SMF and night stacks processed and distributed. * "8/03" new MD01 refstack started, like new MD10 more of continuation of GR0 (separate wiki pages) -- [wiki:MD01.refstack.20120803] * 08/31 MD09.refstack.20120603 replaced with MD09.refstack.20120831 (run with ipp-20120802 tag) to fix the weight pixel arc region of lower sensitivity * 09/27 [wiki:MD02.refstack.20120927] to replace 2011 season refstack * 11/01 [wiki:MD03.refstack.20121101] to replace 2011 season refstack * 11/25 [wiki:MD04.refstack.20121125] to replace 2011 season refstack * 12/06 [wiki:MD05.refstack.20121202] to replace 2011 season refstack * 12/18 final set of exposures setup for reprocessing to finish MD01,02,03 and produce the corresponding night stacks. MD05 finished earlier. * 12/21 [wiki:MD06.refstack.20121221] to replace 2011 season refstack * 01/09/13 [wiki:MD07.refstack.20130102] to replace 2011 season refstack * 01/14 re-running all staticsky+skycal stack photometry on ref+deepstack to incorporate improvements/fixes to Kron code for loading into PSPS soon * 01/28 finished revised staticsky+skycal version except for MD03,05 which are blocked by disk upgrade/permission issues * 02/06 added large number of updates to page and plots, primarily on MD09 and stack photometry -- reprocessing effectively finished -- == Summary plan for finishing MD.GR0 through end of 2012 == (revised Jan. 30, 2013) || '''MD''' || '''Reference Stack''' || '''Deep Stack''' || '''Night Stack''' || '''Refstack Phot''' || '''Deep Phot''' || '''Night Phot''' || '''Camera/SMFs''' || '''for PSPS''' || || || || || || || || || || || || 01 || 20110820 (U) || X || done (mixed) || .20120803.Nx20130115, .cal20130128 || x || x || done (mixed) || SMF+ref phot ''TBD'' || || 02 || 20110907 (U) || X || done (mixed) || .20120927.Nx20130115, .cal20130128 || x || x || done (mixed) || SMF+ref phot ''TBD'' || || 03 || 20111010 (U) || X || done (mixed) || .20111010.Nx20130115, .cal20130128 (I) || x || x || done (mixed) || SMF+ref phot ''TBD'' || || 04 || 20111012 (U) || deeptest.20120727 || done || .20111012.Nx20130115, .cal20130128 || .Nx20130114, .cal20130128 || ''TBD'' || done || SMF+ref+deep+night phot ''TBD'' (secondary) || || 05 || 20111014 (U) || X || done (mixed+full) || .20111014.Nx20130115, .cal20130128 (I) || x || x || done (mixed) || SMF+ref phot ''TBD'' || || 06 || 20111122 (U) || X || done || .20111122.Nx20130115, .cal20130128 || x || x || done || SMF+ref phot ''TBD'' || || 07 || 20111106 (U) || X || done || .20111106.Nx20130115, .cal20130128 || x || x || done || SMF+ref phot ''TBD'' || || 08 || 20120422 || X || done || .20120422.Nx20130115, .cal20130128 || x || x || done || SMF+ref phot ''TBD'' || || 09 || 20120831 (R) || deeptest.20120727 || done || .20120831.Nx20130115, .cal20130128 || .Nx20130114, .cal20130128 || .1x20120926rawtest, .cal20130128 || done || SMF+ref+deep+night phot (primary) || || 10 || 20110821 (U) || X || done || .20120804.Nx20130115, .cal20130128 || x || x || done || SMF+ref phot ''TBD'' || || 11 || not reprocessed in MD.GR0 || || || || || || || || || || || || || || || || || || || || || || || || || || || || || || || M31 || briefly discussed, but likely no || STS also likely no || 20120531 (MOPS needs revision) || X || ? || ? || ? || ? || ? || ? || || || || || || || || || || __Notes__: * TBD=To Be Decided/Done by priority, X=not planned, ?=unknown at this time * 20YYMMDD date entries are part of the data_group name to access via the PSS or Datastore, i.e., * stacks -- MD01.refstack.20110820, MD09.deeptest.20120727 * night stacks -- a mix of label, data_group formats, see specific section * stack photometry -- originally the same date 20120805, but revised runs now have dates for each in the form adding for staticsky .<#>x20YYMMMDD and skycal .cal20YYMMDD. * (U) updated -- new 2012/2013 versions exist for these refstack now (see [wiki:MediumDeepFields]) but not necessarily used for stack photometry (staticsky/skycal) for loading into PSPS (original MD.GR0 may be deeper and better to use). The refstack used is indicated in the Stack Photometry column. All updated refstacks of course use the ubercal catalog for calibration. * (R) replaced -- MD09.refstack.20120603 replaced by 20120831 due to variance problems. * (I) incomplete -- for MD03, 05 stack photometry stalled while data disks being shuffled.. * (mixed) * finish reprocessing all exposures to V3 for warp/night stack for MD01+02+03+05 =840+609+1286+1115 = 3850 (2-3 full days of --only-- MD processing) -- finished 12/30/13 * (mixed) -- both camera/SMF and night stacks can have calibrations based on the synthetic catalogs (initial sample run) and ubercal catalogs (finish V3 run). See specific section for detail. * (mixed+full) -- MD05 -- some older and newer processing, so mix of magic and photometry catalog used and then a re-reprocessing of MD05 was done on the exposures originally processed with synthetic catalog. See [wiki:MD.GR0#MD05.GR0extrareprocessedsetwithallubercalcatalog] * stacks * out-of-season/anti-yearly refstacks, yearly refstack: on hold for MD.GR1 (would need to update/reprocess large numbers of exposures to do). * iq/psf stack on hold until MD.GR1 (like out-of-season, would required many updates) -- should do for MD04,MD09 and requires development/code time. * deep stack for MD04,09 for MD.GR0 only, all others on hold until MD.GR1 (many many updates would be necessary..). * reminder that ubercal of pixels start with MD08 in the date sequence, finished processing of MD01,02,03,05 exposures into V3 will also be ubercal'd (including reprocessing of any needed to make the complete night stacks) __Questions__: * background restored warps/stacks? * SMF loaded into PSPS limited to just the ones used in the refstack (or deepstack if available) or all available exposures Camera/SMFs? * MD01,10 have new refstack for this season, need to decide if using for phot+PSPS (limited set of exposures reprocessed recently to make)? Or use as update test for PSPS? * MD09.refstack.20120603 suffers from the weight image problem making an arc of lower sensitivity, but warps are still available for doing the deep stacks. 4-6 hrs/skycell to run, ~300 skycells, deepstack pantasks typically has 30 nodes, so ~2-3 days processing time, then redo? -- yes, since MD09 is a new test/comparison set with SAS -- done and date updated under refstack as MD09.refstack.20120831 == Feedback is important == We would not want to continue full reprocessing of the MD fields if there has been a gross oversight or problem for a particular science interest. Please let us know of any issues you discover with example details. For example * tessellation and/or offsets issues in any of the fields * corner of skycell.055 seems to be regularly masked (and in neighbors with the overlap) * fields with offsets need to include sufficient images in old and new pointings (i.e., MD10), for the magic reference stack at least. * any issues with masking, mask values * been report of possible ones in early MD10 but still need details * depth/quality of the stacks, reprocessed images in the new format * any missing skycells, SMFs, other data products -- in particular MD10 which has been the only MD field full reprocessed and magicked for release of SMFs, warps. * MD10.refstack.20110722, MD10.refstack.20110821 r-band has two edge skycells (041,051) with poor quality and failed * MD10.refstack.20110722 y-band skycell.048 had rejection problem, fixed with MD10.refstack.20110821 * MD10.refstack.20110722 replacement with proper uncompressed MD10.refstack.20110821 for diffims * with magic turned off now (01/2012), MD04+MD06 have 2+ warp inputs for night stack creation to test how well boundaries come out and has fault/quality issue with 2-3 input cases showing some inputs have incorrect PSF size values measured/passed. rare (~<1%) but should be looked into (ie stack_id=609561) * adding a quality column to ippMonitor for evaluating missing stack skycells would be useful to have on ''stack processed skyfile'' DRAVG analysis/feedback * [http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/DRAVG_telecon_2011#Medium_deep_stacks MD04 number counts] * [http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/DRAVG_telecon_2012#MD09_issues MD09 issues] * [http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/DRAVG_telecon#Medium_Deep_Oddities MD oddities (Jan. 14, 2013)] * [http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/DRAVG_telecon#Overdetection_in_the_y-band_MD_stacks MD-y overdetections (Jan. 28, 2013)] * [http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/DRAVG_telecon#False_detections_in_the_y-band_MD_stacks MD-y false detections (Jan. 28, 2013)] == MD.V3 Tessellation == The new V3 tessellations are defined at [wiki:PS1_Tessellations] with the following general properties: * 0.25"/pix (from natural ~0.258"/pixel, previously warped to 0.2"/pix up through V2) * 10 x 10 skycell grid (previously 12 x 12) * +x is West (flipped from before +x was East), +y is still North * 320 pixel (80") overlap (from 300 pixels, 60", in V2) * revised centers (if necessary) -- see relevant MD section for more details and the MDS field table (http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Medium_Deep_Survey#Actual_Field_Centeres_-_2011-02-21) but in summary * [wiki:MD.GR0#MD10.GR0 MD10.GR0]: from (351.979 -0.433) to (352.312, -0.433) however an additional offset looks like it should have been applied to cover more of the earlier observation dithers. * [wiki:MD.GR0#MD07.GR0 MD07.GR0]: from (213.146, +53.417) to (213.454, +53.243) to best cover as much of the earlier observation dithers as possible. * [wiki:MD.GR0#MD08.GR0 MD08.GR0]: very slight offset in RA to the W to just align with the MDS field pointing. * [wiki:MD.GR0#MD09.GR0 MD09.GR0]: from (333.688, +0.283) closer to the current pointing of (334.188, +0.283) however an addition offset was added to move the center about half a skycell to the E to (333.980, +0.283) in order to cover as much of the earlier observation dithers as possible. * All fields should be looked at to see if there are any problems with the new tessellation and dither pattern. * [wiki:MD.GR0#MD07.GR0 MD07], [wiki:MD.GR0#MD09.GR0 MD09] will have a preliminary night stack made for testing the revised center as well as [wiki:MD.GR0#MD06.GR0 MD06] for NGC4258. == Reference versus Deep Stack == A requirements description for the needs in the different types of stacks would be useful to summarize. Reference stacks -- primarily for difference imaging for transient detection and necessary for magic destreaking. * Magic destreaking: a fairly clean reference stack (uses the convolved stack) with as much spatial area coverage as possible to enable release of non-stack data products. Particularly important is inclusion of older data on different pointings and of course a fairly balanced dither set. * Transient detection: a balance slanted towards best PSF but with some depth (1-1.5 magnitudes over night stacks) and also a fairly balanced dither set. better with more recent data to limit AGN/variable star variations. Deep stacks -- for deep static sky science. * As deep as possible.. maximum FWHM allowance? minimum exposure time? specific request for at least ~10ks has been requested, with best FWHM possible and also a fairly balanced dither set.. See summary for each MD field reference stack for details on specifics for that field. Due to processing+time constraints/early mistakes, an attempt has been made to merge the similar criteria under the ''refstack'' label. Since magic destreaking uses the convolved stack, looking into a quickstack version of the unconvolved stacks for the transients reference and deep stack that will be faster to make with more inputs. For cases when a stack does not have a good sample of exposures with a reasonably small PSF FWHM, it should be possible to generate a reference stack with the fewer 'best' exposures for just SSdiffs and transient detection. MD02-g may be such a case. For fields with shifted centers from offset earlier exposures, inclusion of enough exposures to provide a reference stack for magic is necessary in order for data from those associated chips/skycells to be distributed (being older data, may not have the optimum quality). Mixing older and newer data may also be a detriment. Related to deep stacks, there needs to be enough inputs to obtain minimum depth that is as pixel contiguous as possible (see MD04 deep stack tests). How much of a varying IQ can go into a stack before ppSub has difficulty modeling the variations within a skycell? Deep stacks on hold until finish investigating overly large PSF in the convolved stacks dependent on the number of input warps. Without need for magic for releasing older data, should focus on more recent data going into the refstacks == post-Magic January 2012 == * some (de?)processing had been done to revert the magicked SMF catalogs back to their full form, status and completeness unknown. ---- = Reprocessed Data Products = Each MD field has a variety of ''issues'' and details are summarized in each MD section. Label/data_group names have also evolved and the appropriate MD section should be referred to. == Chip+Warp == Currently processed but not distributed due to disk space load. Can be obtained via the Postage Stamp Server. * can be a mix of synthetic/ubercal catalog calibrations except MD08, 09. Generally if the chip_id, warp_id >= 433987, 393977 respectively it was processed using the 20120524v0 ubercal catalog. == Camera/SMFs == Fully processed and distributed. * can be a mix of synthetic/ubercal catalog calibrations except MD08, 09 * a possible guideline to follow, including nightly science data processing, if MD exposures with a cam_id >=412069 (actually >=411444 of some V2 destined reprocessing) is used, those will be with the ipp-20120404,r33642 tag that includes the faint-end bias fix and the ubercal catalog 20120524v0 version also used for LAP. == Stacks == Generally, if a stack_id is >= 848641 then it was ''likely'' made from processed exposures using the 20120524v0 ubercal catalog (see night stacks where some were made using old warps if a complete set existed for MD01,02,03,05). __Night Stacks__: All nights reprocessed and stacked into a night stack using the current night stack configuration used for nightly science night stacks when processed, including reference catalog * should attempt to make stack with 2 or more input warps now * some nights have >8 exposures, often in marginal conditions. Since marginal conditions can vary greatly, some selection should probably be made for inclusion in the night stack. * the sample of stacks can be a mix of synthetic/ubercal catalog calibrations for the pixels but the warps going into the stack should be of the same type of calibration. __Reference Stacks (refstack)__: To be made prior to the start of the MD field observations for the season to support the difference image products (magic and science). The focus of MD.GR0 was on the 2011/2012 season fields starting with MD10, then 01, 02 ... through MD09. The 2012/2013 season refstacks will to replace the 2011/2012 set and have their own page linked from [wiki:MediumDeepFields]. * out-of-season/anti-yearly refstacks, yearly refstack: on hold for MD.GR1 (would need to update/reprocess large numbers of exposures to do) __Deep Stacks (deeptest)__: Not a full production product yet. Test version exists for MD02.iqtest.20120927, MD10.psfrefstack.20110814. __IQ/PSF Stack__: on hold until MD.GR1 (like out-of-season, would required many updates). A sample for MD09 done but requires development/code time. == Difference Images == Originally a possible reprocessing product, but was decided not useful at this time (for future reprocessing, need to outline interests for these if any). Will need to finish the convolution choice (input stack or reference stack) before started. == Stack Photometry (Static Sky + Skycal) == Stack photometry (staticsky) and recalibration (skycal) is run on refstacks, deepstacks and night stacks. In trying to run as a uniform group, will keep a summary here, but details like specific faults, poor quality etc will be in the respective MD section when time. For example some skycells can have a clear offset fix to the zeropoint, not actually set to 25.0 in stack pixels, for some the majority of the field is off from zeropoint of 25. '''There appears to be upwards of 3-5% error likely in the photometry across the FPA''' (see [wiki:MD.GR0#MD09.GR0]) even in the recalibration of the stack photometry (skycal stage). A much worse example is seen in [wiki:MD01.refstack.20120803]. * '''recalibration of the stack photometry catalogs has no method for distribution''' -- email for access Current run planned for loading into PSPS (January 2013): stack photometry (staticsky) re-run from Summer 2012 set to incorporate improvements/fixes in the Kron code. Photometry zeropoint recalibration (skycal) re-run to incorporate various bug fixes and improved tuning (psastro running on stack catalogs). Several previous re-runs for skycal and settled on .cal20130128 version for now. * ref stack label, data_groups: includes 5-1 filter sets now under same label, staticksky data_group will indicate for # filters attempted, skycal data_group will include all under one data_group {{{ label staticsky: MD09.refstack.20120831.Nx20130115 -- run on refstack data_group MD09.refstack.20120831, label for active refstacks is changed to MD<##>.refstack generically data_group staticsky: MD09.refstack.20120831.x20130115 -- = 1,2,3,4,5 and number of filters available for the set, so 5 different data_group per MD label=data_group skycal: MD09.refstack.20120831.Nx20130115.cal20130128 -- all N filter set is run under one data_group now per MD label staticsky: MD04.refstack.20111012.Nx20130115 data_group staticsky: MD04.refstack.20111012.x20130115 label=data_group skycal: MD04.refstack.20111012.Nx20130115.cal20130128 label staticsky: MD10.refstack.20120804.Nx20130115 (new refstack MD10.refstack.20110821, see [wiki:MD10.refstack.20120705], is deeper and replacing MD.GR0 version MD10.refstack.20110821) data_group staticsky: MD10.refstack.20120804.x20130115 label=data_group skycal: MD10.refstack.20120804.Nx20130115.cal20130128 label staticsky: MD01.refstack.20120803.Nx20130115 (new refstack [wiki:MD01.refstack.20120803] replacing MD.GR0 version since deeper) -- skycal label=data_group, add .cal20130128 label staticsky: MD02.refstack.20120927.Nx20130115 (new refstack [wiki:MD02.refstack.20120927] replacing MD.GR0 version since deeper) -- skycal label=data_group, add .cal20130128 label staticsky: MD03.refstack.20111010.Nx20130115 -- **** stalled while file permissions messed up **** label staticsky: MD05.refstack.20111014.Nx20130115 -- **** stalled while file permissions messed up **** label staticsky: MD06.refstack.20111122.Nx20130115 -- skycal label=data_group, add .cal20130128 label staticsky: MD07.refstack.20111106.Nx20130115 -- skycal label=data_group, add .cal20130128 label staticsky: MD08.refstack.20120422.Nx20130115 -- skycal label=data_group, add .cal20130128 }}} * deep stack label, data_groups: includes 5-1 filter sets now under same label, staticsky data_group will indicate N-filter, skycal data_group all under one name {{{ label staticsky: MD<##>.deeptest.20120727.Nx20130114 -- <##> = 04, 09 only made so far data_group staticsky: MD<##>.deeptest.20120727.x20130114 -- = 1,2,3,4,5 and number of filters available for the set, so 5 different data_group per MD label=data_group skycal: MD09.deeptest.20120727.Nx20130114.cal20130128 -- all N filter set is run under one data_group now per MD label=data_group skycal: MD04.deeptest.20120727.Nx20130114.cal20130128 -- all N filter set is run under one data_group now per MD }}} * night stack label, data_group: only skycal needs to be rerun on problem skycells {{{ -- staticsky label = data_group: MD09.GR0.20120602.nightstack.1x20120926rawtest -- skycal label = data_group: MD09.GR0.20120602.nightstack.1x20120926rawtest.cal20130128 }}} Previous run (summer/fall 2012): labels all changed to the data_group to avoid conflict with active set * ref stack labels, data_groups: only included skycells with full 5-filter set {{{ label = data_group staticsky: | MD10.refstack.20110821.5x20120802 | MD10.refstack.20110821.5x20120802 | | MD01.refstack.20110820.5x20120802 | MD01.refstack.20110820.5x20120802 | | MD02.refstack.20110907.5x20120802 | MD02.refstack.20110907.5x20120802 | | MD03.refstack.20111010.5x20120802 | MD03.refstack.20111010.5x20120802 | | MD04.refstack.20111012.5x20120802 | MD04.refstack.20111012.5x20120802 | | MD05.refstack.20111014.5x20120802 | MD05.refstack.20111014.5x20120802 | | MD06.refstack.20111122.5x20120802 | MD06.refstack.20111122.5x20120802 | | MD07.refstack.20111106.5x20120802 | MD07.refstack.20111106.5x20120802 | | MD08.refstack.20120422.5x20120802 | MD08.refstack.20120422.5x20120802 | | MD09.refstack.20120603.5x20120802 | MD09.refstack.20120603.5x20120802 | | MD10.refstack.20120705.5x20120802 | MD10.refstack.20120705.5x20120802 | | MD09.deeptest.20120727.5x20120806 | MD09.deeptest.20120727.5x20120806 | | MD04.deeptest.20120727.5x20120806 | MD04.deeptest.20120727.5x20120806 | label=staticsky skycal: add .cal20120802 }}} * deep stack label, data_groups: only included skycells with full 5-filter set, <##> = 04, 09 only {{{ label = data_group staticsky: MD<##>.deeptest.20120727.5x20120806 label = data_group skycal: MD<##>.deeptest.20120727.5x20120806.cal20120806 }}} * night stack label, data_group: only running for MD09 {{{ -- staticsky label = data_group: MD09.GR0.20120602.nightstack.1x20120926rawtest -- skycal label = data_group: MD09.GR0.20120602.nightstack.1x20120926rawtest.cal20120927 }}} New .cal20130128 has some bug fixes in the rerun has made some clear improvements in stack photometry uniformity (MD10.refstack.20120804, z-band, difference with matched refcat magnitudes with scaling changed to account for the mag ranges, 1/28/13) : || [[Image(md10v20120804x1skycal_cmf_z.txt_drefmagptmap.png,300px)]] || [[Image(md10refNx20130128skycal_cmf_z.txt_drefmagptmap.png,300px)]] || == Clean-up Cycle == IPP datadisks -- can be remade/distributed via Postage Stamp Server if cleaned, faster if not cleaned || '''stage''' || '''time-frame''' || || chip || once all warps finished || || camera || not cleaned || || warps || once all stack finished || || stacks (ref,deep,night) || not cleaned || || diffims (WS, SS) || not made || || stack photometry (staticsky) || not cleaned || * MD04, MD09 warps available still to make different stacks for development Datastore -- standard distribution method of large sets of data bundles, not an archive. Specified ones can be remade/distributed via Postage Stamp Server. || '''stage''' || '''time-frame''' || || chips || not distributed || || camera || pixel/masks not distributed, catalogs/SMFs in ps1-md-GR-cat not cleaned || || warps || not distributed || || stacks (ref,deep,night) || ref not cleaned, deep not cleaned, night similar to nightly processing ~30 days for pixels || || diffim (WS,SS) || not made/distributed || || stack photometry (staticsky) || not cleaned for now || * '''discovered that no clean-up of Datastore distribution dataproducts being cleaned up (12/2012) -- starting 3/2012 may need to start cleaning up pixel bundles from the GR datastore >1 year old''' = Stack Result Summary Posters at PS1SC 2012 Durham Meeting = Posters include summary map plots of psf_major(minor), 5-sig photometric depth and magnitude @50% recovery of 500 added fakes as well as full field, i-band images of the MD refstacks * http://ipp.ifa.hawaii.edu/mdgr0_durham2012poster/mdgr0_md01_03_posterdurham2012.pdf * http://ipp.ifa.hawaii.edu/mdgr0_durham2012poster/mdgr0_md04_06_posterdurham2012.pdf * http://ipp.ifa.hawaii.edu/mdgr0_durham2012poster/mdgr0_md07_09_posterdurham2012.pdf * for MD10 old/new refstack comparison maps, see [wiki:MD10.refstack.20120705] ---- = MD10.GR0 = ''Status:'' * reference stack (revised) processed and distributed. All exposures since April 2009 reprocessed and distributed except for SSdiffs. SSdiffs being held back until ppSub auto convolution choice is tested on. * 8/2012: to be replaced with [wiki:MD10.refstack.20120705], then replaced with MD10.refstack.20120804. See new refstack wiki for old/new refstack comparison maps. Started for OC142 (July 15, 2011). Start with current statuswith a new reference stack and full sample of reprocessed exposures as an initial reprocessing version. * z,y-band data processed prior to full Dark model improvements (higher background makes effect insignificant). gri-band processed after Dark model improvements. * same photometry reference catalog as MD have previously been using. * * stack-stack diffims still using forced convolution of reference stack to input images MD10.V3 center shifted to the current observation pointing even though there are earlier data that has offset dithered observations that populate the 0X8-0X9 skycells. Not as bad as MD07, but the revised reference stack may not provide for the outer skycells very well. Afterwards it was discovered the full extent of the previous observations wasn't quite accounted for, shown by these exposure plots with the MD10.V3 overlay and compared to a recent night stack set of exposures. This may need to be shifted in the next iteration. || [[Image(MD10_g_smf.png,150px)]] || [[Image(MD10_r_smf.png,150px)]] || [[Image(MD10_i_smf.png,150px)]] || || [[Image(MD10_z_smf.png,150px)]] || [[Image(MD10_y_smf.png,150px)]] || [[Image(MD10ns_i_smf.png,150px)]] || Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack: || '''Filter''' || '''N total''' || '''N (FWHMwarp including camera SMF for all filters grizy: * label: MD01.GR0.20121122 * data_group: MD01.GR0.20121122 Night stack for all filters grizy:: * label: MD01.GR0 * data_group: MD01.GR0.20121122.nightstack. * data_group if warps processed earlier but stack made with new tag: MD01.GR0.201211xx.nightstack. ---- = MD02.GR0 = ''Status:'' * Refstack and sample night stacks finished and distributed. * 10/5/2012: to be replaced with [wiki:MD02.refstack.20120927] -- diffims 20121001 onwards will use. Started for OC144 (September 12, 2011) with a new reference stack and a small sample of reprocessed exposures to release night stacks as well. Due to a need of rebalancing the reprocessing load, only a minimal amount of processing required to make the reference stack was done. || MD02 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many are single exposure coverage and won't typically be enough for a reference stack skycell. While warps will extend to some of the outer skycells, the night stack distribution will define what is likely for the skycells released due to magic. 012-017, 21-28, 31-38, 41-48, 51-58, 61-68, 71-78, 82-87 || [[Image(MD02_g_smf.png,200px)]] || Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack: || '''Filter''' || '''N total''' || '''N (FWHMwarp including camera SMF for all filters grizy: * label: MD02.GR0.20121210 * data_group: MD02.GR0.20121210 Night stack for all filters grizy:: * label: MD02.GR0 * data_group: MD02.GR0.20121210.nightstack. * data_group if warps processed earlier but stack made with new tag: MD02.GR0.201212xx.nightstack. ---- = MD03.GR0 = ''Status:'' refstacks and sample night stacks finished and distributed * 10/3: Sub-sample of y and z-band exposures through warp, sample night stacks made. y-band refstack in progress. * 10/4: r,i through warp. z-band refstack in progress * 10/6: g through warp, ri refstack in progress * 10/8: all refstacks queued for distribution, night stack sample as well. * 11/07/2012: to be replaced with [wiki:MD03.refstack.20121101] -- diffims 20121026 onwards will use this MD03.V3 refstack needed before new observations in OC146 (November 10). || MD03 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many of the outer skycells are only covered by single to a few exposure without the 4 or more for a stack. || [[Image(MD03_g_smf.png,200px)]] || Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack: || '''Filter''' || '''N total''' || '''N (FWHMwarp including camera SMF for all filters grizy: * label: MD03.GR0.20121212 * data_group: MD03.GR0.20121212 Night stack for all filters grizy:: * label: MD03.GR0 * data_group: MD03.GR0.20121212.nightstack. * data_group if warps processed earlier but stack made with new tag: MD02.GR0.201212xx.nightstack. ---- = MD04.GR0 = ''Status:'' Refstacks and sample night stacks finished and distributed. Very deep stack tests in progress. Full reprocessing of all possible night stacks finished and to datastore. * 10/2 Starting with i-band rather than y for the sub-sample reprocessing exposures through warp. * 10/4 z-band through warp done. testing very deep >150 stack on i-band. r-band running through warp * 10/5 gy running through wrap * 10/6: izrgy refstacks ready to queue * 10/8: i-band refstack finished, z-band running * 10/9: r-band running, g-band running * 10/10: remaining y refstack finishing up * 10/16: refstack and sample night stack set finished and distributed * 01/17/12: full reprocessing of all possible night stacks in V3 started (finished 1/23 and to datastore) * 11/25/2012: refstack to be replaced with [wiki:MD04.refstack.20121125] -- diffims onwards will use this New observations aren't planned until OC147 (December 9), but need it for comparison analysis ASAP. Retired MD04.altstack in a pre-V3 tessellation from last November (20101102). Pre-V3 MD04.altstack warps have been purged (no longer updatable) but the stacks are still available for any comparison. || All past g-band exposures in the V3 tessellation (other filters nearly identical). || [[Image(MD04_g_smf.png,200px)]] || Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack: || '''Filter''' || '''N total''' || '''N (FWHM<8 pixels)''' || '''N (FWHMwarp (including camera) and night stack sample: * label: MD04.GR0 * dist_group: MD04.GR * data_group: MD04.GR0.20111002 = i * data_group: MD04.GR0.20111003 = z * data_group: MD04.GR0.20111004 = r * data_group: MD04.GR0.20111005 = g * data_group: MD04.GR0.20111006 = y Distributed on datastore under * ps1-md-GR: sample of night stack images only. SSdiff sample still to be run. * ps1-md-GR-cat: unmagicked camera SMFs with data_group MD04.GR0 released (dupe to the full magicked SMF release from condor processing that was magicked for PSPS loading Nov. 2011) Heather/Serge successful reprocessing of chip-warp under condor: (more to be run with similar label and condor) * label: condor_MD04.V3_01 * data_group: condor_MD04.V3_01_dg * dist_group: MD04.GR Remaining nightly stack data_group: data_group of remaining sample of nightly stacks processed with ipp-20111222 tag and now allowing 2 or more input warps necessary to make stack (down from 4 during the magic era and typically the far outer skycells, but as a new case, will need to be looked at in more detail). Also added ''nightlystack.'' to make a group by night the observation made to help in tracking/monitoring (there will be a mix of data_groups with above MD04.GR0.201110xx earlier sample version) * data_group: MD04.GR0.20120117.nightlystack. = y * data_group: MD04.GR0.20120118.nightlystack. = z * data_group: MD04.GR0.20120119.nightlystack. = i * data_group: MD04.GR0.20120120.nightlystack. = r * data_group: MD04.GR0.20120121.nightlystack. = g * workdir: will now be under a similar directory gpc1/MD04.GR0/MD04.GR0.201201xx.nightlystack. Distributed on datastore under * ps1-md-GR: remaining nightly stacks to be processed from the condor_MD04.V3_01_dg data_group. SSdiff sample still to be run. * ps1-md-GR-cat: magicked camera SMFs released (data_group condor_MD04.V3_01_dg, set used for PSPS loading Nov. 2011) Staticsky run for ingestion into PSPS. Eric notes the RA_PSF, DEC_PSF are NULL in the CMFs. When loaded into DVO coordinates are set and will be okay in PSPS. -- should be fixed in later re-run of stack photometry (Aug. 2012) ---- = MD05.GR0 = ''Status:'' Refstacks and sample night stacks finished and distributed. * 10/7: y-band chip-warp setup. * 10/8: yzi chip-warp run * 10/9: rg chip-warp run, yzi sample night stacks setup * 10/10: sample night stacks yzi finished, gr started, y refstack started * 10/16: refstack and sample night stack set finished and distributed * 10/05/2012: restarted processing remaining exposures into V3 -- will be somewhat a mess * 10/13: night stacks for remaining exposures finished (important to see details below). * 12/06/12: new refstack [wiki:MD05.refstack.20121202] will replace this one for diffims in the 2012/2013 season New observations planned and started OC147 (December 9). || All past g-band exposures in the V3 tessellation (other filters nearly identical). || [[Image(MD05_g_smf.png,200px)]] || Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack: || '''Filter''' || '''N total''' || '''N (FWHM<8 pixels)''' || '''N (FWHMwarp reprocessing from 20111007,08,09,10,11 (pixels calibrated with '''synth catalog'''): * label: MD05.GR0 * data_group: MD05.GR0.201210xx.nightstack. = grizy Remaining set of chip-warp reprocessing, including camera SMFs (reprocessed exposures with '''ubercal catalog and no magic'''): * label: MD05.GR0.20121005 = grizy * data_group: MD05.GR0.20121005 = grizy Night stacks with new chip->warp reprocessing (IPP tag ipp-20120802 and pixels calibrated with ubercal catalog) * label: MD05.GR0 * data_group: MD05.GR0.20121005.nightstack. = grizy Night stack faults: {{{ select stack_id,label,data_group,fault,filter,skycell_id from stackRun join stackSumSkyfile using (stack_id) where data_group like "MD05.GR0.2012%" and fault>0; | stack_id | label | data_group | fault | filter | skycell_id | +----------+----------+---------------------------------------+-------+---------+-------------+ | 1591991 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20100210 | 2 | i.00000 | skycell.045 | | 1584473 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110221 | 5 | z.00000 | skycell.012 | | 1592150 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110412 | 5 | i.00000 | skycell.021 | | 1583687 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110317 | 5 | y.00000 | skycell.028 | | 1584765 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091216 | 5 | z.00000 | skycell.060 | | 1585192 | MD05.GR0 | MD05.GR0.20121005.nightstack.20100211 | 5 | z.00000 | skycell.017 | | 1586381 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110612 | 5 | z.00000 | skycell.012 | | 1587202 | MD05.GR0 | MD05.GR0.20121005.nightstack.20100107 | 5 | r.00000 | skycell.060 | | 1589331 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110610 | 5 | r.00000 | skycell.071 | | 1589629 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091126 | 5 | g.00000 | skycell.059 | | 1591492 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110210 | 5 | g.00000 | skycell.021 | | 1592334 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090405 | 5 | i.00000 | skycell.028 | | 1593532 | MD05.GR0 | MD05.GR0.20121005.nightstack.20100601 | 5 | i.00000 | skycell.021 | | 1593685 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110109 | 5 | i.00000 | skycell.071 | | 1593817 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110124 | 5 | i.00000 | skycell.087 | }}} Night stack quality failures: {{{ select stack_id,label,data_group,quality,filter,skycell_id from stackRun join stackSumSkyfile using (stack_id) where data_group like "MD05.GR0.2012%" and quality>0; | stack_id | label | data_group | quality | filter | skycell_id | +----------+----------+---------------------------------------+---------+---------+-------------+ | 1592090 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20100222 | 257 | i.00000 | skycell.017 | | 1586846 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110201 | 13008 | r.00000 | skycell.021 | | 1589350 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110414 | 13008 | g.00000 | skycell.017 | | 1592216 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110421 | 257 | i.00000 | skycell.028 | | 1592249 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110421 | 13008 | i.00000 | skycell.071 | | 1586910 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110423 | 257 | r.00000 | skycell.028 | | 1589451 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110423 | 13008 | g.00000 | skycell.071 | | 1582918 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110515 | 13008 | y.00000 | skycell.071 | | 1582926 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110515 | 13008 | y.00000 | skycell.082 | | 1589571 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110607 | 257 | g.00000 | skycell.071 | | 1592343 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090405 | 13008 | i.00000 | skycell.041 | | 1592381 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 | 13008 | i.00000 | skycell.017 | | 1592393 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 | 13008 | i.00000 | skycell.037 | | 1592394 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 | 13008 | i.00000 | skycell.038 | | 1592395 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 | 13008 | i.00000 | skycell.041 | | 1592399 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 | 13008 | i.00000 | skycell.045 | | 1592400 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 | 13008 | i.00000 | skycell.046 | | 1592401 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 | 13008 | i.00000 | skycell.047 | | 1592402 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 | 13008 | i.00000 | skycell.052 | | 1592403 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 | 13008 | i.00000 | skycell.053 | | 1584593 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091125 | 257 | z.00000 | skycell.003 | | 1584596 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091125 | 13008 | z.00000 | skycell.006 | | 1584620 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091125 | 13008 | z.00000 | skycell.040 | | 1584638 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091125 | 13008 | z.00000 | skycell.060 | | 1586957 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091126 | 13008 | r.00000 | skycell.004 | | 1589585 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091126 | 13008 | g.00000 | skycell.004 | | 1589646 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091126 | 13008 | g.00000 | skycell.082 | | 1584666 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091128 | 13008 | z.00000 | skycell.021 | | 1584713 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091128 | 3006 | z.00000 | skycell.078 | | 1589657 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091208 | 13008 | g.00000 | skycell.021 | | 1592467 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091215 | 3006 | i.00000 | skycell.082 | | 1587033 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091217 | 3006 | r.00000 | skycell.021 | | 1587041 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091217 | 13008 | r.00000 | skycell.030 | | 1589759 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091217 | 13008 | g.00000 | skycell.060 | | 1584086 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110517 | 13008 | y.00000 | skycell.087 | | 1586316 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110522 | 13008 | z.00000 | skycell.082 | | 1586375 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110525 | 13008 | z.00000 | skycell.082 | | 1589339 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110610 | 13008 | r.00000 | skycell.082 | | 1591959 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110610 | 13008 | g.00000 | skycell.082 | | 1591964 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110610 | 257 | g.00000 | skycell.087 | | 1586427 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110612 | 13008 | z.00000 | skycell.071 | | 1586435 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110612 | 13008 | z.00000 | skycell.082 | | 1586436 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110612 | 13008 | z.00000 | skycell.083 | | 1584093 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110613 | 13008 | y.00000 | skycell.021 | | 1584141 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110613 | 13008 | y.00000 | skycell.082 | | 1584146 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110613 | 13008 | y.00000 | skycell.087 | | 1584201 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110614 | 13008 | y.00000 | skycell.082 | | 1584259 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110615 | 13008 | y.00000 | skycell.082 | }}} == MD05.GR0 extra reprocessed set with all ubercal catalog == Chip->warp including camera SMF for all filters grizy from MD05.GR0 not processed with ubercal: * label: MD05.GR0.20121104 * data_group: MD05.GR0.20121104_redo20111011preubercal Chip->warp including camera SMF for all filters grizy of new MD05 for 2011/2012 season: * label: MD05.GR0.20121030 * data_group: MD05.GR0.20121030_redo2012preubercal Night stack for all filters grizy -- don't call nightlystack.. but now an additional label/data_group.. * label: MD05.GR0 * data_group: MD05.GR0.20121104.nightstack., MD05.GR0.20121030.nightstack. ---- = MD06.GR0 = ''Status:'' Refstacks and full set night stacks finished and distributed. Camera SMFs also distributed. * 11/18: all filters all exposures running chip->warp * 11/23: reftest samples running * 11/28: disk/data corruption. on hold. * 12/22: repairs mostly finished and restarted refstacks. * 12/31: more corrputions found during distribution, fixed. * 01/01/2012: Refstacks and sample night stacks finished and distributed. * 01/11/12: reprocessing of all night stacks possible in V3 started * 01/17/12: majority of all night stacks on datastore. A few faulted with various issues mainly from stacks with 2-3 inputs on far edge skycells. * 12/21/12: new refstack [wiki:MD06.refstack.20121221] will replace this one for diffims in the 2012/2013 season New observations planned and started in OC148 (January 8, 2012). All past g-band exposures in the V3 tessellation (other filters nearly identical). [[Image(MD06_g_smf.png,200px)]] '''Need to investigate tessellation center and dither for science with NGC4258''' Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks. || '''Filter''' || '''N total''' || '''N (FWHM<8 pixels)''' || || || || || || y || 298 || 268 || || z || 471 || 446 || || i || 564 || 513 || || r || 436 || 391 || || g || 407 || 341 || || || || || == MD06.V3 reference stack == * label: MD06.refstack.20111122 * data_group: MD06.refstack.20111122 Test stack (reftest) sample also made again for skycell.055 with different number of input warps. * label: MD06.reftest.20111121 * data_group: MD06.reftest.20111121 || [[Image(MD06.g.00000.expfwhm.png,300px)]] || [[Image(MD06.r.00000.expfwhm.png,300px)]] || || [[Image(MD06.i.00000.expfwhm.png,300px)]] || [[Image(MD06.z.00000.expfwhm.png,300px)]] || || [[Image(MD06.y.00000.expfwhm.png,300px)]] || Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20111122 (top blue) and refstack.20110323 (bottom blue). Any unmatched blue/red overlap for (V0) are due to ones included/rejected in the selected reprocessing. The previous (V0) refstack was only 8 inputs so should be much improvement! || Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, (V0)=078) after running psphot on the stacks. || || || || || || '''version label''' (stage_id) || '''filter''' || '''center skycell input warps''' || '''center skycell FWHM (major)''' || || ''MD06.refstack.20111122 V3'' || y || 107 (8) || 0.92" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.539498.1577033.33 (539498)] || y || 33 (4) || 0.83" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.539499.1577034.33 (539499)] || y || 69 (10) || 0.89" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.539500.1577035.33 (539500)] || y || 107 (11) || 0.92" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.539501.1577036.33 (539501)] || y || 129 (13) || 0.94" || || old MD06.refstack.20110323 (V0) || y || 87 (1) || 1.25" || || || || || || || ''MD06.refstack.20111122 V3'' || z || 110 (2) || 0.93" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.539502.1577037.33 (539502)] || z || 30 (6) || 0.88" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.539503.1577038.33 (539503)] || z || 70 (9) || 0.89" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.539504.1577039.33 (539504)] || z || 102 (6) || 0.95" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.539505.1577040.33 (539505)] || z || 142 (5) || 0.94" || || old MD06.refstack.20110323 (V0) || z || 16 (0) || 1.19" || || || || || || || ''MD06.refstack.20111122 V3'' || i || 108 (1) || 0.98" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.540829.1577041.33 (540829)] || i || 27 (2) || 0.86" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.540830.1577042.33 (540830)] || i || 62 (2) || 0.92" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.540831.1577043.33 (540831)] || i || 108 (3) || 0.97" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.540832.1577044.33 (540832)] || i || 128 (6) || 0.99" || || old MD06.refstack.20110323 (V0) || i || 8 (1) || 1.17" || || || || || || || ''MD06.refstack.20111122 V3'' || r || 114 (7) || 1.10" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.541343.1577045.33 (541343)] || r || 31 (2) || 1.15" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.541344.1577046.33 (541344)] || r || 60 (4) || 1.06" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.541345.1577047.33 (541345)] || r || 114 (8) || 1.10" || || - MD06.reftest.201111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.541346.1577048.33 (541346)] || r || 145 (9) || 1.11" || || old MD06.refstack.20110323 (V0) || r || 12 (1) || 1.13" || || || || || || || ''MD06.refstack.20111122 V3'' || g || 113 (15) || 1.17" || || - MD06.reftest.20111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.541347.1577049.33 (541347)] || g || 25 (0) || 1.20" || || - MD06.reftest.20111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.541348.1577050.33 (541348)] || g || 64 (8) || 1.14" || || - MD06.reftest.20111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.541349.1577051.33 (541349)] || g || 113 (12) || 1.17" || || - MD06.reftest.20111121 V3 [http://datastore.ipp.ifa.hawaii.edu/ps1-md-refstack/stack.541350.1577052.33 (541350)] || g || 144 (25) || 1.31" || || old MD06.refstack.20110323 (V0) || g || 12 (3) || 1.36" || || || || || || * || [[Image(MD06.20111122_g_test.png,300px)]] || [[Image(MD06.20111122_i_test.png,300px)]] || Troubled skycells? * skycell.087 i-band had quality failure, rerun excluding some earlier warps and okay. Needs more investigation into why it had a problem, others with quality issue may be similar. || [[Image(MD06.20091224.NULL.i.00000.stksummap.png,300px)]] || [[Image(MD06.refstack.20111122.NULL.i.00000.stksummap.png,300px)]] || == MD06.GR0 reprocessed sample and finished set == * label: condor_MD06.V3_01 * data_group: condor_MD06.V3_01_dg for chip->warp * dist_group: NULL, changed camera to MD06.GR Reprocessed night stack sample under the following label and data_group. * label: MD06.GR0 * dist_group: MD06.GR * data_group of initial sample of night stacks processed with ipp-20111110 tag and using the default 4 or more input warps necessary to make stack * data_group: MD06.GR0.20111122 = y * data_group: MD06.GR0.20111123 = z * data_group: MD06.GR0.20111124 = i * data_group: MD06.GR0.20111125 = r * data_group: MD06.GR0.20111126 = g * data_group of remaining sample of night stacks processed with ipp-20111222 tag and allowing 2 or more input warps necessary to make stack (typically the far outer skycells, but as a new case, will need to be looked at in more detail) * data_group: MD06.GR0.20120109 = y * data_group: MD06.GR0.20120110 = z * data_group: MD06.GR0.20120112 = i * data_group: MD06.GR0.20120114 = r * data_group: MD06.GR0.20120116 = g Distributed on datastore under * ps1-md-GR: sample of night stack images only (setting up for full set). SSdiff sample still to be run. * ps1-md-GR-cat: camera SMFs done (have the data_group condor_MD06.V3_01_dg) Older night stacks are really not that pretty.. Problem/corrupt bundles: {{{ stage_id,filesetID,time registered,target_id,fs_tag,filter,productID,type,data_group,stage 609903,stack.609903.1628823.53,2012-01-17T19:35:29Z,1985,stack.609903,i.00000,ps1-md-GR,IPP-DIST,MD06.GR0.20120112,stack }}} ---- = MD07.GR0 = ''Status:'' Refstacks and sample night stacks finished and distributed. * 10/12: all filters all exposures running chip->warp * 11/6: all warps finished, y-band refstack started. * 11/07: y-band refstack finished, z-band started, reftest samples run * 11/08: zirg refstack running, night stack samples spanning the period since 4/2009 running. * 11/12: full refstack set, night stack samples finished and distributed * 6/18/12: staticsky run attempted, see notes below * 01/09/13 [wiki:MD07.refstack.20130102] to replace 2011 season refstack Coming soon. New observations planned for OC149 (February 7, 2012). All MD07 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD07 z-band night stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible. [[Image(MD07_g_smf.png,200px)]] [[Image(MD07ns_z_smf.png,200px)]] Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks (more important for this field given the two pointings since 4/2009). || '''Filter''' || '''N total''' || '''N (FWHM<8 pixels)''' || || || || || || y || 391 || 302 || || z || 481 || 422 || || i || 561 || 458 || || r || 423 || 370 || || g || 423 || 343 || || || || || == MD07.V3 reference stack == * label: MD07.refstack.20111106 * data_group: MD07.refstack.20111106 Test stack (reftest) sample also made again for skycell.055 with different number of input warps. * label: MD07.reftest.20111107 * data_group: MD07.reftest.20111107 || [[Image(MD07.g.00000.expfwhm.png,300px)]] || [[Image(MD07.r.00000.expfwhm.png,300px)]] || || [[Image(MD07.i.00000.expfwhm.png,300px)]] || [[Image(MD07.z.00000.expfwhm.png,300px)]] || || [[Image(MD07.y.00000.expfwhm.png,300px)]] || Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20111106 (top blue) and refstack.20100330 (V0) (bottom blue). Any unmatched blue/red overlap for (V0) are due to ones included/rejected in the selected reprocessing. The previous (V0) refstack was only 8 inputs so should be much improvement! || Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, (V0)=078) after running psphot on the stacks. || '''version label''' || '''filter''' || '''center skycell input warps''' || '''center skycell FWHM (major)''' || || ''MD07.refstack.20111106 V3'' || y || 116(8) || 0.81" || || - MD07.reftest.201111107 V3 || y || 31(1) || 0.74" || || - MD07.reftest.201111107 V3 || y || 66(9) || 0.77" || || - MD07.reftest.201111107 V3 || y || 143(10) || 0.85" || || old MD07.refstack.20100330 (V0) || y || 8(0) || 0.91" || || || || || || || ''MD07.refstack.20111106 V3'' || z || 126(10) || 0.96" || || - MD07.reftest.201111107 V3 || z || 29(2) || 0.89" || || - MD07.reftest.201111107 V3 || z || 59(4) || 0.91" || || - MD07.reftest.201111107 V3 || z || 126(10) || 0.96" || || - MD07.reftest.201111107 V3 || z || 154(14) || 0.98" || || old MD07.refstack.20100330 (V0) || z || 8(0) || 1.27" || || || || || || || ''MD07.refstack.20111106 V3'' || i || 126(19) || 1.07" || || - MD07.reftest.201111107 V3 || i || 29(0) || 1.00" || || - MD07.reftest.201111107 V3 || i || 68(9) || 1.03" || || - MD07.reftest.201111107 V3 || i || 98(5) || 1.05" || || - MD07.reftest.201111107 V3 || i || 126(19) || 1.06" || || - MD07.reftest.201111107 V3 || i || 146(16) || 1.07" || || old MD07.refstack.20100330 (V0) || i || 8(0) || 1.28" || || || || || || || ''MD07.refstack.20111106 V3'' || r || 138(6) || 1.11" || || - MD07.reftest.201111107 V3 || r || 25(2) || 1.02" || || - MD07.reftest.201111107 V3 || r || 60(7) || 1.08" || || - MD07.reftest.201111107 V3 || r || 115(18) || 1.10" || || - MD07.reftest.201111107 V3 || r || 138(7) || 1.10" || || old MD07.refstack.20110121 (V0) || r || 8(0) || 1.46" || || || || || || || ''MD07.refstack.20111106 V3'' || g || 152(11) || 1.15" || || - MD07.reftest.20111107 V3 || g || 29(2) || 1.08" || || - MD07.reftest.20111107 V3 || g || 59(4) || 1.11" || || - MD07.reftest.20111107 V3 || g || 126(10) || 1.13" || || - MD07.reftest.20111107 V3 || g || 154(14) || 1.14" || || old MD07.refstack.20100330 (V0) || g || 8(0) || 1.67" || || || || || || * seems to be some strange rejection behavior in at least the i-band. || [[Image(MD07.20111110_g_test.png,300px)]] || [[Image(MD07.20111110_i_test.png,300px)]] || How well does the "best" selected inputs cover the two pointings? * note (V0) y-band refstack was at newer pointing, how many y-bands in old pointing? Troubled skycells? * 030-z quality fail * first pointing (older data) skycell coverage needs further looking into. generally includes 093,094,095,096, 086,087,088, 078,079, 068,069, 059, 049, 039 * 077 large rej %, looks like for older data coverage || [[Image(MD07.20100212.NULL.g.00000.stksummap.png,300px)]] || [[Image(MD07.refstack.20111106.NULL.g.00000.stksummap.png,300px)]] || == MD07.GR0 reprocessed sample and finished set == Reprocessing of chip-->warp done mostly with condor. Hiccup with loss of tmp files stalled things in the middle of warp, but pantasks picked up the processing fine using the label. * label: condor_MD07.V3_01 * data_group: condor_MD07.V3_01_dg * dist_group: NULL, changed camera to MD07.GR Reprocessed night stack sample under the following label and data_group. * label: MD07.GR0 * dist_group: MD07.GR * data_group of initial sample of night stacks processed with both ipp-20110622 and ipp-20111110 tag (see date) and using the default 4 or more input warps necessary to make stack * data_group: MD07.GR0.20111107 = y * data_group: MD07.GR0.20111108 = z * data_group: MD07.GR0.20111109 = i * data_group: MD07.GR0.20111110 = r * data_group: MD07.GR0.20111111 = g * data_group of remaining sample of night stacks processed with ipp-20111222 tag and allowing 2 or more input warps necessary to make stack * data_group: MD07.GR0.201201xx = y * data_group: MD07.GR0.201201xx = z * data_group: MD07.GR0.201201xx = i * data_group: MD07.GR0.201201xx = r * data_group: MD07.GR0.201201xx = g Staticsky: only run on refstack skycells with all 5-filters available. This run is on stacks prior to ubercal catalogs so the zeropoint is fundamentally based on the synthcat values. This run provides a sample for testing the stack calibration stage. However, many faults (25/70 runs, i.e. skycells) occurred in the processing and need to still be investigated. label: MD07.refstack.20111106 data_group: MD07.refstack.20111106.5x20120618 dist_group: MD.staticsky (TBD) Distributed on datastore under * ps1-md-GR: sample of night stack images only (setting up for full set). SSdiff sample still to be run. * ps1-md-GR-cat: camera SMFs loading (have the data_group condor_MD07.V3_01_dg) ---- = MD08.GR0 = ''Status:'' In progress * 11/25: all filters all exposures running chip->warp * 11/28: disk/data corruption extensive. will need to be restarted from scratch once diskspace available. * 4/18/12: restarted from scratch after raid crash in November/December with the new ipp-20120404 tag to include new ubercal refcat (PS1.REF.20120325) and detrend improvements (dark modes, noise map but not the boost factors). * 4/20: gri reprocessing finished and refstacks started. zy prepped but delayed until 4/22 for other tests. * 4/21: gri reprocessed camera SMFs and night stacks distributed, reference and test stacks also distributed * 4/22: z reprocessing started * 4/24: zy reprocessing finished, zy refstack started * 4/25: all night stack reprocessing finished, all refstacks finished and distributed, staticsky run and distributed. Coming soon. New observations planned for OC152 (May 5). All past g-band exposures in the V3 tessellation (other filters nearly identical). [[Image(MD08_g_smf.png,200px)]] Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks. || '''Filter''' || '''N total''' || '''N (FWHM<8 pixels)''' || || || || || || g || 475 || 401 || || r || 499 || 426 || || i || 516 || 472 || || z || 532 || 494 || || y || 565 || 466 || || || || || == MD08.V3 reference stack == Reference stack using data from 2010+2011, a balance of N input warps versus best seeing, excluding exposures with large background, low zeropoints, and large elongation. * label: MD08.refstack.20120422 * data_group: MD08.refstack.20120422 Test stack (reftest) sample also made again for skycell.055 with different number of input warps. * label: MD08.reftest.201204xx (xx=17 for g, 18 for r, 19 for i, 20 for z, 21 for y) * data_group: * MD08.reftest.201204xx.c#f where #=comparison run, =cut where all means 2009+2010+2011 data possible, =cut where 2011 means 2011 data only, =cut = g * data_group: MD08.GR0.20120418.nightstack. = r * data_group: MD08.GR0.20120419.nightstack. = i * data_group: MD08.GR0.201204120.nightstack. = z ('''NOTE the TYPO in the date 201204120 and not 20120420''') * data_group: MD08.GR0.20120421.nightstack. = y * dist_group: MD08.GR * night stack_id/skycell found faulted {{{ 862445 skycell.087 MD08.GR0 MD08.GR0.20120417.nightstack.20090620 MD08.GR g.00000 863535 skycell.021 MD08.GR0 MD08.GR0.20120417.nightstack.20100630 MD08.GR g.00000 867446 skycell.082 MD08.GR0 MD08.GR0.20120418.nightstack.20100603 MD08.GR r.00000 867746 skycell.021 MD08.GR0 MD08.GR0.20120418.nightstack.20100712 MD08.GR r.00000 869430 skycell.071 MD08.GR0 MD08.GR0.20120418.nightstack.20110731 MD08.GR r.00000 870405 skycell.005 MD08.GR0 MD08.GR0.20120419.nightstack.20090627 MD08.GR i.00000 870411 skycell.017 MD08.GR0 MD08.GR0.20120419.nightstack.20090627 MD08.GR i.00000 870649 skycell.005 MD08.GR0 MD08.GR0.20120419.nightstack.20090919 MD08.GR i.00000 870995 skycell.012 MD08.GR0 MD08.GR0.20120419.nightstack.20100601 MD08.GR i.00000 871557 skycell.071 MD08.GR0 MD08.GR0.20120419.nightstack.20100821 MD08.GR i.00000 871573 skycell.017 MD08.GR0 MD08.GR0.20120419.nightstack.20100830 MD08.GR i.00000 871904 skycell.021 MD08.GR0 MD08.GR0.20120419.nightstack.20100917 MD08.GR i.00000 872195 skycell.012 MD08.GR0 MD08.GR0.20120419.nightstack.20110512 MD08.GR i.00000 872677 skycell.021 MD08.GR0 MD08.GR0.20120419.nightstack.20110629 MD08.GR i.00000 872725 skycell.082 MD08.GR0 MD08.GR0.20120419.nightstack.20110629 MD08.GR i.00000 873156 skycell.017 MD08.GR0 MD08.GR0.20120419.nightstack.20110801 MD08.GR i.00000 873397 skycell.021 MD08.GR0 MD08.GR0.20120419.nightstack.20110822 MD08.GR i.00000 877403 skycell.004 MD08.GR0 MD08.GR0.201204120.nightstack.20090622 MD08.GR z.00000 877937 skycell.012 MD08.GR0 MD08.GR0.201204120.nightstack.20100409 MD08.GR z.00000 878062 skycell.017 MD08.GR0 MD08.GR0.201204120.nightstack.20100418 MD08.GR z.00000 878747 skycell.078 MD08.GR0 MD08.GR0.201204120.nightstack.20100702 MD08.GR z.00000 879379 skycell.078 MD08.GR0 MD08.GR0.201204120.nightstack.20100912 MD08.GR z.00000 880042 skycell.021 MD08.GR0 MD08.GR0.201204120.nightstack.20110706 MD08.GR z.00000 881059 skycell.005 MD08.GR0 MD08.GR0.20120421.nightstack.20090706 MD08.GR y.00000 882054 skycell.082 MD08.GR0 MD08.GR0.20120421.nightstack.20100728 MD08.GR y.00000 884385 skycell.082 MD08.GR0 MD08.GR0.20120421.nightstack.20110925 MD08.GR y.00000 884546 skycell.082 MD08.GR0 MD08.GR0.20120421.nightstack.20110928 MD08.GR y.00000 }}} Staticsky: only run on refstack skycells with all 5-filters available and distributed under ps1-md-staticsky. Convolved matching to target PSF much improved, generally the 4-6 pixels now similar to upper input PSF cutoff except for oddities that drive edge skycells >10,20,70 pixels. Work continues to better constrain selection of the target PSF. * label: MD08.refstack.20120422 * data_group: MD08.refstack.20120422.5x20120425 * dist_group: MD.staticsky * '''Eric Morganson has noted that the RA_PSF,DEC_PSF is NULL''' http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/ticket/1505, need to track down if fixed and broken again. Should be able to use skycell WCS to get coordinates from xy -- should be fixed in later runs of stack photometry (Aug. 2012) Reprocessing notes: * lost raw files: {{{ -- g-band neb://ipp047.0/gpc1/20100606/o5353g0215o/o5353g0215o.ota45.fits neb://ipp051.0/gpc1/20100621/o5368g0189o/o5368g0189o.ota64.fits neb://ipp013.0/gpc1/20100814/o5422g0083o/o5422g0083o.ota22.fits -- r-band neb://ipp040.0/gpc1/20100513/o5329g0443o/o5329g0443o.ota15.fits -- i-band neb://ipp040.0/gpc1/20100803/o5411g0127o/o5411g0127o.ota15.fits -- z-band neb://ipp049.0/gpc1/20100816/o5424g0110o/o5424g0110o.ota54.fits -- y-band neb://ipp009.0/gpc1/20100626/o5373g0272o/o5373g0272o.ota12.fits neb://ipp006.0/gpc1/20100627/o5374g0261o/o5374g0261o.ota03.fits neb://ipp023.0/gpc1/20100727/o5404g0108o/o5404g0108o.ota06.fits }}} * lost but regenerated intermediate files: {{{ -- r-band neb://ipp027.0/gpc1/20100712/o5389g0088o/o5389g0088o.ota26.burn.tbl -- i-band neb://ipp031.0/gpc1/20100619/o5366g0076o/o5366g0076o.ota43.burn.tbl neb://ipp025.0/gpc1/20100619/o5366g0076o/o5366g0076o.ota76.burn.tbl neb://ipp050.0/gpc1/20100701/o5378g0154o/o5378g0154o.ota61.burn.tbl neb://ipp029.0/gpc1/20100917/o5456g0036o/o5456g0036o.ota33.burn.tbl -- z-band neb://ipp040.0/gpc1/20100521/o5337g0137o/o5337g0137o.ota14.burn.tbl neb://ipp045.0/gpc1/20100702/o5379g0086o/o5379g0086o.ota35.burn.tbl neb://ipp048.0/gpc1/20100702/o5379g0087o/o5379g0087o.ota50.burn.tbl neb://ipp019.0/gpc1/20100702/o5379g0088o/o5379g0088o.ota36.burn.tbl neb://ipp008.0/gpc1/20100702/o5379g0090o/o5379g0090o.ota10.burn.tbl neb://ipp044.0/gpc1/20100702/o5379g0090o/o5379g0090o.ota31.burn.tbl neb://ipp030.0/gpc1/20100702/o5379g0090o/o5379g0090o.ota37.burn.tbl -- y-band neb://ipp034.0/gpc1/20100626/o5373g0271o/o5373g0271o.ota57.burn.tbl }}} * features/artifacts * seeing "schools" of ghosts in the stacks (ie skycell.063 in g) * some valleys (044-g,r) * edge "pox" (028-i, 087-y) * dither junk? (074-r) * odd images to beware of {{{ -- either tracking jump or mirror solution issue - check if used/rejected in z night stack, yes but rejected at end before final stack made by PSF chi2 clip. neb://ipp048.0/gpc1/MD08.GR0.20120420/o5103g0041o.101597/o5103g0041o.101597.ch.442508 }}} ---- = MD09.GR0 = ''Status:'' * 5/30: prep work started for chip->warp reprocessing. * 6/01: chip->warp reprocessing started with new ops tag ipp-20120531 (improved REFCAT 20120524, ghost models, corner under-glow masking). seeing some trouble with new tag in z,i,r for XY67,71. * 6/02: reftests run for g,y,z * 6/03: refstack for g,y started, night stacks run. * 6/04: refstack for z started with exposures available, full night stack processing on hold for z,r,i until solution with XY68,71 found. Will try an do whatever is possible. * 6/06: MD09 r,i refstacks started. z refstack, 90% night stacks out to datastore. * 6/07: refstacks finished and distributed, staticsky on refstack finished, 90% of camera SMF and night stacks that are possible distributed (see listing below for dates of incomplete processing). * 7/29 MD09.GR0 XY67,71 chip problem (~48 exposures) cannot be fixed so set to poor quality, remaining camera SMF and night stacks processed and distributed. * '''8/31 replaced with MD09.refstack.20120831 to fix the weigh pixel problem causing arc region of lower sensitivity''' Coming soon for new observations planned for OC153 (June 3). Observations attempted earlier but weather has prevented. || [[Image(MD09_g_smf.png,200px)]] [[Image(MD09ns_z_smf.png,200px)]] || || All MD09 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD09 z-band night stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible. || Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks. || '''Filter''' || '''N total''' || '''N (FWHM<8 pixels)''' || || || || || || g || 471 || 415 || || r || 459 || 427 || || i || 478 || 432 || || z || 491 || 469 || || y || 303 || 283 || || || || || == MD09.V3 reference stack == Reference stack using data from 2010+2011, a balance of N input warps versus best seeing, excluding exposures with large background, low zeropoints, and large elongation. * label: MD09.refstack.20120603 * data_group: MD09.refstack.20120603 Test stack (reftest) sample also made again for skycell.055 with different number of input warps. * label: MD09.reftest.20120602 * data_group: * MD09.reftest.20120602.c#f where #=comparison run, =cut where all means 2009+2010+2011 data possible, =cut where 2011 means 2011 data only, =cut x20130114 -- = 1,2,3,4,5 and number of filters available for the set, so 5 different data_groups * skycal label, data_group: MD09.deeptest.20120727.Nx20130114.cal20130128 Source photometry psf_major,_minor maps (grizy): || [[Image(md09deepNx20130128skycal_cmf_g.txt_psfmap.png,300px)]] || [[Image(md09deepNx20130128skycal_cmf_r.txt_psfmap.png,300px)]] || [[Image(md09deepNx20130128skycal_cmf_i.txt_psfmap.png,300px)]] || [[Image(md09deepNx20130128skycal_cmf_z.txt_psfmap.png,300px)]] || [[Image(md09deepNx20130128skycal_cmf_y.txt_psfmap.png,300px)]] || Magnitude of sources@5-sigma maps (grizy): || [[Image(md09deepNx20130128skycal_cmf_g.txt_mag5sigmap.png,300px)]] || [[Image(md09deepNx20130128skycal_cmf_r.txt_mag5sigmap.png,300px)]] || [[Image(md09deepNx20130128skycal_cmf_i.txt_mag5sigmap.png,300px)]] || [[Image(md09deepNx20130128skycal_cmf_z.txt_mag5sigmap.png,300px)]] || [[Image(md09deepNx20130128skycal_cmf_y.txt_mag5sigmap.png,300px)]] || Detection efficiency magnitude @50% recovery (500 fakes, grizy) including total N warps in and rejected: || [[Image(md09deepNx20130128skycal_cmf_g.txt_mag50gridmap.png,300px)]] || [[Image(md09deepNx20130128skycal_cmf_r.txt_mag50gridmap.png,300px)]] || [[Image(md09deepNx20130128skycal_cmf_i.txt_mag50gridmap.png,300px)]] || [[Image(md09deepNx20130128skycal_cmf_z.txt_mag50gridmap.png,300px)]] || [[Image(md09deepNx20130128skycal_cmf_y.txt_mag50gridmap.png,300px)]] || Stack photometry comparisons -- see next section for i-band example and problem for refstack as well. * calibrated stack photometry appears suspect to at least 0.03-0.05 (have seen much worse in other fields): || skycell overlap match mag comparison || [[Image(md09deepNx20130128skycal_cmf_i.txt_dmagptmap.png,300px)]] || || refcat match mag comparison example: appears 0.03-0.05 mag error in pixels (staticsky) || [[Image(md09deep20130114x1fsky_cmf_i.txt_drefmagptmap.png,300px)]] || || refcat match mag comparison example: appears 0.03-0.05 mag error still in photometry recalibration (skycal) || [[Image(md09deepNx20130128skycal_cmf_i.txt_drefmagptmap.png,300px)]] || == MD09.GR0 reprocessed set == __Chips/Camera/Warp__: camera SMFs distributed under ps1-md-GR-cat * label: MD09.GR0 * data_group: MD09.GR0.20120601 = includes all filters * dist_group: MD09.GR Reprocessing notes: * lost raw files: {{{ -- i-band neb://ipp029.0/gpc1/20100731/o5408g0319o/o5408g0319o.ota33.fits neb://ipp026.0/gpc1/20100806/o5414g0429o/o5414g0429o.ota17.fits }}} * features/artifacts * seeing "schools" of ghosts in the stacks like MD08? * odd images to beware of -- * sample still unprocessed due to XY71,67 faults. tracing/debugging of new ops tag required to fix. * seems to be problem with psphot from ipp-20120531 tag, tests work with ipp-20120626 tag. these will be given a quality fault of 42 so that the exposures and night stacks can finish. {{{ | chip_id | exp_id | exp_name | dateobs | filter | comment | +---------+--------+-------------+---------------------+---------+--------------------------------+ -- | 467866 | 214900 | o5436g0340o | 2010-08-28 10:47:07 | z.00000 | MD09 z N5436 MD09 dither 6 | | 467880 | 218834 | o5442g0484o | 2010-09-03 12:45:57 | z.00000 | MD09 z N5442 MD09 dither 4 | | 467907 | 223972 | o5451g0362o | 2010-09-12 11:15:50 | z.00000 | MD09 z N5451 MD09 dither 7 | | 468019 | 353527 | o5739g0231o | 2011-06-27 12:45:40 | z.00000 | MD09 z N5739 MD09 dither 7 | | 468043 | 360681 | o5754g0447o | 2011-07-12 14:04:56 | z.00000 | MD09 z N5754 MD09 dither 7 | | 468058 | 367560 | o5766g0335o | 2011-07-24 13:52:27 | z.00000 | MD09 z N5766 MD09 dither 2 | | 468059 | 367564 | o5766g0340o | 2011-07-24 14:13:49 | z.00000 | MD09 z N5766 MD09 dither 7 | | 468062 | 369197 | o5769g0479o | 2011-07-27 13:11:44 | z.00000 | MD09 z N5769 MD09 dither 2 | | 468066 | 369201 | o5769g0483o | 2011-07-27 13:28:45 | z.00000 | MD09 z N5769 MD09 dither 6 | | 468077 | 371776 | o5775g0526o | 2011-08-02 13:38:31 | z.00000 | MD09 z N5775 MD09 dither 1 | | 468079 | 371778 | o5775g0528o | 2011-08-02 13:47:02 | z.00000 | MD09 z N5775 MD09 dither 3 | | 468080 | 371779 | o5775g0529o | 2011-08-02 13:51:17 | z.00000 | MD09 z N5775 MD09 dither 4 | | 468095 | 381404 | o5796g0497o | 2011-08-23 13:28:05 | z.00000 | MD09 z N5796 MD09 dither 3 | | 468112 | 383829 | o5802g0378o | 2011-08-29 12:07:06 | z.00000 | MD09 z N5802 MD09 dither 4 | | 468123 | 385223 | o5805g0137o | 2011-09-01 07:48:19 | z.00000 | MD09 z N5805 MD09 dither 7 | | 468167 | 400366 | o5832g0065o | 2011-09-28 07:47:34 | z.00000 | MD09 z N5832 MD09 dither 6 | | 468175 | 400374 | o5832g0073o | 2011-09-28 08:24:06 | z.00000 | MD09 z N5832 MD09 dither 6 | | 468184 | 401319 | o5835g0049o | 2011-10-01 05:49:30 | z.00000 | MD09 z N5835 MD09 dither 7 | | 468187 | 406280 | o5846g0222o | 2011-10-12 08:40:09 | z.00000 | MD09 z,y N5846 z MD09 dither 2 | | 468192 | 406285 | o5846g0227o | 2011-10-12 09:01:27 | z.00000 | MD09 z,y N5846 z MD09 dither 7 | -- | 468770 | 200329 | o5408g0319o | 2010-07-31 12:14:02 | i.00000 | MD09 i N5408 MD09 dither 4 | | 468773 | 200342 | o5408g0318o | 2010-07-31 12:09:46 | i.00000 | MD09 i N5408 MD09 dither 3 | | 468811 | 208985 | o5426g0478o | 2010-08-18 13:12:34 | i.00000 | MD09 i N5426 MD09 dither 5 | | 468812 | 208986 | o5426g0479o | 2010-08-18 13:16:47 | i.00000 | MD09 i N5426 MD09 dither 6 | | 468819 | 210454 | o5429g0326o | 2010-08-21 11:15:35 | i.00000 | MD09 i N5429 MD09 dither 5 | | 468836 | 218088 | o5441g0356o | 2010-09-02 11:18:09 | i.00000 | MD09 i N5441 MD09 dither 6 | | 468843 | 219741 | o5444g0239o | 2010-09-05 09:46:59 | i.00000 | MD09 i N5444 MD09 dither 5 | | 468852 | 221522 | o5447g0278o | 2010-09-08 09:43:15 | i.00000 | MD09 i N5447 MD09 dither 6 | | 468889 | 231309 | o5465g0446o | 2010-09-26 11:22:13 | i.00000 | MD09 i N5465 MD09 dither 4 | | 468903 | 236298 | o5477g0051o | 2010-10-08 05:36:12 | i.00000 | MD09 i N5477 MD09 dither 1 | | 468917 | 237920 | o5480g0086o | 2010-10-11 06:00:05 | i.00000 | MD09 i N5480 MD09 dither 7 | | 468943 | 248892 | o5501g0244o | 2010-11-01 08:28:26 | i.00000 | MD09 i N5501 MD09 dither 1 | | 468978 | 353262 | o5738g0243o | 2011-06-26 13:36:11 | i.00000 | MD09 i N5738 MD09 dither 4 | | 468997 | 354596 | o5741g0564o | 2011-06-29 14:01:00 | i.00000 | MD09 i N5741 MD09 dither 5 | | 469023 | 368656 | o5768g0501o | 2011-07-26 13:50:56 | i.00000 | MD09 i N5768 MD09 dither 7 | | 469053 | 379157 | o5792g0496o | 2011-08-19 13:15:52 | i.00000 | MD09 i N5792 MD09 dither 2 | | 469055 | 379159 | o5792g0498o | 2011-08-19 13:24:25 | i.00000 | MD09 i N5792 MD09 dither 4 | | 469063 | 383099 | o5801g0134o | 2011-08-28 08:18:38 | i.00000 | MD09 i N5801 MD09 dither 4 | | 469069 | 384934 | o5804g0438o | 2011-08-31 12:29:08 | i.00000 | MD09 i N5804 MD09 dither 2 | | 469087 | 387780 | o5810g0168o | 2011-09-06 07:35:28 | i.00000 | MD09 i N5810 MD09 dither 7 | | 469111 | 395309 | o5825g0061o | 2011-09-21 06:32:28 | i.00000 | MD09 i N5825 MD09 dither 7 | | 469118 | 399219 | o5829g0169o | 2011-09-25 07:54:00 | i.00000 | MD09 r,i N5829 i MD09 dither 6 | | 469135 | 402155 | o5837g0066o | 2011-10-03 06:51:54 | i.00000 | MD09 i N5837 MD09 dither 7 | | 469160 | 413826 | o5859g0142o | 2011-10-25 07:25:05 | i.00000 | MD09 r,i N5859 i MD09 dither 6 | -- | 468395 | 226092 | o5455g0293o | 2010-09-16 09:35:27 | r.00000 | MD09 g,r N5455 r MD09 dither 7 | | 468510 | 372293 | o5776g0424o | 2011-08-03 13:33:22 | r.00000 | MD09 g,r N5776 r MD09 dither 2 | | 468534 | 378593 | o5791g0497o | 2011-08-18 14:13:15 | r.00000 | MD09 g,r N5791 r MD09 dither 2 | | 468535 | 378594 | o5791g0498o | 2011-08-18 14:15:55 | r.00000 | MD09 g,r N5791 r MD09 dither 3 | | 468580 | 387141 | o5809g0169o | 2011-09-05 07:30:26 | r.00000 | MD09 g,r N5809 r MD09 center | | 468589 | 389143 | o5812g0319o | 2011-09-08 10:55:03 | r.00000 | MD09 g,r N5812 r MD09 dither 1 | | 468617 | 398815 | o5828g0218o | 2011-09-24 08:58:45 | r.00000 | MD09 g,r N5828 r MD09 dither 5 | }}} __Night stacks__: under the following label and data_group and distributed under ps1-md-GR * label: MD09.GR0 * data_group: MD09.GR0.20120602.nightstack. = includes all filters * dist_group: MD09.GR * night stack_id/skycell found faulted (5), may be manually recoverable but not in near future. {{{ Stack ID Skycell ID Label data grp dist grp Filter 961400 skycell.030 MD09.GR0 MD09.GR0.20120602.nightstack.20100928 MD09.GR g.00000 963743 skycell.031 MD09.GR0 MD09.GR0.20120602.nightstack.20090706 MD09.GR y.00000 963802 skycell.045 MD09.GR0 MD09.GR0.20120602.nightstack.20090707 MD09.GR y.00000 964810 skycell.082 MD09.GR0 MD09.GR0.20120602.nightstack.20110812 MD09.GR y.00000 965323 skycell.054 MD09.GR0 MD09.GR0.20120602.nightstack.20111023 MD09.GR y.00000 967167 skycell.068 MD09.GR0 MD09.GR0.20120602.nightstack.20100711 MD09.GR z.00000 967913 skycell.038 MD09.GR0 MD09.GR0.20120602.nightstack.20101006 MD09.GR z.00000 969776 skycell.061 MD09.GR0 MD09.GR0.20120602.nightstack.20090915 MD09.GR r.00000 969954 skycell.061 MD09.GR0 MD09.GR0.20120602.nightstack.20090924 MD09.GR r.00000 973855 skycell.038 MD09.GR0 MD09.GR0.20120602.nightstack.20101017 MD09.GR i.00000 }}} * also appears some night stacks missing from regular nightly science processing picked up in GR0: 20111021 gr __Stack Photometry (staticsky+skycal)__: Originally, only run on refstack skycells with all 5-filters available and distributed under ps1-md-staticsky. Convolved matching to target PSF much improved, generally the 4-6 pixels now similar to upper input PSF cutoff except for oddities that drive edge skycells >10 pixels (extended source measurements should be on the RAW/unconv image in this tag however). Work continues to better constrain selection of the target PSF. * label: MD09.refstack.20120603 * data_group: MD09.refstack.20120603.5x20120606 * dist_group: MD.staticsky * staticsky faults/problems {{{ | sky_id | state | workdir | label | data_group | dist_group | reduction | note | registered | path_base | dtime_phot | dtime_script | sources | num_inputs | hostname | good_frac | quality | fault | +--------+-------+--------------------------------------------------------------------------+------------------------+-----------------------------------+--------------+-----------+------+---------------------+------------------------------------------------------------------------------------------------------------------------+------------+--------------+---------+------------+----------+-----------+---------+-------+ | 190101 | new | neb://@HOST@.0/gpc1/MD09.refstack.20120603/MD09.GR0.staticsky.5x20120606 | MD09.refstack.20120603 | MD09.refstack.20120603.5x20120606 | NULL | NULL | NULL | 2012-06-07 09:21:29 | neb://any/gpc1/MD09.refstack.20120603/MD09.GR0.staticsky.5x20120606/MD09.V3/skycell.022/MD09.V3.skycell.022.sky.190101 | NULL | 172 | 0 | 0 | ippc31 | NULL | 0 | 3 | | 190133 | new | neb://@HOST@.0/gpc1/MD09.refstack.20120603/MD09.GR0.staticsky.5x20120606 | MD09.refstack.20120603 | MD09.refstack.20120603.5x20120606 | NULL | NULL | NULL | 2012-06-07 09:21:29 | neb://any/gpc1/MD09.refstack.20120603/MD09.GR0.staticsky.5x20120606/MD09.V3/skycell.058/MD09.V3.skycell.058.sky.190133 | NULL | 3880 | 0 | 0 | ippc38 | NULL | 0 | 4 | }}} * Nigel finds RA_PSF,DEC_PSF still NULL for these -- should be fixed now in future versions (8/2012) Re-run version Jan. 2013: '''using replacement refstack.20120831''' * staticsky (stack photometry): * label: MD09.refstack.20120831.N20130115 * data_group: MD09.refstack.20120831.x20130115 -- = 1,2,3,4,5 and number of filters available for the set, so 5 different data_group per MD * skycal (stack photometry recalibration): * label: MD09.refstack.20120831.Nx20130115.cal20130128 * data_group: MD09.refstack.20120831.Nx20130115.cal20130128 * dist_group: MD.staticsky Problems: * code fault: sky_id=329074, skycell.058 * calibrated stack photometry appears suspect to at least 0.03-0.05 (have seen much worse in other fields): || skycell overlap match mag comparison || [[Image(md09refNx20130128skycal_cmf_i.txt_dmagptmap.png,300px)]] || || refcat match mag comparison example: appears 0.03-0.05 mag error in pixels (staticsky) || [[Image(md09refNx20130115sky_cmf_i.txt_drefmagptmap.png,300px)]] || || refcat match mag comparison example: appears 0.03-0.05 mag error still in photometry recalibration (skycal) || [[Image(md09refNx20130128skycal_cmf_i.txt_drefmagptmap.png,300px)]] || __Difference Images__: sample still to be run? Not going to be produced for MD.GR0 ---- = MD11.GR0 = Not to be reprocessed in MD.GR0 (1/14/13) ---- ----