Version 578 (modified by mhuber@…, 5 years ago)

--

Summary of MD.GR0 ---closing process updates in progress---

(back to MediumDeepFields summary)

Summary page of information for the MD GR0 (grand reprocessing version 0) including the new reference stack in the V3 tessellation with the start of new observations of MD10 in OC142. This is version 0 as the test start of full reprocessing of the MD fields to be balanced along with other reprocessing, 3PI stacks and nightly science, generation of new reference stacks. A new ubercal reference catalog (PS1.REF.20120325) is available and applies only to MD08, MD09 uses REF.20120524, all others (MD10, 01, 02, 03, 04, 05, 06, 07) use the original synthetic catalog.

Creation of the new reference stacks need to be, at the very least, made prior to the return to the particular MD field given in the current schedule http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Medium_Deep_Schedule. After the July, August, September time crunch of a new field a month, there should be time to continue reprocessing and creation of new reference stacks in advance.

  • 8/15/11 minimum mode re-processing for reference stacks after MD10 for evaluation and get next MD field ready (MD01)
  • 9/30/11 more success weaving in reprocessing with LAP, started reprocessing for refstacks MD03,04 (with more reprocessed exposures)
  • 10/09/11 MD03 refstack and sample stacks available, testing MD04 ~200 very deep stack, MD05 running
  • 10/17/11 MD07 chip-->warp processing started with condor, using any extra cpu cycles.
  • 11/06/11 MD07 refstacks being made with order ~120 inputs. MD04 deepstack testing continued, gri sample released, zy on hold until fixes made.
  • 11/18/11 MD06 chip->warp processing running with condor, refstack preparation (using new ipp-20111110 ops tag now)
  • 11/25/11 MD06 reftests made and y-band refstack started, MD08 chip->warp processing via condor, refstack preparation
  • 12/02/11 MD06/08 on hold, data uncertain with ipp064 down.
  • 12/20/11 MD06 processing recoverable, refstacks proceeding. MD08 a mess and will need to be re-run.
  • 12/31/11 MD06 finally finished, distributed.
  • 1/09/12 MD08, 09 on hold while finishing up full reprocessing and release of V3 products now (i.e., nightly stacks, SMFs, SSdiffs).
  • 1/11/12 MD06,04 full camera SMFs on datastore, MD07 later today. MD04 set has some duplicates with both unmagicked SMFs (data_group=MD04.GR0, sub-sample of ~150/filter best seeing) and magicked SMFs (data_group=condor_MD04.V3_01_dg, full sample of ~400/filter loaded into PSPS Nov. 2011)
  • 1/11 full V3 nightly stack reprocessing with MD06 started. -- important note: these are now being run allowing 2 or more inputs to make a stack (down from 4 during the magic era) so outer skycells may have strange behavior that we need feedback on.
  • 1/17 majority of MD06 full reprocessing of nightly stacks on datastore. MD04 full V3 nightly stack reprocessing started.
  • 1/23 MD04.V3 full reprocessing of nightly stacks finished and pushed out to datastore.
  • 4/17 MD08.V3 restarted from scratch after raid crash in November/December with the new ipp-20120404 tag that will include new ubercal refcat (PS1.REF.20120325) and detrend improvements (dark modes, noise map, but not boost factors).
  • 4/25 MD08.GR0 reprocessed and distributed, refstacks made and distributed, staticsky run and distributed.
  • 6/01 MD09.GR0 chip->warp reprocessing started with new ops tag ipp-20120531 (improved REFCAT 20120524, ghost models, corner under-glow masking). seeing some trouble with new tag in z,i,r for XY67,71.
  • 6/03 MD09 g,y refstack started, nightly stack reprocessing running.
  • 6/04 MD09 z refstack started. g,y refstack and reftests and nightly stack pushed to distribution.
  • 6/06 MD09 r,i refstacks started. z refstack, 90% nightly stacks out to datastore.
  • 6/07 MD09 refstacks finished and distributed, staticsky on refstack finished, 90% of camera SMF and nightly stacks that are possible distributed.
  • 6/18 MD07 (pre-ubercal) staticsky run attempted, see notes below
  • "7/05" new MD10 refstack started, see MD10.refstack.20120705
  • "7/29" MD09.GR0 XY67,71 chip problem (~48 exposures) not fixable so set to poor quality, remaining camera SMF and nightly stacks processed and distributed.
  • "8/03" new MD01 refstack started, like new MD10 more of continuation of GR0 (separate wiki pages) -- MD01.refstack.20120804?
  • 08/31 MD09.refstack.20120603 replaced with MD09.refstack.20120831? (run with ipp-20120802 tag) to fix the weight pixel arc region of lower sensitivity
  • 09/27 MD02.refstack.20120927 to replace 2011 season refstack
  • 11/01 MD03.refstack.20121101 to replace 2011 season refstack
  • 11/25 MD04.refstack.20121125 to replace 2011 season refstack
  • 12/06 MD05.refstack.20121202 to replace 2011 season refstack
  • 12/21 MD06.refstack.20121221 to replace 2011 season refstack
  • 01/09/13 MD07.refstack.20130102 to replace 2011 season refstack

* Feedback is important: We would not want to continue full reprocessing of the MD fields if there has been a gross oversight or problem for a particular science interest. Please let us know of any issues you discover with example details. For example

  • tessellation and/or offsets issues in any of the fields
    • corner of skycell.055 seems to be regularly masked (and in neighbors with the overlap)
    • fields with offsets need to include sufficient images in old and new pointings (i.e., MD10), for the magic reference stack at least.
  • any issues with masking, mask values
    • been report of possible ones in early MD10 but still need details
  • depth/quality of the stacks, reprocessed images in the new format
  • any missing skycells, SMFs, other data products -- in particular MD10 which has been the only MD field full reprocessed and magicked for release of SMFs, warps.
    • MD10.refstack.20110722, MD10.refstack.20110821 r-band has two edge skycells (041,051) with poor quality and failed
    • MD10.refstack.20110722 y-band skycell.048 had rejection problem, fixed with MD10.refstack.20110821
  • MD10.refstack.20110722 replacement with proper uncompressed MD10.refstack.20110821 for diffims
  • with magic turned off now, MD04+MD06 have 2+ warp inputs for nightly stack creation to test how well boundaries come out and has fault/quality issue with 2-3 input cases showing some inputs have incorrect PSF size values measured/passed. rare (~<1%) but should be looked into (ie stack_id=609561)
  • adding quality column to ippMonitor for evaluating missing stack skycells would be useful to have on stack processed skyfile

DRAVG analysis/feedback

Where to find the data:

MD.V3 Tessellation

The new V3 tessellations are defined at PS1_Tessellations with the following general properties:

  • 0.25"/pix (from natural ~0.258"/pixel, previously warped to 0.2"/pix up through V2)
  • 10 x 10 skycell grid (previously 12 x 12)
  • +x is West (flipped from before +x was East), +y is still North
  • 320 pixel (80") overlap (from 300 pixels, 60", in V2)
  • revised centers (if necessary) -- see relevant MD section for more details and the MDS field table (http://ps1sc.ifa.hawaii.edu/PS1wiki/index.php/Medium_Deep_Survey#Actual_Field_Centeres_-_2011-02-21) but in summary
    • MD10.GR0: from (351.979 -0.433) to (352.312, -0.433) however an additional offset looks like it should have been applied to cover more of the earlier observation dithers.
    • MD07.GR0: from (213.146, +53.417) to (213.454, +53.243) to best cover as much of the earlier observation dithers as possible.
    • MD08.GR0: very slight offset in RA to the W to just align with the MDS field pointing.
    • MD09.GR0: from (333.688, +0.283) closer to the current pointing of (334.188, +0.283) however an addition offset was added to move the center about half a skycell to the E to (333.980, +0.283) in order to cover as much of the earlier observation dithers as possible.
  • All fields should be looked at to see if there are any problems with the new tessellation and dither pattern.
  • MD07, MD09 will have a preliminary nightly stack made for testing the revised center as well as MD06 for NGC4258.

Reference versus Deep Stack

A requirements description for the needs in the different types of stacks would be useful to summarize.

Reference stacks -- primarily for difference imaging for transient detection and necessary for magic destreaking.

  • Magic destreaking: a fairly clean reference stack (uses the convolved stack) with as much spatial area coverage as possible to enable release of non-stack data products. Particularly important is inclusion of older data on different pointings and of course a fairly balanced dither set.
  • Transient detection: a balance slanted towards best PSF but with some depth (1-1.5 magnitudes over nightly stacks) and also a fairly balanced dither set. better with more recent data to limit AGN/variable star variations.

Deep stacks -- for deep static sky science.

  • As deep as possible.. maximum FWHM allowance? minimum exposure time? specific request for at least ~10ks has been requested, with best FWHM possible and also a fairly balanced dither set..

See summary for each MD field reference stack for details on specifics for that field. Due to processing+time constraints/early mistakes, an attempt has been made to merge the similar criteria under the refstack label.

Since magic destreaking uses the convolved stack, looking into a quickstack version of the unconvolved stacks for the transients reference and deep stack that will be faster to make with more inputs.

For cases when a stack does not have a good sample of exposures with a reasonably small PSF FWHM, it should be possible to generate a reference stack with the fewer 'best' exposures for just SSdiffs and transient detection. MD02-g may be such a case.

For fields with shifted centers from offset earlier exposures, inclusion of enough exposures to provide a reference stack for magic is necessary in order for data from those associated chips/skycells to be distributed (being older data, may not have the optimum quality). Mixing older and newer data may also be a detriment.

Related to deep stacks, there needs to be enough inputs to obtain minimum depth that is as pixel contiguous as possible (see MD04 deep stack tests).

How much of a varying IQ can go into a stack before ppSub has difficulty modeling the variations within a skycell?

Deep stacks on hold until finish investigating overly large PSF in the convolved stacks dependent on the number of input warps.

Without need for magic for releasing older data, should focus on more recent data going into the refstacks

post-Magic January 2012

Reprocessed Data Products

Chip/Warp

Currently processed but not distributed due to disk space load. Can be obtained via the Postage Stamp Server.

Camera/SMFs

Fully processed and distributed.

Nightly Stacks

All nights reprocessed and stacked into a nightly stack using the current nightly stack configurations.

  • should attempt to make stack with 2 or more input warps now
  • some nights have >8 exposures, often in marginal conditions. Choices could/should be made in those cases (need to add details on current settings)

Difference Images

Need to finish the convolution choice (input stack or reference stack) before started. (Need to outline interests for these if any)

Static Sky

What to run on, reference stack and/or deep stack (TBD)

Summary Results Posters at PS1SC 2012 Durham Meeting

Summary plan for finishing MD.GR0 through end of 2012

(under evaluation -- last update Jan. 11, 2013)

MD reference stack deep stack nightly stack (all exposures into V3) ref phot deep phot nightly phot camera SMFs for PSPS
01 20110820 (A) x done (mixed) staticsky+skycal 20120805 x x done (mixed) SMF+ref phot TBD
02 20110907 x done (mixed) staticsky+skycal 20120805 x x done (mixed) SMF+ref phot TBD
03 20111010 x done (mixed) staticsky+skycal 20120805 x x done (mixed) SMF+ref phot TBD
04 20111012 deeptest.20120727 done staticsky+skycal 20120805 staticsky+skycal 20120806 TBD done SMF+ref+deep+nightly phot TBD (secondary)
05 20111014 x done (mixed and full) staticsky+skycal 20120805 x x done (mixed) SMF+ref phot TBD
06 20111122 x done staticsky+skycal 20120805 x x done SMF+ref phot TBD
07 20111106 x done staticsky+skycal 20120805 x x done SMF+ref phot TBD
08 20120422 x done staticsky+skycal 20120805 x x done SMF+ref phot TBD
09 20120603 (replaced by 20120831) deeptest.20120727 done staticsky+skycal 20120805 staticsky+skycal 20120806 done done SMF+ref+deep+nightly phot (priority)
10 20110821 (A) x done staticsky+skycal 20120805 x x done SMF+ref phot TBD
M31 (add to MD.GR0?) include/make/redo? x ? if made ? ? ? ?
STS (add to MD.GR0?) 20120531 (MOPS needs revision) x ? if made ? ? ? ?

Comments:

  • TBD=To Be Done/Decided by priority, X=not planned, ?=unknown at this time
  • the date entries are part of the data_group name to access directly via the PSS or Datastore, i.e., MD01.refstack.20110820, MD09.deeptest.20120727
  • (A) new 2012 versions exist for these refstack now but staticsky/skycal not necessarily available
  • finish reprocessing all exposures to V3 for warp/nightly stack for MD01+02+03+05 =840+609+1286+1115 = 3850 (2-3 full days of --only-- MD processing)
    • MD05 mixed -- old and new processing, so magic mix and photometry catalog mix. see specific section for detail.
  • out-of-season/anti-yearly refstacks, yearly refstack: on hold for MD.GR1 (would need to update/reprocess large numbers of exposures to do).
  • iq/psf stack on hold for MD.GR1 (like out-of-season, would required many updates) -- should do for MD04,MD09 and requires development/code time.
  • deep stack for MD04,09 for MD.GR0 only, all others on hold for MD.GR1 (many many updates..).
  • ubsercal of pixels start with MD08 in the date sequence, finished processing of MD01,02,03,05 exposures into V3 will also be ubercal'd (including reprocessing of any needed to make the complete nightly stacks)

Questions:

  • background restored warps/stacks?
  • SMF loaded into PSPS limited to just the ones used in the refstack (or deepstack if available) or all SMF?
  • MD01,10 have new refstack for this season, need to decide if using for phot+PSPS (limited set of exposures reprocessed recently to make)? Or use as update test for PSPS?
  • MD09.refstack.20120603 suffers from the weight image problem making an arc of lower sensitivity, but warps are still available for doing the deep stacks. 4-6 hrs/skycell to run, ~300 skycells, deepstack pantasks typically has 30 nodes, so ~2-3 days processing time, redo? -- yes, since MD09 is a new test/comparison set with SAS -- done and date updated under refstack

MD10.GR0

Status:

  • reference stack (revised) processed and distributed. All exposures since April 2009 reprocessed and distributed except for SSdiffs. SSdiffs being held back until ppSub auto convolution choice is tested on.
  • 8/2012: to be replaced with MD10.refstack.20120705, then replaced with MD10.refstack.20120804.

Started for OC142 (July 15, 2011). Start with current statuswith a new reference stack and full sample of reprocessed exposures as an initial reprocessing version.

  • z,y-band data processed prior to full Dark model improvements (higher background makes effect insignificant). gri-band processed after Dark model improvements.
  • same photometry reference catalog as MD have previously been using.
  • stack-stack diffims still using forced convolution of reference stack to input images

MD10.V3 center shifted to the current observation pointing even though there are earlier data that has offset dithered observations that populate the 0X8-0X9 skycells. Not as bad as MD07, but the revised reference stack may not provide for the outer skycells very well. Afterwards it was discovered the full extent of the previous observations wasn't quite accounted for, shown by these exposure plots with the MD10.V3 overlay and compared to a recent nightly stack set of exposures. This may need to be shifted in the next iteration.

MD10.V3-g shifted to the current pointing center for MD10. MD10.V3-r shifted to the current pointing center for MD10. MD10.V3-i shifted to the current pointing center for MD10.
MD10.V3-z shifted to the current pointing center for MD10. MD10.V3-y shifted to the current pointing center for MD10. MD10.V3-i nightly stack exposures from 9/29/11

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter N total N (FWHM<upper limit GPC1 pixels)
y 136 all
z 246 136 (<5.0 pixels), then finished reprocessing all
i 256 149 (<7.0 pixels), then finished reprocessing all
r 253 159 (<7.0 pixels), then finished reprocessing all
g 242 118 (<7.0 pixels), then finished reprocessing all
  • also initially attempted to include some rough cuts by zeropoint and background (in 20110722) but after reprocessing all exposures wouldn't be included in the stacktool selection based solely on basically FWHM major. (improvements to selection and rejection for a future test)

MD10.V3 reference stack

  • label: MD01.refstack.20110820
  • data_group: MD01.refstack.20110820

Initial goal was to maximize depth while minimizing the PSF FWHM with enough inputs (4 or more) to produce a stack for all skycells covered, in particular

Compared to the cumulative metrics at http://ipp0022.ifa.hawaii.edu/ps1sc/metrics/range_2010-05-13_current/

Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20110722 (top blue), psfrefstack.20110814 (middle blue) and refstack.20100721 V2 (bottom blue). The unmatched blue/red overlap for refstack.20110722 FWHM major<6.5 pixels for exposures in the refstack is from using also low zeropoint and bright background cuts. Psfrefstack used <5.1 pixels. MD10.refstack.20110722 g-band FWHM histogram versus V2
The same refstacks with the selected exposures over time MD10.refstack.20110722 g-band FWHM, ZPT, BG vs MJD
Sample skycell coverage map by good_frac with N input warps, N rejected warps, dtime_script for making stack, and if quality error that rejects entire skycell. The ~130 input warps provided enough of a sample for the outer edge skycells from earlier exposures. MD10.refstack.20110722 g-band stack skycell summary map - n warps in, rejected, good_frac, dtime

In tracking down an odd skycell (048, y-band), an odd rejection case was highlighted and prompted a modification to the rejection criteria based on the RMS of the stamps (see r32170). This may help reduce large number of rejections in future refstacks but likely needs some tuning (currently still rejecting too many?). Using the PSF FWHM cuts in the DEEP_STACK reduction also helps reject strange input warps as well.

Initially the stack processing was run without a reduction type and thus with just defaults that compresses the images (version 20110722gri, 20110715y, 20110718z) along with a test sample to produce the best PSF focused set (MD10.psfrefstack.20110814) and a test best deep stack (just skycell.055, i-band, not distributed). This is not optimal and a re-run was done with the DEEP_STACK reduction type (version 20110821) and should be the one used. The different reduction types include:

NULL (none/default values used from recipes/ppStack.config unless defined in gpc1/ppStack.config):

  • PSF.INPUT.MAX F32 NAN
  • PSF.INPUT.CLIP.NSIGMA F32 NAN
  • OUTPUT.NOCOMP BOOL FALSE
  • OUTPUT.LOGFLUX BOOL FALSE
  • OUTPUT.REPLICATE BOOL TRUE

DEEP_STACK:

  • OUTPUT.NOCOMP BOOL TRUE
  • OUTPUT.LOGFLUX BOOL FALSE
  • PSF.INPUT.MAX F32 7.5 (should add to PS1_REFERENCE as the reduction to use for refstacks)
  • PSF.INPUT.CLIP.NSIGMA F32 5.0 (should add to PS1_REFERENCE as the reduction to use for refstacks)

PS1_REFERENCE:

  • OUTPUT.NOCOMP BOOL TRUE
  • OUTPUT.LOGFLUX BOOL FALSE
  • OUTPUT.REPLICATE BOOL TRUE

NIGHTLY_STACK:

  • OUTPUT.NOCOMP BOOL FALSE
  • OUTPUT.LOGFLUX BOOL FALSE

THREEPI_STACK: (for comparison)

  • PSF.INPUT.MAX F32 12.0
  • OUTPUT.NOCOMP BOOL FALSE
  • OUTPUT.LOGFLUX BOOL TRUE

QUICKSTACK_DEEP METADATA (testing unconvolved stacks version only, similar to QUICKSTACK)

  • CONVOLVE BOOL FALSE
  • COMBINE.REJ F32 3.5 # Rejection threshold in combination (sigma). tune?
  • COMBINE.DISCARD F32 0.2 # Discard fraction for Olympic weighted mean. tune?
  • OUTPUT.NOCOMP BOOL TRUE
  • OUTPUT.LOGFLUX BOOL FALSE
  • PSF.INPUT.MAX F32 8.0 # tune?
  • PSF.INPUT.CLIP.NSIGMA F32 5.0

END

A summary table of the reprocessed reference stacks and the former V2 where center skycell for V3=skycell.055, V2=skycell.078. (need to add FWHM upper limit used)

version label filter center skycell input warps (rejected) center skycell FWHM (major)
MD10.refstack.20110821 V3 y 63(11) 0.96"
MD10.refstack.20110715 V3 error y 93(6) 0.99"
MD10.psfrefstack.20110814 V3 error y 41(3) 0.9"
MD10.refstack.20100721 V2 y 28(1) 1.06"
MD10.refstack.20110821 V3 z 72(0) 1.02"
MD10.refstack.20110718 V3 error z 174(6) 1.09"
MD10.psfrefstack.20110814 V3 error z 44(4) 0.98"
MD10.refstack.20100721 V2 z 23(0) 1.09"
MD10.refstack.20110821 V3 i 75(1) 0.99"
MD10.refstack.20110722 V3 error i 132(6) 1.07"
MD10.psfrefstack.20110814 V3 error i 46(2) 1.00"
MD10.deeprefstack.20110814 V3 error i 217(11) 1.08"
MD10.refstack.20100721 V2 i 18(1) 1.32"
MD10.refstack.20110821 V3 r 76(0) 1.28"
MD10.refstack.20110722 V3 error r 134(2) 1.33"
MD10.psfrefstack.20110814 V3 error r 47(3) 1.28"
MD10.refstack.20100721 V2 r 19(1) 1.46"
MD10.refstack.20110821 V3 g 79(3) 1.4"
MD10.refstack.20110722 V3 error g 123(0) 1.41"
MD10.psfrefstack.20110814 V3 error g 45(0) 1.31"
MD10.refstack.20100721 V2 g 22(0) 1.57"

Using the MD10.refstack.20110722 and related set, histograms and mag-magerr plots for the reference stack, deeprefstack test, psfrefstack, typical nightly stack and V2 refstack are compared. The re-run MD10.refstack.20110821 will go a little deeper than the psfrefstack version. Sample psphot run on MD10.refstack.20110722, MD10.psfrefstack.20110814, deepstacktest, sample nightly stack and the previous V2 MD10.refstack.20100721
A representative skycell mapping of the new tessellation and good_fraction coverage. Nightly stacks will have similar, if not more, trouble with covering the corner edge skycells: 012, 017, 021, 028, 071, 078, 082, 087. Sample MD10.refstack.20110821 skycell map of good_frac, N warp inputs, N rejections, dtime, and faults.

Other plots that may be good to see

  • Mask fractions vs V2
  • mapping of all exposures WRT new tessellation to check maximum coverage given some fields offset and others have had extra dither patterns (i.e., like MD10)

Troubled skycells for MD10.refstack.20110821:

  • z: 012, 018,021,029,071,079,082 quality issue with edge skycells
  • y: 012, 018,021,029,071,079,082 quality issue with edge skycells
  • r: 012, 017,021,028,041,051,071,078,082,087 quality issue with edge skycells except 041, 051 should be recovered
  • i: 012,017,021,028,071,078,082,087 quality issue with edge skycells
  • g: 012, 017, 021, 071,078, 082 quality issue with edge skycells

MD10.GR0 reprocessed exposures

The full set of exposures since April 2009 were reprocessed until the following tags.

  • label: MD10.GR0
  • data_group: MD10.GR0.20110715 = y
  • data_group: MD10.GR0.20110718 = z
  • data_group: MD10.GR0.20110720 = g
  • data_group: MD10.GR0.20110721 = r
  • data_group: MD10.GR0.20110722 = i
  • dist_group: MD10.GR

Distribution

Planned distribution of data products:

NOTE: full chip and camera products will not be included as datastore release bundles unless there is interest

Available data products on the datastore:

  • warp (full): done
  • nightly stack (full): done
  • warp-stack diffim (full, cleaned): done, done
  • camera/SMF: done
  • stack-stack diffims: TBD - are still awaiting the ppSub final version of the auto convolution direction choice addition

Data bundles will unpack in the following ways

  • warp-stack diffims: all under gpc1/MD10.GR0/2011/08/09/MD10.V3/skycell.0NN -- very large directory *

Postage-stamp Server

Access to be tested still.

Missing/Failed Data

Summary of any failed data products during processing.

  • two skycell.021(and?) failed to produce a nightly stack (date?)

Cleanup Cycle

IPP (for faster postage stamp requests):

stage time-frame
chip similar timescale (V0-V2)
camera not cleaned
warps similar timescale (V0-V2)
stacks similar timescale (V0-V2)
WSdiff similar timescale (V0-V2), must wait for MOPS postage stamp requests to finish, but not this round
SSdiff similar timescale (V0-V2)
refstack not cleaned

Datastore:

stage time-frame
chips none distributed
camera none distributed
warps TBD
stacks TBD
WSdiff TBD
SSdiff TBD
refstack TBD

Reprocessed Status


MD01.GR0

Status:

  • Refstack and sample nightly stacks finished and distributed.
  • 08/2012: to be replaced with MD01.refstack.20120803

Started for OC143 (August 13, 2011) with a new reference stack and small sample of reprocessed exposures released as nightly stacks. Due to a need of rebalancing the reprocessing load, the reprocessed sample includes only those exposures that may be used in the reference stack.

MD03 does have some offset dither exposures illustrated in the sample g-band V3 plot. MD01.V3-g exposure set

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter N total N (FWHM<upper limit GPC1 pixels)'"
y 178 106 (<5.0 pixels)
z 377 214 (<5.0 pixels)
i 351 172 (<5.0 pixels)
r 330 120 (<5.0 pixels)
g 340 124 (<5.5 pixels)

MD01.V3 reference stack

  • label: MD01.refstack.20110820
  • data_group: MD01.refstack.20110820
Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20110820 (top blue) and refstack.2010812 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label filter center skycell input warps (rejected) center skycell FWHM (major)
MD01.refstack.20110820 V3 y 52(1) 0.84"
MD01.refstack.20110813 V3 error y 106(1)
MD01.refstack.20100812 V2 y 21(3) 1.04"
MD01.refstack.20110820 V3 z 58(3) 0.87"
MD01.refstack.20110813 V3 error z 115(2)
MD01.refstack.20100812 V2 z 60(2) 1.12"
MD01.refstack.20110820 V3 i 74(3) 1.00"
MD01.refstack.20100812 V2 i 50(0) 1.13"
MD01.refstack.20110820 V3 r 54(7) 1.10"
MD01.refstack.20100812 V2 r 56(1) 1.24"
MD01.refstack.20110820 V3 g 65(7) 1.20"
MD01.refstack.20100812 V2 g 54(5) 1.24"

with the approximate FWHM (major) as the PSF FWHM from a center field skycell after running psphot on the stacks. The MD01.refstack.20110813 V3 error was a run without the DEEP_STACK reduction

Distributed under ps1-md-refstack -- stack images only.

Troubled skycells: none on initial inspection for refstack.20110820

  • 087-y,z (MD01.refstack.20110813 V3 error): quality issue rejected, edge skycell.
  • 086-z (MD01.refstack.20110813 V3 error): quality issue rejected, edge skycell.

Data volume on disk:

Processing time:

MD01.GR0 reprocessed sample

  • label: MD01.GR0
  • data_group: MD01.GR0.20110812 = y
  • data_group: MD01.GR0.20110815 = z
  • data_group: MD01.GR0.20110816 = i
  • data_group: MD01.GR0.20110817 = r
  • data_group: MD01.GR0.20110818 = g
  • dist_group: MD01.GR

Distributed under ps1-md-GR -- currently only a small sub-sample of nightly stack images only. SSdiff sample to be processed still.

Data volume on disk:

Processing time:

MD01.GR0 reprocessed remaining set

Chip->warp including camera SMF for all filters grizy:

  • label: MD01.GR0.20121122
  • data_group: MD01.GR0.20121122

Nightly stack for all filters grizy::

  • label: MD01.GR0
  • data_group: MD01.GR0.20121122.nightstack.<obsdate>
  • data_group if warps processed earlier but stack made with new tag: MD01.GR0.201211xx.nightstack.<obsdate>

MD02.GR0

Status:

  • Refstack and sample nightly stacks finished and distributed.
  • 10/5/2012: to be replaced with MD02.refstack.20120927 -- diffims 20121001 onwards will use.

Started for OC144 (September 12, 2011) with a new reference stack and a small sample of reprocessed exposures to release nightly stacks as well. Due to a need of rebalancing the reprocessing load, only a minimal amount of processing required to make the reference stack was done.

MD02 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many are single exposure coverage and won't typically be enough for a reference stack skycell. While warps will extend to some of the outer skycells, the nightly stack distribution will define what is likely for the skycells released due to magic. 012-017, 21-28, 31-38, 41-48, 51-58, 61-68, 71-78, 82-87 MD02.V3-g exposures

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter N total N (FWHM<upper limit GPC1 pixels)
y 147 85 (<6.0 pixels)
z 305 101 (<4.5 pixels)
i 266 117 (<5.5 pixels)
r 215 118 (<6.5 pixels)
g 218 121 (<7.0 pixels)

MD02.V3 reference stack

  • label: MD02.refstack.20110907
  • data_group: MD02.refstack.20110907
MD02.refstack.20110907 g-band reprocessed exposure FWHM histogram versus V2
Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20110907 (top blue) and refstack.20101126 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label filter center skycell input warps(rejected) center skycell FWHM (major)
MD02.refstack.20110907 V3 y 58(3) 1.01"
MD02.refstack.20101126 V2 y 47(3) 1.02"
MD02.refstack.20110907 V3 z 67(5) 0.90"
MD02.refstack.20101126 V2 z 84(10) 1.03"
MD02.refstack.20110907 V3 i 61(2) 1.00"
MD02.refstack.20101126 V2 i 102(15) 1.16"
MD02.refstack.20110907 V3 r 63(6) 1.15"
MD02.refstack.20101126 V2 r 67(15) 1.10"
MD02.refstack.20110907 V3 g 64(6) 1.32"
MD02.refstack.20101126 V2 g 71(24) 1.26"

with the approximate FWHM (major) as the PSF FWHM from a center field skycell after running psphot on the stacks.

Troubled skycells: none on initial inspection. 040 in g,r oddly seems to have enough coverage to make a stack.

Distributed under ps1-md-refstack -- stack images only.

Data volume on disk:

Processing time:

MD02.GR0 reprocessed sample

  • label: MD02.GR0
  • data_group: MD02.GR0.20110902 = y
  • data_group: MD02.GR0.20110903 = z
  • data_group: MD02.GR0.20110906 = g
  • data_group: MD02.GR0.20110905 = r
  • data_group: MD02.GR0.20110904 = i
  • dist_group: MD02.GR

Distributed under ps1-md-GR -- sample on nightly stack images only. SSdiff sample still to be run.

A typo in the loading script set the workdir to be MD01 rather than MD02 for all filter data groups (see string below) but will not cause any trouble other than the directory name in mirroring data

/gpc1/MD02.GR0/MD01.2011090*/MD02.V3/skycell.*/*unconv.fits

Data volume on disk:

Processing time:

  • chip-warp:
  • refstack: 8-9000s/skycell for the ~60 input warps, on averages looks longer than the previous refstack MD02.refstack.20101126 (to be investigated)
  • night stack: order 800-1200s/skycell

MD02.GR0 reprocessed remaining set


MD03.GR0

Status: refstacks and sample nightly stacks finished and distributed

  • 10/3: Sub-sample of y and z-band exposures through warp, sample nightly stacks made. y-band refstack in progress.
  • 10/4: r,i through warp. z-band refstack in progress
  • 10/6: g through warp, ri refstack in progress
  • 10/8: all refstacks queued for distribution, nightly stack sample as well.
  • 11/07/2012: to be replaced with MD03.refstack.20121101 -- diffims 20121026 onwards will use this

MD03.V3 refstack needed before new observations in OC146 (November 10).

MD03 does have some offset dither exposures illustrated in the sample g-band V3 plot, but many of the outer skycells are only covered by single to a few exposure without the 4 or more for a stack. MD03.V3-g exposures

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter N total N (FWHM<upper limit GPC1 pixels)'"
y 224 97 (<5.0 pixels)
z 526 150 (<4.2 pixels)
i 488 163 (<4.8 pixels)
r 377 160 (<4.9 pixels)
g 388 147 (5.5 pixels)
  • the number of exposures being reprocessed have been increased from that in MD01,02 in order to at least attempt to match the input warp numbers in the previous MD03.refstack (~100).

MD03.V3 reference stack

  • label: MD03.refstack.20111010
  • data_group: MD03.refstack.20111010
MD03.refstack.20111010 g-band reprocessed exposure FWHM histogram versus V2
Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20111010 (top blue) and refstack.20101202 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label filter center skycell input warps(rejected) center skycell FWHM (major)
MD03.refstack.20111010 V3 y 108(7) 0.91"
- MD03.reftest.20111011 V3 y 26(2) 0.83"
- MD03.reftest.20111011 V3 y 57(2) 0.87"
old MD03.refstack.20101202 V2 y 75(12) 1.13"
MD03.refstack.20111010 V3 z 125(18) 0.85"
- MD03.reftest.20111011 V3 z 19(1) 0.84"
- MD03.reftest.20111011 V3 z 60(2) 0.84"
old MD03.refstack.20101202 V2 z 101(27) 1.04"
MD03.refstack.20111010 V3 i 116(7) 0.97"
- MD03.reftest.20111011 V3 i 24(0) 0.98"
- MD03.reftest.20111011 V3 i 67(5) 0.97"
old MD03.refstack.20101202 V2 i 108(17) 1.12"
MD03.refstack.20111010 V3 r 123(4) 1.05"
- MD03.reftest.20111011 V3 r 24(1) 1.02"
- MD03.reftest.20111011 V3 r 64(3) 0.97"
old MD03.refstack.20101202 V2 r 115(44) 1.10"
MD03.refstack.20111010 V3 g 114(5) ~1.08" (looked at 054,053 as well)
- MD03.reftest.20111011 V3 g 17(2) 1.03"
- MD03.reftest.20111011 V3 g 67(2) 1.08"
old MD03.refstack.20101202 V2 g 90(29) 1.13"
  • overshot the number of input warps compared to V2. The average PSF FWHM_major seems much better than expected from the histograms (above) of input exposures.

Sample psphot run on MD03.refstack.20111010 and the previous V2 MD03.refstack.20101202

Troubled skycells: none on initial inspection. Far outer edges with few dithers may/may not have successful skycells. Sample skycell coverage maps for V2 and V3 with good_frac with N input warps, N rejected warps, dtime_script for making stack, and flag of any quality error that rejects an entire skycell (ERR). The ~110-125 input warps provided a sample for some of the far outer skycells from earlier dithers to be covered still.

MD03.refstack.20101202 V2 g-band stack skycell summary map - n warps in, rejected, good_frac, dtime MD03.refstack.2011010 V3 g-band stack skycell summary map - n warps in, rejected, good_frac, dtime

Distributed under ps1-md-refstack

Data volume on disk: ~330GB

Processing time summary: 12-18000 sec for 100-125 input warps. ~6 days to reprocess a ~50% of exposures through warp, create reference stack, and distribute (along with LAP and normal nightly science).

MD03.GR0 reprocessed sample

  • label: MD03.GR0
  • data_group: MD03.GR0.20111001 = y
  • data_group: MD03.GR0.20111002 = z
  • data_group: MD03.GR0.20111003 = i
  • data_group: MD03.GR0.20111004 = r
  • data_group: MD03.GR0.20111005 = g
  • dist_group: MD03.GR

Distributed under ps1-md-GR -- sample of nightly stack images only. SSdiff sample still to be run. Magicked camera SMFs to be done.

Data volume on disk:

Processing times:


MD04.GR0

Status: Refstacks and sample nightly stacks finished and distributed. Very deep stack tests in progress. Full reprocessing of all possible nightly stacks finished and to datastore.

  • 10/2 Starting with i-band rather than y for the sub-sample reprocessing exposures through warp.
  • 10/4 z-band through warp done. testing very deep >150 stack on i-band. r-band running through warp
  • 10/5 gy running through wrap
  • 10/6: izrgy refstacks ready to queue
  • 10/8: i-band refstack finished, z-band running
  • 10/9: r-band running, g-band running
  • 10/10: remaining y refstack finishing up
  • 10/16: refstack and sample nightly stack set finished and distributed
  • 01/17/12: full reprocessing of all possible nightly stacks in V3 started (finished 1/23 and to datastore)
  • 11/25/2012: refstack to be replaced with MD04.refstack.20121125 -- diffims onwards will use this

New observations aren't planned until OC147 (December 9), but need it for comparison analysis ASAP.

Retired MD04.altstack in a pre-V3 tessellation from last November (20101102). Pre-V3 MD04.altstack warps have been purged (no longer updatable) but the stacks are still available for any comparison.

All past g-band exposures in the V3 tessellation (other filters nearly identical). MD04.V3-g exposure set

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter N total N (FWHM<8 pixels) N (FWHM<upper limit GPC1 pixels)'"
y 209 177 146 (<5.5 pixels)
z 343 321 196 (<5.2 pixels)
i 428 392 208 (<5.0 pixels)
r 357 327 203 (<5.6 pixels)
g 364 313 206 (<6 pixels)
  • the number of exposures being reprocessed have been increased from that in MD01,02 in order to at least attempt to match the input warp numbers in the previous MD03.refstack (~100). This field has also been the focus of deep stack tests, so the initial reprocessed set will attempt to be double the number from past versions (~200).

MD04.V3 reference stack

  • label: MD04.refstack.20111012 (~100)
  • data_group: MD04.refstack.20111012 (~100)
Histograms of the FWHM major and minor from camera stage for reprocessed grizy exposures (red) and refstack.20111012 (top blue) and refstack.20101102 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label filter center skycell input warps center skycell FWHM (major)
MD04.refstack.20111012 V3 y 107(0) 0.90"
- MD04.reftest.20111011 V3 y 20(0) 0.88"?
- MD04.reftest.20111011 V3 y 58(5) 0.87"
old MD04.refstack.20101102 V2 y 100(14) 1.13"
MD04.refstack.20111012 V3 z 109(2) 0.90"
- MD04.reftest.20111011 V3 z 21(1) 0.92"?
- MD04.reftest.20111011 V3 z 52(1) 0.8"
old MD04.refstack.20101102 V2 z 105(8) 1.14"
MD04.refstack.20111012 V3 i 92(11) 0.88"
- MD04.reftest.20111011 V3 i 26(2) 0.93"?
- MD04.reftest.20111011 V3 i 55(5) 0.98"
- MD04.refstack.20111011 V3 (deep stack test, bad label) i 206(16) 1.08"
- MD04.deeptest.20111012 V3 i 308(17) 1.16"
old MD04.refstack.20101102 V2 i 108(15) 1.22"
MD04.refstack.20111012 V3 r 128(9) 1.12"
- MD04.reftest.20111011 V3 r 26(0) 1.13"?
- MD04.reftest.20111011 V3 r 57(7) 1.06"
old MD04.refstack.20101102 V2 r 92(8) 1.24"
MD04.refstack.20111012 V3 g 113(8) 1.21"
- MD04.reftest.20111011 V3 g 22(0) 1.29"?
- MD04.reftest.20111011 V3 g 60(0) 1.34"?
old MD04.refstack.20101102 V2 g 101(13) 1.21"
  • was an minor error in the cutoff for i,r: a fewer number in i-band and few more in r-band compared to others. Should be okay for basic refstack, and planned deeper stack should offset any deepstack concerns.

Troubled skycells: none on initial inspection. Far outer edges with few dithers may/may not have successful skycells. Sample skycell coverage maps for V2 and V3 with good_frac with N input warps, N rejected warps, dtime_script for making stack, and flag of any quality error that rejects an entire skycell (ERR). The ~110 input warps provided a sample for some of the far outer skycells from earlier dithers to be covered still.

MD04.V3 deep stack

  • label/data_group:
    • MD04.refstack.20111011 (~200) for skycell.055, i-band
    • MD04.deeptest.20111012 (~300) for skycell.055, i-band
    • MD04.deeptest.20111015 (~300) full FPA, i-band
    • MD04.deeptest.20111017 (~300) full FPA, g-band

Histograms of the FWHM major and minor from camera stage for reprocessed exposures (red) and deeptest.20111012 (top blue, ~300 inputs) and refstack.20101102 V2 (bottom blue, ~100 inputs). Any unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.
  • platescale for the camera exposure fwhm_major is ~0.258 "/pixel and processing is limited to FWHM<12 pixels. For skycells is now 0.25 "/pixel and for ref/deep stack is limited to 7.5 pixels currently. May want to raise to 8-10.

Distribution:

Test skycell.055 distributed for

MD04.GR0 reprocessed sample

Initial set of chip->warp (including camera) and nightly stack sample:

  • label: MD04.GR0
  • dist_group: MD04.GR
  • data_group: MD04.GR0.20111002 = i
  • data_group: MD04.GR0.20111003 = z
  • data_group: MD04.GR0.20111004 = r
  • data_group: MD04.GR0.20111005 = g
  • data_group: MD04.GR0.20111006 = y

Distributed on datastore under

  • ps1-md-GR: sample of nightly stack images only. SSdiff sample still to be run.
  • ps1-md-GR-cat: unmagicked camera SMFs with data_group MD04.GR0 released (dupe to the full magicked SMF release from condor processing that was magicked for PSPS loading Nov. 2011)

Heather/Serge successful reprocessing of chip-warp under condor: (more to be run with similar label and condor)

  • label: condor_MD04.V3_01
  • data_group: condor_MD04.V3_01_dg
  • dist_group: MD04.GR

Remaining nightly stack data_group: data_group of remaining sample of nightly stacks processed with ipp-20111222 tag and now allowing 2 or more input warps necessary to make stack (down from 4 during the magic era and typically the far outer skycells, but as a new case, will need to be looked at in more detail). Also added nightlystack.<obsdate> to make a group by night the observation made to help in tracking/monitoring (there will be a mix of data_groups with above MD04.GR0.201110xx earlier sample version)

  • data_group: MD04.GR0.20120117.nightlystack.<obsdate> = y
  • data_group: MD04.GR0.20120118.nightlystack.<obsdate> = z
  • data_group: MD04.GR0.20120119.nightlystack.<obsdate> = i
  • data_group: MD04.GR0.20120120.nightlystack.<obsdate> = r
  • data_group: MD04.GR0.20120121.nightlystack.<obsdate> = g
  • workdir: will now be under a similar directory gpc1/MD04.GR0/MD04.GR0.201201xx.nightlystack.<obsdate>

Distributed on datastore under

  • ps1-md-GR: remaining nightly stacks to be processed from the condor_MD04.V3_01_dg data_group. SSdiff sample still to be run.
  • ps1-md-GR-cat: magicked camera SMFs released (data_group condor_MD04.V3_01_dg, set used for PSPS loading Nov. 2011)

Staticsky run for ingestion into PSPS. Eric notes the RA_PSF, DEC_PSF are NULL in the CMFs. When loaded into DVO coordinates are set and will be okay in PSPS.


MD05.GR0

Status: Refstacks and sample nightly stacks finished and distributed.

  • 10/7: y-band chip-warp setup.
  • 10/8: yzi chip-warp run
  • 10/9: rg chip-warp run, yzi sample night stacks setup
  • 10/10: sample nightly stacks yzi finished, gr started, y refstack started
  • 10/16: refstack and sample nightly stack set finished and distributed
  • 10/05/2012: restarted processing remaining exposures into V3 -- will be somewhat a mess
  • 10/13: nightly stacks for remaining exposures finished (important to see details below).
  • 12/06/12: new refstack MD05.refstack.20121202 will replace this one for diffims in the 2012/2013 season

New observations planned and started OC147 (December 9).

All past g-band exposures in the V3 tessellation (other filters nearly identical). MD05.V3-g exposure set

Exposures available with preliminary FWHM cut for initial reprocessed sample to create the refstack:

Filter N total N (FWHM<8 pixels) N (FWHM<upper limit GPC1 pixels)'"
y 338 224 174 (<5.0 pixels)
z 383 325 164 (<4.5 pixels)
i 406 343 185 (<5.0 pixels)
r 416 386 168 (<5.2 pixels)
g 426 379 163 (<5.7 pixels)

MD05.V3 reference stack

  • label: MD05.refstack.20111014 -- try to get out by 10/14
  • data_group: MD05.refstack.20111014
Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20111014 (top blue) and refstack.20110121 V2 (bottom blue). The unmatched blue/red overlap for V2 are due to ones included/rejected in the selected reprocessing.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version label filter center skycell input warps center skycell FWHM (major)
MD05.refstack.20111014 V3 y 133(4) 0.90"
- MD05.reftest.20111011 V3 y 22(0) 0.86"?
- MD05.reftest.20111011 V3 y 69(3) 0.85"
old MD05.refstack.20110121 V2 y 100(25) 1.16"
MD05.refstack.20111014 V3 z 132(4) 0.84"
- MD05.reftest.20111011 V3 z 25(0) 0.88"?
- MD05.reftest.20111011 V3 z 70(9) 0.83"
old MD05.refstack.20110121 V2 z 100(28) 1.01"
MD05.refstack.20111014 V3 i 147(17) 1.03"?
- MD05.reftest.20111011 V3 i 19(0) 0.99"?
- MD05.reftest.20111011 V3 i 63(8) 1.11"?
old MD05.refstack.20110121 V2 i 100(14) 1.24"
MD05.refstack.20111014 V3 r 156(15) 1.15"
- MD05.reftest.20111011 V3 r 23(1) 1.23"?
- MD05.reftest.20111011 V3 r 61(1) 1.12"
old MD05.refstack.20110121 V2 r 95(37) 1.31"
MD05.refstack.20111014 V3 g 154(8) 1.24"
- MD05.reftest.20111011 V3 g 21(5) ?1.28"?
- MD05.reftest.20111011 V3 g 70(5) 1.21"
old MD05.refstack.20110121 V2 g 95(10) 1.38"

*

Error: Macro Image(MD05.201111xx_g_test.png,300px) failed
Attachment 'wiki:MD.GR0: MD05.201111xx_g_test.png' does not exist.
Error: Macro Image(MD05.201111xx_i_test.png,300px) failed
Attachment 'wiki:MD.GR0: MD05.201111xx_i_test.png' does not exist.

Troubled skycells?

MD05.GR0 reprocessed set

Initial sample set of chip-warp and sub-sample nightly stack reprocessing:

  • label: MD05.GR0
  • data_group: MD05.GR0.20111007 = y
  • data_group: MD05.GR0.20111008 = z
  • data_group: MD05.GR0.20111009 = i
  • data_group: MD05.GR0.20111010 = r
  • data_group: MD05.GR0.20111011 = g
  • dist_group: MD05.GR

Finished nightly stacks with past chip->warp reprocessing from 20111007,08,09,10,11 (pixels calibrated with synth catalog):

  • label: MD05.GR0
  • data_group: MD05.GR0.201210xx.nightstack.<obsdate> = grizy

Remaining set of chip-warp reprocessing, including camera SMFs (reprocessed exposures with ubercal catalog and no magic):

  • label: MD05.GR0.20121005 = grizy
  • data_group: MD05.GR0.20121005 = grizy

Nightly stacks with new chip->warp reprocessing (IPP tag ipp-20120802 and pixels calibrated with ubercal catalog)

  • label: MD05.GR0
  • data_group: MD05.GR0.20121005.nightstack.<obsdate> = grizy

Nightly stack faults:

select stack_id,label,data_group,fault,filter,skycell_id from stackRun join stackSumSkyfile using (stack_id) where data_group like "MD05.GR0.2012%" and fault>0;
| stack_id | label    | data_group                            | fault | filter  | skycell_id  |
+----------+----------+---------------------------------------+-------+---------+-------------+
|  1591991 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20100210 |     2 | i.00000 | skycell.045 | 
|  1584473 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110221 |     5 | z.00000 | skycell.012 | 
|  1592150 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110412 |     5 | i.00000 | skycell.021 | 

|  1583687 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110317 |     5 | y.00000 | skycell.028 | 
|  1584765 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091216 |     5 | z.00000 | skycell.060 | 
|  1585192 | MD05.GR0 | MD05.GR0.20121005.nightstack.20100211 |     5 | z.00000 | skycell.017 | 
|  1586381 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110612 |     5 | z.00000 | skycell.012 | 
|  1587202 | MD05.GR0 | MD05.GR0.20121005.nightstack.20100107 |     5 | r.00000 | skycell.060 | 
|  1589331 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110610 |     5 | r.00000 | skycell.071 | 
|  1589629 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091126 |     5 | g.00000 | skycell.059 | 
|  1591492 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110210 |     5 | g.00000 | skycell.021 | 
|  1592334 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090405 |     5 | i.00000 | skycell.028 | 
|  1593532 | MD05.GR0 | MD05.GR0.20121005.nightstack.20100601 |     5 | i.00000 | skycell.021 | 
|  1593685 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110109 |     5 | i.00000 | skycell.071 | 
|  1593817 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110124 |     5 | i.00000 | skycell.087 | 

Nightly stack quality failures:

select stack_id,label,data_group,quality,filter,skycell_id from stackRun join stackSumSkyfile using (stack_id) where data_group like "MD05.GR0.2012%" and quality>0;
| stack_id | label    | data_group                            | quality | filter  | skycell_id  |
+----------+----------+---------------------------------------+---------+---------+-------------+
|  1592090 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20100222 |     257 | i.00000 | skycell.017 | 
|  1586846 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110201 |   13008 | r.00000 | skycell.021 | 
|  1589350 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110414 |   13008 | g.00000 | skycell.017 | 
|  1592216 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110421 |     257 | i.00000 | skycell.028 | 
|  1592249 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110421 |   13008 | i.00000 | skycell.071 | 
|  1586910 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110423 |     257 | r.00000 | skycell.028 | 
|  1589451 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110423 |   13008 | g.00000 | skycell.071 | 
|  1582918 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110515 |   13008 | y.00000 | skycell.071 | 
|  1582926 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110515 |   13008 | y.00000 | skycell.082 | 
|  1589571 | MD05.GR0 | MD05.GR0.201210xx.nightstack.20110607 |     257 | g.00000 | skycell.071 | 

|  1592343 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090405 |   13008 | i.00000 | skycell.041 | 
|  1592381 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 |   13008 | i.00000 | skycell.017 | 
|  1592393 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 |   13008 | i.00000 | skycell.037 | 
|  1592394 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 |   13008 | i.00000 | skycell.038 | 
|  1592395 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 |   13008 | i.00000 | skycell.041 | 
|  1592399 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 |   13008 | i.00000 | skycell.045 | 
|  1592400 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 |   13008 | i.00000 | skycell.046 | 
|  1592401 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 |   13008 | i.00000 | skycell.047 | 
|  1592402 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 |   13008 | i.00000 | skycell.052 | 
|  1592403 | MD05.GR0 | MD05.GR0.20121005.nightstack.20090407 |   13008 | i.00000 | skycell.053 | 
|  1584593 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091125 |     257 | z.00000 | skycell.003 | 
|  1584596 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091125 |   13008 | z.00000 | skycell.006 | 
|  1584620 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091125 |   13008 | z.00000 | skycell.040 | 
|  1584638 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091125 |   13008 | z.00000 | skycell.060 | 
|  1586957 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091126 |   13008 | r.00000 | skycell.004 | 
|  1589585 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091126 |   13008 | g.00000 | skycell.004 | 
|  1589646 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091126 |   13008 | g.00000 | skycell.082 | 
|  1584666 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091128 |   13008 | z.00000 | skycell.021 | 
|  1584713 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091128 |    3006 | z.00000 | skycell.078 | 
|  1589657 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091208 |   13008 | g.00000 | skycell.021 | 
|  1592467 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091215 |    3006 | i.00000 | skycell.082 | 
|  1587033 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091217 |    3006 | r.00000 | skycell.021 | 
|  1587041 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091217 |   13008 | r.00000 | skycell.030 | 
|  1589759 | MD05.GR0 | MD05.GR0.20121005.nightstack.20091217 |   13008 | g.00000 | skycell.060 | 
|  1584086 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110517 |   13008 | y.00000 | skycell.087 | 
|  1586316 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110522 |   13008 | z.00000 | skycell.082 | 
|  1586375 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110525 |   13008 | z.00000 | skycell.082 | 
|  1589339 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110610 |   13008 | r.00000 | skycell.082 | 
|  1591959 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110610 |   13008 | g.00000 | skycell.082 | 
|  1591964 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110610 |     257 | g.00000 | skycell.087 | 
|  1586427 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110612 |   13008 | z.00000 | skycell.071 | 
|  1586435 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110612 |   13008 | z.00000 | skycell.082 | 
|  1586436 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110612 |   13008 | z.00000 | skycell.083 | 
|  1584093 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110613 |   13008 | y.00000 | skycell.021 | 
|  1584141 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110613 |   13008 | y.00000 | skycell.082 | 
|  1584146 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110613 |   13008 | y.00000 | skycell.087 | 
|  1584201 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110614 |   13008 | y.00000 | skycell.082 | 
|  1584259 | MD05.GR0 | MD05.GR0.20121005.nightstack.20110615 |   13008 | y.00000 | skycell.082 | 

MD06.GR0

Status: Refstacks and full set nightly stacks finished and distributed. Camera SMFs also distributed.

  • 11/18: all filters all exposures running chip->warp
  • 11/23: reftest samples running
  • 11/28: disk/data corruption. on hold.
  • 12/22: repairs mostly finished and restarted refstacks.
  • 12/31: more corrputions found during distribution, fixed.
  • 01/01/2012: Refstacks and sample nightly stacks finished and distributed.
  • 01/11/12: reprocessing of all nightly stacks possible in V3 started
  • 01/17/12: majority of all nightly stacks on datastore. A few faulted with various issues mainly from stacks with 2-3 inputs on far edge skycells.
  • 12/21/12: new refstack MD06.refstack.20121221 will replace this one for diffims in the 2012/2013 season

New observations planned and started in OC148 (January 8, 2012).

All past g-band exposures in the V3 tessellation (other filters nearly identical). MD06.V3-g exposure set

Need to investigate tessellation center and dither for science with NGC4258

Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks.

Filter N total N (FWHM<8 pixels)
y 298 268
z 471 446
i 564 513
r 436 391
g 407 341

MD06.V3 reference stack

  • label: MD06.refstack.20111122
  • data_group: MD06.refstack.20111122

Test stack (reftest) sample also made again for skycell.055 with different number of input warps.

  • label: MD06.reftest.20111121
  • data_group: MD06.reftest.20111121
Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20111122 (top blue) and refstack.20110323 (bottom blue). Any unmatched blue/red overlap for (V0) are due to ones included/rejected in the selected reprocessing. The previous (V0) refstack was only 8 inputs so should be much improvement!

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, (V0)=078) after running psphot on the stacks.

version label (stage_id) filter center skycell input warps center skycell FWHM (major)
MD06.refstack.20111122 V3 y 107 (8) 0.92"
- MD06.reftest.201111121 V3 (539498) y 33 (4) 0.83"
- MD06.reftest.201111121 V3 (539499) y 69 (10) 0.89"
- MD06.reftest.201111121 V3 (539500) y 107 (11) 0.92"
- MD06.reftest.201111121 V3 (539501) y 129 (13) 0.94"
old MD06.refstack.20110323 (V0) y 87 (1) 1.25"
MD06.refstack.20111122 V3 z 110 (2) 0.93"
- MD06.reftest.201111121 V3 (539502) z 30 (6) 0.88"
- MD06.reftest.201111121 V3 (539503) z 70 (9) 0.89"
- MD06.reftest.201111121 V3 (539504) z 102 (6) 0.95"
- MD06.reftest.201111121 V3 (539505) z 142 (5) 0.94"
old MD06.refstack.20110323 (V0) z 16 (0) 1.19"
MD06.refstack.20111122 V3 i 108 (1) 0.98"
- MD06.reftest.201111121 V3 (540829) i 27 (2) 0.86"
- MD06.reftest.201111121 V3 (540830) i 62 (2) 0.92"
- MD06.reftest.201111121 V3 (540831) i 108 (3) 0.97"
- MD06.reftest.201111121 V3 (540832) i 128 (6) 0.99"
old MD06.refstack.20110323 (V0) i 8 (1) 1.17"
MD06.refstack.20111122 V3 r 114 (7) 1.10"
- MD06.reftest.201111121 V3 (541343) r 31 (2) 1.15"
- MD06.reftest.201111121 V3 (541344) r 60 (4) 1.06"
- MD06.reftest.201111121 V3 (541345) r 114 (8) 1.10"
- MD06.reftest.201111121 V3 (541346) r 145 (9) 1.11"
old MD06.refstack.20110323 (V0) r 12 (1) 1.13"
MD06.refstack.20111122 V3 g 113 (15) 1.17"
- MD06.reftest.20111121 V3 (541347) g 25 (0) 1.20"
- MD06.reftest.20111121 V3 (541348) g 64 (8) 1.14"
- MD06.reftest.20111121 V3 (541349) g 113 (12) 1.17"
- MD06.reftest.20111121 V3 (541350) g 144 (25) 1.31"
old MD06.refstack.20110323 (V0) g 12 (3) 1.36"

Troubled skycells?

  • skycell.087 i-band had quality failure, rerun excluding some earlier warps and okay. Needs more investigation into why it had a problem, others with quality issue may be similar.

MD06.GR0 reprocessed sample

  • label: condor_MD06.V3_01
  • data_group: condor_MD06.V3_01_dg for chip->warp
  • dist_group: NULL, changed camera to MD06.GR

Reprocessed nightly stack sample under the following label and data_group.

  • label: MD06.GR0
  • dist_group: MD06.GR
  • data_group of initial sample of nightly stacks processed with ipp-20111110 tag and using the default 4 or more input warps necessary to make stack
    • data_group: MD06.GR0.20111122 = y
    • data_group: MD06.GR0.20111123 = z
    • data_group: MD06.GR0.20111124 = i
    • data_group: MD06.GR0.20111125 = r
    • data_group: MD06.GR0.20111126 = g
  • data_group of remaining sample of nightly stacks processed with ipp-20111222 tag and allowing 2 or more input warps necessary to make stack (typically the far outer skycells, but as a new case, will need to be looked at in more detail)
    • data_group: MD06.GR0.20120109 = y
    • data_group: MD06.GR0.20120110 = z
    • data_group: MD06.GR0.20120112 = i
    • data_group: MD06.GR0.20120114 = r
    • data_group: MD06.GR0.20120116 = g

Distributed on datastore under

  • ps1-md-GR: sample of nightly stack images only (setting up for full set). SSdiff sample still to be run.
  • ps1-md-GR-cat: camera SMFs done (have the data_group condor_MD06.V3_01_dg)

Older nightly stacks are really not that pretty..

Problem/corrupt bundles:

stage_id,filesetID,time registered,target_id,fs_tag,filter,productID,type,data_group,stage
609903,stack.609903.1628823.53,2012-01-17T19:35:29Z,1985,stack.609903,i.00000,ps1-md-GR,IPP-DIST,MD06.GR0.20120112,stack

MD07.GR0

Status: Refstacks and sample nightly stacks finished and distributed.

  • 10/12: all filters all exposures running chip->warp
  • 11/6: all warps finished, y-band refstack started.
  • 11/07: y-band refstack finished, z-band started, reftest samples run
  • 11/08: zirg refstack running, nightly stack samples spanning the period since 4/2009 running.
  • 11/12: full refstack set, nightly stack samples finished and distributed
  • 6/18/12: staticsky run attempted, see notes below
  • 01/09/13 MD07.refstack.20130102 to replace 2011 season refstack

Coming soon. New observations planned for OC149 (February 7, 2012).

All MD07 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD07 z-band nightly stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible. MD07.V3-g shifted from the current pointing center for MD07 to cover more of the earlier exposures. MD07-i nightly stack exposures from 7/12/11 on the MD07.V3 tessellation

Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks (more important for this field given the two pointings since 4/2009).

Filter N total N (FWHM<8 pixels)
y 391 302
z 481 422
i 561 458
r 423 370
g 423 343

MD07.V3 reference stack

  • label: MD07.refstack.20111106
  • data_group: MD07.refstack.20111106

Test stack (reftest) sample also made again for skycell.055 with different number of input warps.

  • label: MD07.reftest.20111107
  • data_group: MD07.reftest.20111107
Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20111106 (top blue) and refstack.20100330 (V0) (bottom blue). Any unmatched blue/red overlap for (V0) are due to ones included/rejected in the selected reprocessing. The previous (V0) refstack was only 8 inputs so should be much improvement!

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, (V0)=078) after running psphot on the stacks.

version label filter center skycell input warps center skycell FWHM (major)
MD07.refstack.20111106 V3 y 116(8) 0.81"
- MD07.reftest.201111107 V3 y 31(1) 0.74"
- MD07.reftest.201111107 V3 y 66(9) 0.77"
- MD07.reftest.201111107 V3 y 143(10) 0.85"
old MD07.refstack.20100330 (V0) y 8(0) 0.91"
MD07.refstack.20111106 V3 z 126(10) 0.96"
- MD07.reftest.201111107 V3 z 29(2) 0.89"
- MD07.reftest.201111107 V3 z 59(4) 0.91"
- MD07.reftest.201111107 V3 z 126(10) 0.96"
- MD07.reftest.201111107 V3 z 154(14) 0.98"
old MD07.refstack.20100330 (V0) z 8(0) 1.27"
MD07.refstack.20111106 V3 i 126(19) 1.07"
- MD07.reftest.201111107 V3 i 29(0) 1.00"
- MD07.reftest.201111107 V3 i 68(9) 1.03"
- MD07.reftest.201111107 V3 i 98(5) 1.05"
- MD07.reftest.201111107 V3 i 126(19) 1.06"
- MD07.reftest.201111107 V3 i 146(16) 1.07"
old MD07.refstack.20100330 (V0) i 8(0) 1.28"
MD07.refstack.20111106 V3 r 138(6) 1.11"
- MD07.reftest.201111107 V3 r 25(2) 1.02"
- MD07.reftest.201111107 V3 r 60(7) 1.08"
- MD07.reftest.201111107 V3 r 115(18) 1.10"
- MD07.reftest.201111107 V3 r 138(7) 1.10"
old MD07.refstack.20110121 (V0) r 8(0) 1.46"
MD07.refstack.20111106 V3 g 152(11) 1.15"
- MD07.reftest.20111107 V3 g 29(2) 1.08"
- MD07.reftest.20111107 V3 g 59(4) 1.11"
- MD07.reftest.20111107 V3 g 126(10) 1.13"
- MD07.reftest.20111107 V3 g 154(14) 1.14"
old MD07.refstack.20100330 (V0) g 8(0) 1.67"
  • seems to be some strange rejection behavior in at least the i-band.

How well does the "best" selected inputs cover the two pointings?

  • note (V0) y-band refstack was at newer pointing, how many y-bands in old pointing?

Troubled skycells?

  • 030-z quality fail
  • first pointing (older data) skycell coverage needs further looking into. generally includes 093,094,095,096, 086,087,088, 078,079, 068,069, 059, 049, 039
  • 077 large rej %, looks like for older data coverage

MD07.GR0 reprocessed sample

Reprocessing of chip-->warp done mostly with condor. Hiccup with loss of tmp files stalled things in the middle of warp, but pantasks picked up the processing fine using the label.

  • label: condor_MD07.V3_01
  • data_group: condor_MD07.V3_01_dg
  • dist_group: NULL, changed camera to MD07.GR

Reprocessed nightly stack sample under the following label and data_group.

  • label: MD07.GR0
  • dist_group: MD07.GR
  • data_group of initial sample of nightly stacks processed with both ipp-20110622 and ipp-20111110 tag (see date) and using the default 4 or more input warps necessary to make stack
    • data_group: MD07.GR0.20111107 = y
    • data_group: MD07.GR0.20111108 = z
    • data_group: MD07.GR0.20111109 = i
    • data_group: MD07.GR0.20111110 = r
    • data_group: MD07.GR0.20111111 = g
  • data_group of remaining sample of nightly stacks processed with ipp-20111222 tag and allowing 2 or more input warps necessary to make stack
    • data_group: MD07.GR0.201201xx = y
    • data_group: MD07.GR0.201201xx = z
    • data_group: MD07.GR0.201201xx = i
    • data_group: MD07.GR0.201201xx = r
    • data_group: MD07.GR0.201201xx = g

Staticsky: only run on refstack skycells with all 5-filters available. This run is on stacks prior to ubercal catalogs so the zeropoint is fundamentally based on the synthcat values. This run provides a sample for testing the stack calibration stage. However, many faults (25/70 runs, i.e. skycells) occurred in the processing and need to still be investigated.

label: MD07.refstack.20111106 data_group: MD07.refstack.20111106.5x20120618 dist_group: MD.staticsky (TBD)

Distributed on datastore under

  • ps1-md-GR: sample of nightly stack images only (setting up for full set). SSdiff sample still to be run.
  • ps1-md-GR-cat: camera SMFs loading (have the data_group condor_MD07.V3_01_dg)

MD08.GR0

Status: In progress

  • 11/25: all filters all exposures running chip->warp
  • 11/28: disk/data corruption extensive. will need to be restarted from scratch once diskspace available.
  • 4/18/12: restarted from scratch after raid crash in November/December with the new ipp-20120404 tag to include new ubercal refcat (PS1.REF.20120325) and detrend improvements (dark modes, noise map but not the boost factors).
  • 4/20: gri reprocessing finished and refstacks started. zy prepped but delayed until 4/22 for other tests.
  • 4/21: gri reprocessed camera SMFs and nightly stacks distributed, reference and test stacks also distributed
  • 4/22: z reprocessing started
  • 4/24: zy reprocessing finished, zy refstack started
  • 4/25: all nightly stack reprocessing finished, all refstacks finished and distributed, staticsky run and distributed.

Coming soon. New observations planned for OC152 (May 5).

All past g-band exposures in the V3 tessellation (other filters nearly identical). MD08.V3-g exposure set

Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks.

Filter N total N (FWHM<8 pixels)
g 475 401
r 499 426
i 516 472
z 532 494
y 565 466

MD08.V3 reference stack

Reference stack using data from 2010+2011, a balance of N input warps versus best seeing, excluding exposures with large background, low zeropoints, and large elongation.

  • label: MD08.refstack.20120422
  • data_group: MD08.refstack.20120422

Test stack (reftest) sample also made again for skycell.055 with different number of input warps.

  • label: MD08.reftest.201204xx (xx=17 for g, 18 for r, 19 for i, 20 for z, 21 for y)
  • data_group:
    • MD08.reftest.201204xx.c#f<FWHM cut> where #=comparison run, <FWHM cut>=cut <fwhm_major from camProcessedExp
    • MD08.reftest.201204xx.all.<FWHM cut> where all means 2009+2010+2011 data possible, <FWHM cut>=cut <fwhm_major from camProcessedExp
    • MD08.reftest.201204xx.2011.<FWHM cut> where 2011 means 2011 data only, <FWHM cut>=cut <fwhm_major from camProcessedExp
Histograms of the FWHM major and minor from camera stage for reprocessed g-band exposures (red) and refstack.20120422 (top blue) and refstack.20100713 (bottom blue). Any unmatched blue/red overlap for (V2) are due to exposures rejected in the selection cuts even at smaller FWHM.

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks.

version data_group (stack_id) filter center skycell input warps (rej) center skycell FWHM (major)
new MD08.refstack.20120422 V3 g 96 (2) 1.27"
MD08.reftest.20120417.c1f49 V3 (873695) g 34 (1) 1.20"
MD08.reftest.20120417.c1f51 V3 (873696) g 66 (2) 1.25"
MD08.reftest.20120417.c1f53 V3 (873697) g 96 (2) 1.25"
MD08.reftest.20120417.c1f57 V3 (873698) g 119 (7) 1.28"
MD08.reftest.20120417.all.50 V3 (861913) g 70 (8) 1.21"
MD08.reftest.20120417.2011.50 V3 (861914) g 50 (5) 1.21"
old MD08.refstack.20100713 V2 g 78 (4) 1.45"
new MD08.refstack.20120422 V3 r 90 (1) 1.19"
MD08.reftest.20120418.c1f42 V3 (873699) r 33 (0) 1.16"
MD08.reftest.20120418.c1f45 V3 (873700) r 72 (3) 1.19"
MD08.reftest.20120418.c1f46 V3 (873701) r 90 (2) 1.19"
MD08.reftest.20120418.c1f50 V3 (873702) r 119 (1) 1.21"
MD08.reftest.20120418.all.45 V3 (861915) r 86 (4) 1.18"
MD08.reftest.20120418.2011.45 V3 (861916) r 73 (6) 1.18"
old MD08.refstack.20110713 V2 r 89 (2) corrupted wt file
new MD08.refstack.20120422 V3 i 91 (3) 1.09"
MD08.reftest.20120419.c1f41 V3 (873703) i 33 (0) 0.92"
MD08.reftest.20120419.c1f43 V3 (873704) i 55 (0) 0.94"
MD08.reftest.20120419.c1f45 V3 (873705) i 91 (3) 1.09"
MD08.reftest.20120419.c1f46 V3 (873706) i 117 (4) 1.10"
MD08.reftest.20120419.all.43 V3 (869922) i 76 (8) 0.95"
MD08.reftest.20120419.2011.43 V3 (869923) i 54 (3) 1.08"
old MD08.refstack.20100713 V2 i 81 (2) 1.34"
new MD08.refstack.20120422 V3 z 94 (3) 0.95"
MD08.reftest.20120420.c1f38 V3 (877041) z 30 (0) 0.90"
MD08.reftest.20120420.c1f40 V3 (877042) z 70 (2) 0.94"
MD08.reftest.20120420.c1f42 V3 (877043) z 94 (3) 0.95"
MD08.reftest.20120420.c1f44 V3 (877044) z 126 (4) 0.97"
MD08.reftest.20120420.all.39 V3 (877037) z 73 (8) 0.93"
MD08.reftest.20120420.2011.39 V3 (877038) z 43 (4) 0.92"
old MD08.refstack.20100713 V2 z 119 (8) 1.39"
new MD08.refstack.20120422 V3 y 93 (7) 0.84"
MD08.reftest.20120421.c1f35 V3 (880936) y 31 (1) 0.79"
MD08.reftest.20120421.c1f38 V3 (880937) y 66 (5) 0.82"
MD08.reftest.20120421.c1f40 V3 (880938) y 93 (7) 0.84"
MD08.reftest.20120421.c1f42 V3 (880939) y 117 (4) 0.85"
MD08.reftest.20120421.all.38 V3 (877039) y 87 (11) 0.83"
MD08.reftest.20120421.2011.38 V3 (877040) y 56 (3) 0.83"
old MD08.refstack.20100713 V2 y 74 (17) 1.68"

* i-band is an odd situation, 2010 had smaller FWHM with many exposures <1" versus 2011 with only a few <1" so 2010+2011 FWHM~1.09" with just ~15 more inputs.

Test psphot run on refstack

Troubled skycells

  • Previously had a few missing skycells in center and around edges (g-band example left). New has only a couple quality failures in i-band for skycell.087,012 (right) that at some point should be manually recovered.

MD08.GR0 reprocessed set

Chips/Camera/Warp: camera SMFs distributed under ps1-md-GR-cat

  • label: MD08.GR0
  • data_group: MD08.GR0.20120417 = g (starting with g this time for testing variance detrend improvements)
  • data_group: MD08.GR0.20120418 = r
  • data_group: MD08.GR0.20120419 = i
  • data_group: MD08.GR0.20120420 = z
  • data_group: MD08.GR0.20120421 = y
  • dist_group: MD08.GR

Nightly stacks under the following label and data_group and distributed under ps1-md-GR

  • label: MD08.GR0
  • data_group: MD08.GR0.20120417.nightstack.<obsdate> = g
  • data_group: MD08.GR0.20120418.nightstack.<obsdate> = r
  • data_group: MD08.GR0.20120419.nightstack.<obsdate> = i
  • data_group: MD08.GR0.201204120.nightstack.<obsdate> = z (NOTE the TYPO in the date 201204120 and not 20120420)
  • data_group: MD08.GR0.20120421.nightstack.<obsdate> = y
  • dist_group: MD08.GR
  • nightly stack_id/skycell found faulted
            862445 	skycell.087 	MD08.GR0 	MD08.GR0.20120417.nightstack.20090620 	MD08.GR 	g.00000  
            863535 	skycell.021 	MD08.GR0 	MD08.GR0.20120417.nightstack.20100630 	MD08.GR 	g.00000
    	867446 	skycell.082 	MD08.GR0 	MD08.GR0.20120418.nightstack.20100603 	MD08.GR 	r.00000
    	867746 	skycell.021 	MD08.GR0 	MD08.GR0.20120418.nightstack.20100712 	MD08.GR 	r.00000
    	869430 	skycell.071 	MD08.GR0 	MD08.GR0.20120418.nightstack.20110731 	MD08.GR 	r.00000
    	870405 	skycell.005 	MD08.GR0 	MD08.GR0.20120419.nightstack.20090627 	MD08.GR 	i.00000
    	870411 	skycell.017 	MD08.GR0 	MD08.GR0.20120419.nightstack.20090627 	MD08.GR 	i.00000
    	870649 	skycell.005 	MD08.GR0 	MD08.GR0.20120419.nightstack.20090919 	MD08.GR 	i.00000
    	870995 	skycell.012 	MD08.GR0 	MD08.GR0.20120419.nightstack.20100601 	MD08.GR 	i.00000
    	871557 	skycell.071 	MD08.GR0 	MD08.GR0.20120419.nightstack.20100821 	MD08.GR 	i.00000
    	871573 	skycell.017 	MD08.GR0 	MD08.GR0.20120419.nightstack.20100830 	MD08.GR 	i.00000
    	871904 	skycell.021 	MD08.GR0 	MD08.GR0.20120419.nightstack.20100917 	MD08.GR 	i.00000
    	872195 	skycell.012 	MD08.GR0 	MD08.GR0.20120419.nightstack.20110512 	MD08.GR 	i.00000
    	872677 	skycell.021 	MD08.GR0 	MD08.GR0.20120419.nightstack.20110629 	MD08.GR 	i.00000
    	872725 	skycell.082 	MD08.GR0 	MD08.GR0.20120419.nightstack.20110629 	MD08.GR 	i.00000
    	873156 	skycell.017 	MD08.GR0 	MD08.GR0.20120419.nightstack.20110801 	MD08.GR 	i.00000
    	873397 	skycell.021 	MD08.GR0 	MD08.GR0.20120419.nightstack.20110822 	MD08.GR 	i.00000
    	877403 	skycell.004 	MD08.GR0 	MD08.GR0.201204120.nightstack.20090622 	MD08.GR 	z.00000
    	877937 	skycell.012 	MD08.GR0 	MD08.GR0.201204120.nightstack.20100409 	MD08.GR 	z.00000
    	878062 	skycell.017 	MD08.GR0 	MD08.GR0.201204120.nightstack.20100418 	MD08.GR 	z.00000
    	878747 	skycell.078 	MD08.GR0 	MD08.GR0.201204120.nightstack.20100702 	MD08.GR 	z.00000
    	879379 	skycell.078 	MD08.GR0 	MD08.GR0.201204120.nightstack.20100912 	MD08.GR 	z.00000
    	880042 	skycell.021 	MD08.GR0 	MD08.GR0.201204120.nightstack.20110706 	MD08.GR 	z.00000
    	881059 	skycell.005 	MD08.GR0 	MD08.GR0.20120421.nightstack.20090706 	MD08.GR 	y.00000
    	882054 	skycell.082 	MD08.GR0 	MD08.GR0.20120421.nightstack.20100728 	MD08.GR 	y.00000 
    	884385 	skycell.082 	MD08.GR0 	MD08.GR0.20120421.nightstack.20110925 	MD08.GR 	y.00000
    	884546 	skycell.082 	MD08.GR0 	MD08.GR0.20120421.nightstack.20110928 	MD08.GR 	y.00000
    

Staticsky: only run on refstack skycells with all 5-filters available and distributed under ps1-md-staticsky. Convolved matching to target PSF much improved, generally the 4-6 pixels now similar to upper input PSF cutoff except for oddities that drive edge skycells >10,20,70 pixels. Work continues to better constrain selection of the target PSF.

  • label: MD08.refstack.20120422
  • data_group: MD08.refstack.20120422.5x20120425
  • dist_group: MD.staticsky
  • Eric Morganson has noted that the RA_PSF,DEC_PSF is NULL http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/ticket/1505, need to track down if fixed and broken again. should be able to use skycell WCS to get coordinates from xy.

SSdiff sample still to be run?

Reprocessing notes:

  • lost raw files:
    -- g-band
    neb://ipp047.0/gpc1/20100606/o5353g0215o/o5353g0215o.ota45.fits
    neb://ipp051.0/gpc1/20100621/o5368g0189o/o5368g0189o.ota64.fits
    neb://ipp013.0/gpc1/20100814/o5422g0083o/o5422g0083o.ota22.fits
    -- r-band
    neb://ipp040.0/gpc1/20100513/o5329g0443o/o5329g0443o.ota15.fits 
    -- i-band
    neb://ipp040.0/gpc1/20100803/o5411g0127o/o5411g0127o.ota15.fits
    -- z-band
    neb://ipp049.0/gpc1/20100816/o5424g0110o/o5424g0110o.ota54.fits
    -- y-band
    neb://ipp009.0/gpc1/20100626/o5373g0272o/o5373g0272o.ota12.fits
    neb://ipp006.0/gpc1/20100627/o5374g0261o/o5374g0261o.ota03.fits
    neb://ipp023.0/gpc1/20100727/o5404g0108o/o5404g0108o.ota06.fits
    
  • lost but regenerated intermediate files:
    -- r-band
    neb://ipp027.0/gpc1/20100712/o5389g0088o/o5389g0088o.ota26.burn.tbl
    -- i-band
    neb://ipp031.0/gpc1/20100619/o5366g0076o/o5366g0076o.ota43.burn.tbl
    neb://ipp025.0/gpc1/20100619/o5366g0076o/o5366g0076o.ota76.burn.tbl
    neb://ipp050.0/gpc1/20100701/o5378g0154o/o5378g0154o.ota61.burn.tbl
    neb://ipp029.0/gpc1/20100917/o5456g0036o/o5456g0036o.ota33.burn.tbl
    -- z-band
    neb://ipp040.0/gpc1/20100521/o5337g0137o/o5337g0137o.ota14.burn.tbl
    neb://ipp045.0/gpc1/20100702/o5379g0086o/o5379g0086o.ota35.burn.tbl
    neb://ipp048.0/gpc1/20100702/o5379g0087o/o5379g0087o.ota50.burn.tbl
    neb://ipp019.0/gpc1/20100702/o5379g0088o/o5379g0088o.ota36.burn.tbl
    neb://ipp008.0/gpc1/20100702/o5379g0090o/o5379g0090o.ota10.burn.tbl
    neb://ipp044.0/gpc1/20100702/o5379g0090o/o5379g0090o.ota31.burn.tbl
    neb://ipp030.0/gpc1/20100702/o5379g0090o/o5379g0090o.ota37.burn.tbl
    -- y-band
    neb://ipp034.0/gpc1/20100626/o5373g0271o/o5373g0271o.ota57.burn.tbl
    
  • features/artifacts
    • seeing "schools" of ghosts in the stacks (ie skycell.063 in g)
    • some valleys (044-g,r)
    • edge "pox" (028-i, 087-y)
    • dither junk? (074-r)
  • odd images to beware of
    -- either tracking jump or mirror solution issue - check if used/rejected in z nightly stack, yes but rejected at end before final stack made by PSF chi2 clip. 
    neb://ipp048.0/gpc1/MD08.GR0.20120420/o5103g0041o.101597/o5103g0041o.101597.ch.442508
    

MD09.GR0

Status:

  • 5/30: prep work started for chip->warp reprocessing.
  • 6/01: chip->warp reprocessing started with new ops tag ipp-20120531 (improved REFCAT 20120524, ghost models, corner under-glow masking). seeing some trouble with new tag in z,i,r for XY67,71.
  • 6/02: reftests run for g,y,z
  • 6/03: refstack for g,y started, nightly stacks run.
  • 6/04: refstack for z started with exposures available, full nightly stack processing on hold for z,r,i until solution with XY68,71 found. Will try an do whatever is possible.
  • 6/06: MD09 r,i refstacks started. z refstack, 90% nightly stacks out to datastore.
  • 6/07: refstacks finished and distributed, staticsky on refstack finished, 90% of camera SMF and nightly stacks that are possible distributed (see listing below for dates of incomplete processing).
  • 7/29 MD09.GR0 XY67,71 chip problem (~48 exposures) cannot be fixed so set to poor quality, remaining camera SMF and nightly stacks processed and distributed.
  • 8/31 replaced with MD09.refstack.20120831 to fix the weigh pixel problem causing arc region of lower sensitivity

Coming soon for new observations planned for OC153 (June 3). Observations attempted earlier but weather has prevented.

MD09.V3-g shifted from the current pointing center for MD09 to cover more of the earlier exposures. MD09.V3-i nightly stack exposures from 9/28/11
All MD09 g-band images over the V3 tessellation with an offset from the current pointing center (shown by the MD09 z-band nightly stack exposures from 9/28/11) in order to include as much of the earlier exposures as possible.

Exposures available and with preliminary FWHM cut of ones <2". All exposures were reprocessed for this field to be available in creating the refstacks and deepstacks.

Filter N total N (FWHM<8 pixels)
g 471 415
r 459 427
i 478 432
z 491 469
y 303 283

MD09.V3 reference stack

Reference stack using data from 2010+2011, a balance of N input warps versus best seeing, excluding exposures with large background, low zeropoints, and large elongation.

  • label: MD09.refstack.20120603
  • data_group: MD09.refstack.20120603

Test stack (reftest) sample also made again for skycell.055 with different number of input warps.

  • label: MD09.reftest.20120602
  • data_group:
    • MD09.reftest.20120602.c#f<FWHM cut> where #=comparison run, <FWHM cut>=cut <fwhm_major from camProcessedExp
    • MD09.reftest.20120602.all.<FWHM cut> where all means 2009+2010+2011 data possible, <FWHM cut>=cut <fwhm_major from camProcessedExp
    • MD09.reftest.20120602.2011.<FWHM cut> where 2011 means 2011 data only, <FWHM cut>=cut <fwhm_major from camProcessedExp
Histograms of the FWHM major and minor from camera stage for reprocessed all exposures (red) and refstack.20120603 (top blue) and refstack.20100721 (bottom blue).

Summary comparison table with the approximate FWHM (major) as the PSF FWHM from a center field skycell (V3=055, V2=078) after running psphot on the stacks. Because the reftests were all run on the same date, the stack_id is important to note to separate by filter as well.

version data_group (stack_id) filter center skycell input warps (rej) center skycell FWHM (major)
new MD09.refstack.20120603 V3 g 97 (0) 1.22"
MD09.reftest.20120602.c1f48 V3 (963600) g 30(0) 1.17"
MD09.reftest.20120602.c1f50 V3 (963601) g 56(1) 1.19"
MD09.reftest.20120602.c1f54 V3 (963602) g 96(1) 1.22"
MD09.reftest.20120602.c1f56 V3 (963603) g 117(2) 1.24"
MD09.reftest.20120602.all.49 V3 (963598) g 95(8) 1.17"
MD09.reftest.20120602.2011.49 V3 (963599) g 81(8) 1.16"
old MD09.refstack.20100721 V2 g 17 (2) 1.52"
new MD09.refstack.20120603 V3 r 89 (2) 1.09"
MD09.reftest.20120602.c1f41 V3 (975093) r 31 (0) 1.02"
MD09.reftest.20120602.c1f43 V3 (975094) r 59 (2) 1.05"
MD09.reftest.20120602.c1f45 V3 (975095) r 89 (2) 1.09"
MD09.reftest.20120602.c1f47 V3 (975096) r 126 (4) 1.13"
MD09.reftest.20120602.all.43 V3 (975092) r 69 (7) 1.05"
MD09.reftest.20120602.2011.43 V3 (975091) r 65 (8) 1.04"
old MD09.refstack.20100721 V2 r 17 (3) 1.79"
new MD09.refstack.20120603 V3 i 85 (2) 0.96"
MD09.reftest.20120602.c1f38 V3 (975158) i 26 (1) 0.88"
MD09.reftest.20120602.c1f41 V3 (975159) i 53 (3) 0.91"
MD09.reftest.20120602.c1f43 V3 (975160) i 85 (5) 0.96"
MD09.reftest.20120602.c1f45 V3 (975161) i 120 (5) 0.99"
MD09.reftest.20120602.all.43 V3 () i 97 (4) 0.97"
MD09.reftest.20120602.2011.43 V3 (975157) i 57 (4) 0.94"
old MD09.refstack.20100721 V2 i 38(1) 1.42"
new MD09.refstack.20120603 V3 z 95 (3) 0.89"
MD09.reftest.20120602.c1f37 V3 (966289) z 36 (1) 0.82"
MD09.reftest.20120602.c1f38 V3 (966290) z 56 (2) 0.85"
MD09.reftest.20120602.c1f40 V3 (966291) z 95 (2) 0.89"
MD09.reftest.20120602.c1f42 V3 (966292) z 119 (2) 0.90"
MD09.reftest.20120602.all.38 V3 (966288) z 87 (4) 0.85"
MD09.reftest.20120602.2011.38 V3 (966287) z 64 (5) 0.83"
old MD09.refstack.20100721 V2 z 29(0) 1.17"
new MD09.refstack.20120603 V3 y 87 (8) 0.85"
MD09.reftest.20120602.c1f37 V3 (963607) y 31(3) 0.79"
MD09.reftest.20120602.c1f41 V3 (963608) y 59(2) 0.82"
MD09.reftest.20120602.c1f45 V3 (963609) y 86(5) 0.86"
MD09.reftest.20120602.c1f52 V3 (963610) y 119(0) 0.89"
MD09.reftest.20120602.all.38 V3 (963606) y 81 (8) 0.78"
MD09.reftest.20120602.2011.38 V3 (963605) y 64 (7) 0.78"
old MD09.refstack.20100721 V2 y 20(3) 1.35"

Test psphot run on refstack

Troubled skycells

  • Previously had a few missing/faulted skycells around edges (i-band example left). New refstacks do not have any quality failures so far and note the current nominal position of the populated skycells is shifted leftward (middle) to accomodate the reprocessing of the nightly stacks with the early pointing (right).

MD09 deep stack test

more details needed

MD09 revised refstack

MD09.refstack.20120831 more details to follow including in table

MD09.GR0 reprocessed set

Chips/Camera/Warp: camera SMFs distributed under ps1-md-GR-cat

  • label: MD09.GR0
  • data_group: MD09.GR0.20120601 = includes all filters
  • dist_group: MD09.GR

  • sample still unprocessed due to XY71,67 faults. tracing/debugging of new ops tag required to fix.
    • seems to be problem with psphot from ipp-20120531 tag, tests work with ipp-20120626 tag. these will be given a quality fault of 42 so that the exposures and nightly stacks can finish.
      | chip_id | exp_id | exp_name    | dateobs             | filter  | comment                        |
      +---------+--------+-------------+---------------------+---------+--------------------------------+
      -- 
      |  467866 | 214900 | o5436g0340o | 2010-08-28 10:47:07 | z.00000 | MD09 z N5436 MD09 dither 6     | 
      |  467880 | 218834 | o5442g0484o | 2010-09-03 12:45:57 | z.00000 | MD09 z N5442 MD09 dither 4     | 
      |  467907 | 223972 | o5451g0362o | 2010-09-12 11:15:50 | z.00000 | MD09 z N5451 MD09 dither 7     | 
      |  468019 | 353527 | o5739g0231o | 2011-06-27 12:45:40 | z.00000 | MD09 z N5739 MD09 dither 7     | 
      |  468043 | 360681 | o5754g0447o | 2011-07-12 14:04:56 | z.00000 | MD09 z N5754 MD09 dither 7     | 
      |  468058 | 367560 | o5766g0335o | 2011-07-24 13:52:27 | z.00000 | MD09 z N5766 MD09 dither 2     | 
      |  468059 | 367564 | o5766g0340o | 2011-07-24 14:13:49 | z.00000 | MD09 z N5766 MD09 dither 7     | 
      |  468062 | 369197 | o5769g0479o | 2011-07-27 13:11:44 | z.00000 | MD09 z N5769 MD09 dither 2     | 
      |  468066 | 369201 | o5769g0483o | 2011-07-27 13:28:45 | z.00000 | MD09 z N5769 MD09 dither 6     | 
      |  468077 | 371776 | o5775g0526o | 2011-08-02 13:38:31 | z.00000 | MD09 z N5775 MD09 dither 1     | 
      |  468079 | 371778 | o5775g0528o | 2011-08-02 13:47:02 | z.00000 | MD09 z N5775 MD09 dither 3     | 
      |  468080 | 371779 | o5775g0529o | 2011-08-02 13:51:17 | z.00000 | MD09 z N5775 MD09 dither 4     | 
      |  468095 | 381404 | o5796g0497o | 2011-08-23 13:28:05 | z.00000 | MD09 z N5796 MD09 dither 3     | 
      |  468112 | 383829 | o5802g0378o | 2011-08-29 12:07:06 | z.00000 | MD09 z N5802 MD09 dither 4     | 
      |  468123 | 385223 | o5805g0137o | 2011-09-01 07:48:19 | z.00000 | MD09 z N5805 MD09 dither 7     | 
      |  468167 | 400366 | o5832g0065o | 2011-09-28 07:47:34 | z.00000 | MD09 z N5832 MD09 dither 6     | 
      |  468175 | 400374 | o5832g0073o | 2011-09-28 08:24:06 | z.00000 | MD09 z N5832 MD09 dither 6     |  
      |  468184 | 401319 | o5835g0049o | 2011-10-01 05:49:30 | z.00000 | MD09 z N5835 MD09 dither 7     | 
      |  468187 | 406280 | o5846g0222o | 2011-10-12 08:40:09 | z.00000 | MD09 z,y N5846 z MD09 dither 2 |
      |  468192 | 406285 | o5846g0227o | 2011-10-12 09:01:27 | z.00000 | MD09 z,y N5846 z MD09 dither 7 |
      -- 
      |  468770 | 200329 | o5408g0319o | 2010-07-31 12:14:02 | i.00000 | MD09 i N5408 MD09 dither 4     | 
      |  468773 | 200342 | o5408g0318o | 2010-07-31 12:09:46 | i.00000 | MD09 i N5408 MD09 dither 3     | 
      |  468811 | 208985 | o5426g0478o | 2010-08-18 13:12:34 | i.00000 | MD09 i N5426 MD09 dither 5     | 
      |  468812 | 208986 | o5426g0479o | 2010-08-18 13:16:47 | i.00000 | MD09 i N5426 MD09 dither 6     | 
      |  468819 | 210454 | o5429g0326o | 2010-08-21 11:15:35 | i.00000 | MD09 i N5429 MD09 dither 5     | 
      |  468836 | 218088 | o5441g0356o | 2010-09-02 11:18:09 | i.00000 | MD09 i N5441 MD09 dither 6     | 
      |  468843 | 219741 | o5444g0239o | 2010-09-05 09:46:59 | i.00000 | MD09 i N5444 MD09 dither 5     | 
      |  468852 | 221522 | o5447g0278o | 2010-09-08 09:43:15 | i.00000 | MD09 i N5447 MD09 dither 6     | 
      |  468889 | 231309 | o5465g0446o | 2010-09-26 11:22:13 | i.00000 | MD09 i N5465 MD09 dither 4     | 
      |  468903 | 236298 | o5477g0051o | 2010-10-08 05:36:12 | i.00000 | MD09 i N5477 MD09 dither 1     | 
      |  468917 | 237920 | o5480g0086o | 2010-10-11 06:00:05 | i.00000 | MD09 i N5480 MD09 dither 7     | 
      |  468943 | 248892 | o5501g0244o | 2010-11-01 08:28:26 | i.00000 | MD09 i N5501 MD09 dither 1     | 
      |  468978 | 353262 | o5738g0243o | 2011-06-26 13:36:11 | i.00000 | MD09 i N5738 MD09 dither 4     | 
      |  468997 | 354596 | o5741g0564o | 2011-06-29 14:01:00 | i.00000 | MD09 i N5741 MD09 dither 5     | 
      |  469023 | 368656 | o5768g0501o | 2011-07-26 13:50:56 | i.00000 | MD09 i N5768 MD09 dither 7     | 
      |  469053 | 379157 | o5792g0496o | 2011-08-19 13:15:52 | i.00000 | MD09 i N5792 MD09 dither 2     | 
      |  469055 | 379159 | o5792g0498o | 2011-08-19 13:24:25 | i.00000 | MD09 i N5792 MD09 dither 4     | 
      |  469063 | 383099 | o5801g0134o | 2011-08-28 08:18:38 | i.00000 | MD09 i N5801 MD09 dither 4     | 
      |  469069 | 384934 | o5804g0438o | 2011-08-31 12:29:08 | i.00000 | MD09 i N5804 MD09 dither 2     | 
      |  469087 | 387780 | o5810g0168o | 2011-09-06 07:35:28 | i.00000 | MD09 i N5810 MD09 dither 7     | 
      |  469111 | 395309 | o5825g0061o | 2011-09-21 06:32:28 | i.00000 | MD09 i N5825 MD09 dither 7     | 
      |  469118 | 399219 | o5829g0169o | 2011-09-25 07:54:00 | i.00000 | MD09 r,i N5829 i MD09 dither 6 | 
      |  469135 | 402155 | o5837g0066o | 2011-10-03 06:51:54 | i.00000 | MD09 i N5837 MD09 dither 7     | 
      |  469160 | 413826 | o5859g0142o | 2011-10-25 07:25:05 | i.00000 | MD09 r,i N5859 i MD09 dither 6 | 
      -- 
      |  468395 | 226092 | o5455g0293o | 2010-09-16 09:35:27 | r.00000 | MD09 g,r N5455 r MD09 dither 7 | 
      |  468510 | 372293 | o5776g0424o | 2011-08-03 13:33:22 | r.00000 | MD09 g,r N5776 r MD09 dither 2 | 
      |  468534 | 378593 | o5791g0497o | 2011-08-18 14:13:15 | r.00000 | MD09 g,r N5791 r MD09 dither 2 | 
      |  468535 | 378594 | o5791g0498o | 2011-08-18 14:15:55 | r.00000 | MD09 g,r N5791 r MD09 dither 3 | 
      |  468580 | 387141 | o5809g0169o | 2011-09-05 07:30:26 | r.00000 | MD09 g,r N5809 r MD09 center   | 
      |  468589 | 389143 | o5812g0319o | 2011-09-08 10:55:03 | r.00000 | MD09 g,r N5812 r MD09 dither 1 | 
      |  468617 | 398815 | o5828g0218o | 2011-09-24 08:58:45 | r.00000 | MD09 g,r N5828 r MD09 dither 5 |
      

Nightly stacks under the following label and data_group and distributed under ps1-md-GR

  • label: MD09.GR0
  • data_group: MD09.GR0.20120602.nightstack.<obsdate> = includes all filters
  • dist_group: MD09.GR
  • nightly stack_id/skycell found faulted (5), may be manually recoverable but not in near future.
            Stack ID Skycell ID 	Label 	        data grp 	                       dist grp 	Filter
            961400 	skycell.030 	MD09.GR0 	MD09.GR0.20120602.nightstack.20100928 	MD09.GR 	g.00000
    	963743 	skycell.031 	MD09.GR0 	MD09.GR0.20120602.nightstack.20090706 	MD09.GR 	y.00000
    	963802 	skycell.045 	MD09.GR0 	MD09.GR0.20120602.nightstack.20090707 	MD09.GR 	y.00000
    	964810 	skycell.082 	MD09.GR0 	MD09.GR0.20120602.nightstack.20110812 	MD09.GR 	y.00000
    	965323 	skycell.054 	MD09.GR0 	MD09.GR0.20120602.nightstack.20111023 	MD09.GR 	y.00000
    	967167 	skycell.068 	MD09.GR0 	MD09.GR0.20120602.nightstack.20100711 	MD09.GR 	z.00000
    	967913 	skycell.038 	MD09.GR0 	MD09.GR0.20120602.nightstack.20101006 	MD09.GR 	z.00000
    	969776 	skycell.061 	MD09.GR0 	MD09.GR0.20120602.nightstack.20090915 	MD09.GR 	r.00000 
    	969954 	skycell.061 	MD09.GR0 	MD09.GR0.20120602.nightstack.20090924 	MD09.GR 	r.00000 
    	973855 	skycell.038 	MD09.GR0 	MD09.GR0.20120602.nightstack.20101017 	MD09.GR 	i.00000 
    

Staticsky: only run on refstack skycells with all 5-filters available and distributed under ps1-md-staticsky. Convolved matching to target PSF much improved, generally the 4-6 pixels now similar to upper input PSF cutoff except for oddities that drive edge skycells >10 pixels (extended source measurements should be on the RAW/unconv image in this tag however). Work continues to better constrain selection of the target PSF.

  • label: MD09.refstack.20120603
  • data_group: MD09.refstack.20120603.5x20120606
  • dist_group: MD.staticsky
  • staticsky faults/problems
    | sky_id | state | workdir                                                                  | label                  | data_group                        | dist_group   | reduction | note | registered          | path_base                                                                                                              | dtime_phot | dtime_script | sources | num_inputs | hostname | good_frac | quality | fault |
    +--------+-------+--------------------------------------------------------------------------+------------------------+-----------------------------------+--------------+-----------+------+---------------------+------------------------------------------------------------------------------------------------------------------------+------------+--------------+---------+------------+----------+-----------+---------+-------+
    | 190101 | new   | neb://@HOST@.0/gpc1/MD09.refstack.20120603/MD09.GR0.staticsky.5x20120606 | MD09.refstack.20120603 | MD09.refstack.20120603.5x20120606 | NULL       | NULL      | NULL | 2012-06-07 09:21:29 | neb://any/gpc1/MD09.refstack.20120603/MD09.GR0.staticsky.5x20120606/MD09.V3/skycell.022/MD09.V3.skycell.022.sky.190101 |       NULL |          172 |       0 |          0 | ippc31   |      NULL |       0 |     3 | 
    | 190133 | new   | neb://@HOST@.0/gpc1/MD09.refstack.20120603/MD09.GR0.staticsky.5x20120606 | MD09.refstack.20120603 | MD09.refstack.20120603.5x20120606 | NULL       | NULL      | NULL | 2012-06-07 09:21:29 | neb://any/gpc1/MD09.refstack.20120603/MD09.GR0.staticsky.5x20120606/MD09.V3/skycell.058/MD09.V3.skycell.058.sky.190133 |       NULL |         3880 |       0 |          0 | ippc38   |      NULL |       0 |     4 | 
    
    
  • Nigel finds RA_PSF,DEC_PSF still NULL.

SSdiff sample still to be run?

Reprocessing notes:

  • lost raw files:
    -- i-band
    neb://ipp029.0/gpc1/20100731/o5408g0319o/o5408g0319o.ota33.fits
    neb://ipp026.0/gpc1/20100806/o5414g0429o/o5414g0429o.ota17.fits
    
  • features/artifacts
    • seeing "schools" of ghosts in the stacks like MD08?
  • odd images to beware of?

MD11.GR0

TBD


Attachments