| Version 22 (modified by , 12 years ago) ( diff ) |
|---|
Summary for MD.PV3 (updates in progress)
- MediumDeepFields summary page -- quick links to all up to date refstacks
- MD.GR0 summary page -- summary of all reprocessing details and 2011/2012 refstacks and some deepstack samples
- MD.PV1 summary page -- the data set that PV2 update reprocessing is based on
- MD.PV2 summary page -- update type reprocessing
This is a summary of the MD.PV3 full pixel reprocessing. Start date and schedule TBD.
This processing will also involve other MD-like fields, including M31, STS and CNP. Each field can be similar and different to the typical MD criteria for the stacks and diffims, summary TBD.
From Ken's talk at PS1SC@STSCI meeting in June 2014, the diffims described there may need to modifications:
- stack:
- MDN -- night stack -- only made for normal MD fields?
- MD0 -- seasonal reference stack
- MDD -- deep stack
- MDT -- iq stack
- diffim:
- warp-stack (WS) -- (warp-ref) only, no (warp-deep) or only for CNP (i.e., is deep and ref the same basically -- difficult w/ possible iq issues)
- stack-stack (SS) -- must add (night-ref) but no (stack-deep) and only for normal MD fields?
| field | stage | N job units | ave_dtime(s) | sig_dtime(s) | status | comment | label/data_group, release name |
| MD | chip/camera/warp | ~2500k/41717/2500k | 130/610/85 | 130/500/65 | TBD | 41717 exposures *60 imfiles/55-60 skycells | MD<f#>.pv3.20141001? |
| night stk | ~291k | 860 | 500 | TBD | typically 8 inputs, sometimes >20 in marginal nights, ~4849*55-60 skycells | ||
| ref stk | 15141 | 6700 | 1500 | TBD | Nin~80, 5 seasons | ||
| deep stk | 3033 | 37.5k* | 13k | TBD | *using 8 theads, really depends on Nin | ||
| iq stk | 3325 | 2215 | 310 | TBD | edge skycells of course run faster in all stacks | ||
| WSref diff | 2500k | ~200 | ~50 | TBD | done by season, same number as warps | ||
| WSdeep diff | x | x | x | x | not going to do, already have WSref by season and using deep stack not a gain | ||
| SS diff | ~291k | ~200 | ~50 | TBD | done by season, same number as night stacks | ||
| night staticsky | ~291k | 110 | 40 | TBD | same number as night stacks (no filter groups) | ||
| ref staticsky | 3025 | 4960 | 1700 | TBD | |||
| deep staticsky | 605 | 6300 | 2550 | TBD | |||
| iq staticsky | 665 | 4660 | 1600 | TBD | |||
| night skycal | ~291k | 30 | 43 | TBD | same as for staticsky since single filter | ||
| ref skycal | 15093 | 26 | 5 | TBD | |||
| deep skycal | 3008 | 31 | 8 | TBD | |||
| iq skycal | 3018 | 40 | 34 | TBD | |||
| full-forced phot | 2500k | ? | ? | TBD | |||
| CNP | chip/camera/warp | ~162k/2695/270k | ~115/410/60 | 150/430/125 | TBD | 2695 exposures *60 imfiles/100? skycells | |
| ref stk | ~500? | ~6.7k? | ? | TBD | Nin~80?, no seasons? one season -- make one as first/last month for ref stack so reduce to Nin~60? | ||
| deep stk | ~500? | ~37k? | ? | TBD | Nin~500? | ||
| iq stk | ~500? | 2200? | ? | TBD | Nin~25? | ||
| WSref diff | ~270k | ~200? | ? | TBD | |||
| ref staticsky | ~100? | ~5k? | ? | TBD | |||
| deep staticsky | ~100? | ~6.3k? | ? | TBD | |||
| iq staticsky | ~100? | 4660 | ? | TBD | |||
| ref skycal | ~500? | ~30? | ? | TBD | |||
| deep skycal | ~500? | ~30? | ? | TBD | |||
| iq skycal | ~500? | ~30? | ? | TBD | |||
| full-forced phot | ~270k | ? | ? | TBD | |||
| M31 | chip/camera/warp | ~261k/4347/261k | 129/624/62 | 190/313/30 | TBD | 4347 exposures *60 imfiles/87-96 skycells | |
| background preserve/restore | ~261k | ||||||
| night stk | ~13.8k? | ~1k? | ? | TBD | treat similar to a normal MD field, 6-20? exposures in r,i | ||
| ref stk | ~475x4? | ~6.7k? | ? | TBD | Nin~60-80, 4 seasons | ||
| deep stk | ~475? | ~37k? | ? | TBD | Nin~500-max? by grizy, or only deep in i and others just ref Nin? | ||
| iq stk | ~475? | 2200? | ? | TBD | Nin~25? | ||
| WSref diff | ~261k | ~200? | ? | TBD | |||
| SSref diff | ~13.8k | ~200? | ? | TBD | |||
| night staticsky | ~13.8k | 110? | ? | TBD | |||
| ref staticsky | ~95*4 | ~5000? | ? | TBD | |||
| deep staticsky | ~95 | ~6300? | ? | TBD | |||
| iq staticsky | ~95 | ~4660? | ? | TBD | |||
| night skycal | ~475 | ~30? | ? | TBD | |||
| ref skycal | ~475*4 | ~30? | ? | TBD | |||
| deep skycal | ~475 | ~30? | ? | TBD | |||
| iq skycal | ~475 | ~30? | ? | TBD | |||
| full-forced phot | ~261k | ? | ? | TBD | |||
| STS | chip/camera/warp | ~996k/16606/996k | 210/1200/100 | 120/575/70 | TBD | 16606 exposures *60 imfiles/55-60 skycells?, 3->7 pointings | |
| ref stk | ~438*4? | ~6.7k? | ? | TBD | Nin~60-80?, 4 seasons for i only | ||
| deep stk | ~438*2? | ~37k? | ? | TBD | Nin~500?, will have some r | ||
| iq stk | ~438*2? | ~2200? | ? | TBD | Nin~25?, will have some r | ||
| WSref diff | ~996k | ~200? | ? | TBD | |||
| ref staticsky | ~1752? | ~110?/5k? | ? | TBD | single filter so no extended source fits? or w/ r? | ||
| deep staticsky | ~876? | ~110?/5k? | ? | TBD | |||
| iq staticsky | ~876? | ~110?/5k? | ? | TBD | |||
| ref skycal | ~1752? | ~30? | ? | TBD | |||
| deep skycal | ~876? | ~30? | ? | TBD | |||
| iq skycal | ~876? | ~30? | ? | TBD | |||
| full-forced phot | ~996k | ? | ? | TBD | |||
- label/data_group, release name -- for gpc1 DB queries/datastore identification and specific postage-stamp server (PSS) association
- ave/sig_dtime -- roughly the dtime_script for the jobs in the gpc1 DB
- N job units -- depend on stage/task will be exposure, imfile or skycell
If run only on the 15 ippsXX processing nodes with high memory, 12 cores -- individual processing estimates only, some can be run ~same time like deep+iq/night/dif/staticsky/skycal possibly
- MD: will be run one field at a time to limit the queued jobs
- chip/cam/warp ~6 jobs/node ~90 jobs, ~42+3+27 ~72 days
- night stack ~4 jobs/node ~60 jobs, ~48.2 d
- ref stack ~4 jobs/node ~60 jobs, ~19.5 d (pv2 took ~2 weeks but also used extra nodes)
- deep stack ~1 jobs/node ~15 jobs, ~88 days (pv2 took ~70d, may take longer for pv3 if don't let as much get clipped for Nin on edge skycells)
- iq stack ~4 jobs/node ~60 jobs, ~1.4 d
- WSref diff ~5 jobs/node 75 jobs, ~77.1 d
- SSref diff ~5 jobs/node 75 jobs, ~9 d
- all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, ~10.4, 0.9 d
- total ~ 333.1 d, ~worst case but >60 dedicated processing nodes will likely be needed to get done w/in 3 months
- CNP:
- chip/cam/warp ~6 jobs/node ~90 jobs, !2.4+<1+4.3 ~8 d
- ref stack ~4 jobs/node ~60 jobs, ~1 d
- deep stack ~1 jobs/node ~15 jobs, ~14.3 d
- iq stack ~4 jobs/node ~60 jobs, <1 d
- WSref diff ~5 jobs/node 75 jobs, ~8 d
- all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, <1 d
- total ~ 32 d
- M31: -- rough estimate based on typical MD times only for now --
- chip/cam/warp ~6 jobs/node ~90 jobs, ~(4.3+4.3)+<1d+2.1 ~11 d -- chip done 2x for background, but process needs to be finalized still
- night stack ~4 jobs/node ~60 jobs, ~2.7 d
- ref stack ~4 jobs/node ~60 jobs, ~2.5 d
- deep stack ~1 jobs/node ~15 jobs, ~13.6 d
- iq stack ~4 jobs/node ~60 jobs, <1 d
- WSref diff ~5 jobs/node 75 jobs, ~8.1 d
- SSref diff ~5 jobs/node 75 jobs, ~2.1 d
- all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, ~1 d
- total ~ 42 d
- STS: -- rough based on typical MD times only for now --
- chip/cam/warp ~6 jobs/node ~90 jobs, ~32.3+3.1+15.4 d ~51 d
- ref stack ~4 jobs/node ~60 jobs, ~2.2 d? -- under/overestimated?
- deep stack ~1 jobs/node ~15 jobs, ~25 d? -- assuming early 3 proj match 3 of the later 7 proj well
- iq stack ~4 jobs/node ~60 jobs, ~0.4 d? -- same as for deep
- WSref diff ~5 jobs/node 75 jobs, ~30.7 d
- all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, ~1-2 d?
- total ~ 110 d? very uncertain
Issues to address before starting:
- disk space limits
- config reorg, or specific PV3 named ones to create
- chip-camera-warp:
- check if refcat oddities in field problem here
- background restore M31 and masking for M31/STS -- test samples to be run and verified for final version
- CNP tessellation check
- stack
- stack quality fault
- night stack config, refstack edge loss -- see diffim as well
- ref+deep stack input cuts -- need feedback on if 2009 and or early 2010 data is okay to use
- iq+ref+deep test stacks for CNP,M31,STS and to verify Nin and input cuts -- if no night stacks for these so iq Nin modification necessary?
- staticsky/skycal
- stack photometry sky oddities
- check refcat oddities in the MD fields
- zeropt recalib in skycal improvement and characterization (y-band at least needs own config)
- test samples for CNP,M31,STS -- equivalent stacks in available filters vs deeper in primary filter, single filter only for STS/M31 so no extended source fits?
- diffim
- verify similar photometry
- verify similar detections/coverage (no sig loss on edges)
- test samples for CNP,M31,STS -- ref vs deep stack for diffim, define seasons (if appropriate for CNP)
- full-forced photometry -- unknown, untested still
_General Summary Status:_
- 10/1 start chip-warp for MD should be possible, STS+M31 sample should be possible
chip -- camera -- warp
Should be started ASAP, so that the warps are available for some early testing of the stacks and diffims.
Testing necessary for the M31, STS and CNP processing still.
stack
Order of operations important -- while deep stacks take 1.5 months or so to finish, night and seasonal reference stacks must be finished before diffims can be started. Lighter jobs like difference images and iq (even reference stacks) can be run along with the deep stacks for some fields.
night
seasonal reference
deep
iq
photometry
difference image
Cannot start until both night and seasonal reference stacks finish.
Note:
See TracWiki
for help on using the wiki.
