Summary for MD.PV3 (updates in progress Sept 2017)

  • MediumDeepFields summary page -- quick links to all up to date refstacks
  • MD.GR0 summary page -- summary of all reprocessing details and 2011/2012 refstack and some deepstack samples
  • MD.PV1 summary page -- the data set that PV2 update reprocessing is based on
  • MD.PV2 summary page -- update type reprocessing version (pixels remade, PV1 camera detections used), stack photometry on new stacks
  • MD_stack_types summary page -- general guidelines for the stack types
  • MD.pv2.iqdeepNtest.20140720 data sample page -- two skycells (edge, center annulus) in MD04 in grizy as a test case for stacks spanning ~15 input stack through an all exposure input stack

This is a overview summary of the full pixel reprocessing of the Medium Deep Survey version 3 (MD.PV3, similar code base to 3PI.PV3). Schedule outline and completion date is currently being revised yet again, loading/release of catalog products in the PSPS will follow after the 3PI DR2 public release.

  • once pixel products are verified complete, will attempt to make them accessible via the postage stamp server for consortium access -- chips may be problem, diffs will not have images, but warp+stacks ok (TBD)
  • distribution of large products like warps likely not possible, unclear if night stacks will be possible, but maybe iq+ref+deep stacks will be
  • catalog products (SMF, staticsky CMF, forcedwarp CMF, diff CMF) pre-PSPS still TBD

The MD.PV3 processing will also include the CNP (Celestial North Pole) observations as it is also a targeted field with many exposures and a very deep stack possible. M31 and STS may be done at a later time, but is not part of this reprocessing. The MD and CNP will have its own configuration and summary analysis

_General Summary Status:_

  • 09/20/2017 -- full file scan integrity check summary --
  • 04/16/2017 -- still on hold -- entire cluster hardware moved Maui->Oahu finished, intermediate product loss to be evaluated and code issues to be addressed over next few months -- will not be loaded into PSPS until after DR2 (only MD01-10, CNP to be done)
  • 05/12/2016 -- hardware issues continue to block starting, but hopefully soon
  • 04/01/2016 -- close to restart now that 3PI.PV3 is finished and loading starting, hardware issues preventing access to data currently -- estimate 1-2 wks before restart
  • 10/07/2015 -- WS mostly setup and will trickle along as much as possible with files still missing
  • 10/04/2015 -- MD.PV3 warp files still missing and disk space is not available to finish stacking -- WSdiffs do not save large files for PV3 and can easily be reverted once missing files recovered, so it has been started instead (MD01,MD02 mostly finished), estimate that processing to finish mid-October
  • 08/28/2015 -- on hold again, once stsci data nodes disconnected and shipped, turns out some warp files are not available to continue MD processing
  • 08/22/2015 -- 3PI mostly finished, MD to start up again soon
  • 06/04/2015 -- night stack ~50% finished in bulk, still problem with excess edge loss and fields not complete until fixed -- timeframe estimated end of June with ref stack verification
  • 04/26/2015 -- UH HPC now dominated by 3PI-full forced jobs -- no nodes available
  • 04/01/2015 -- setting up using UH HPC for seasonal refstacks, then night stacks -- refstacks running
  • 02/24/2015 -- limited number of high-mem nodes returned to MD for deep stack processing
  • 01/15/2015 all MD.PV3 processing suspended
  • 01/14/2015 -- MD09 deepstack ~finished after multiple faults and problems due to overloaded system for LAP/3PI processing
  • 01/06/2015 -- MD09 deepstack started
  • 01/02/2015 -- all MD chip-warp ~finished
  • 12/16/2014 -- CNP deepstack started -- label: CNP.PV3.deepstack.20141104
  • 11/23/2014 -- CNP chip-warp finished
  • 11/20/2014 -- MD04 chip-warp finished
  • 11/11/2014 -- CNP chip-warp started -- label: CNP.PV3.20141104
  • 11/07/2014 -- MD chip-warp started with MD04 -- label: MD##.PV3.20141104
  • 11/04/2014 -- MD PV3 tag made for processing -- ipp-mdpv3-20141104
  • 10/01/2014 start chip-warp for MD should be possible, STS+M31 sample should be possible -- new refcat responding different w/ past configs, verification necessary

currently being revised below

PV3 Summary Processing Products

From Ken's talk at PS1SC@STSCI meeting in June 2014, the products to produce include

  • stack:
    • MDN -- night stack -- only made for normal MD fields
    • MD0 -- seasonal reference stack -- only where seasons exist (CNP yearly so no seasonal refstack)
    • MDD -- deep stack --
    • MDT -- IQ stack --
  • diffim: (the diffims described there may need modifications)
    • warp-stack (WS) -- warp-ref only. No warp-deep (deep stack PSF can be a problem for diffims)
    • stack-stack (SS) -- night/stack-deep in talk but should be night-ref and typically for MD. M31 with multiple exposures per night may be possible. STS has upwards 100/night, so probably not.
field stage/data product N job units ave_dtime(s) sig_dtime(s) status comment label; data_group; release name
MD chip/camera/warp ~2500k/41717/2500k 130/610/85 130/500/65 finished 41717 exposures *60 imfiles/55-60 skycells MD<f#>.PV3.20141104; MD<f#>.<obsdate>.PV3.<process date>; (MD<f#>.PV3?)
night stk ~291k 860 500 TBD typically 8 inputs, sometimes >20 in marginal nights, ~4849*55-60 skycells
ref stk 15141 6700 1500 95% finished, verifying Nin~80, 5 seasons
deep stk 3033 37.5k* 13k 95% finished, verifiying *using 8 theads, really depends on Nin. starting MD09 since doesn't require Jan/Jun 2010 start decision
iq stk 3325 2215 310 TBD edge skycells of course run faster in all stacks
WSref diff 2500k ~200 ~50 TBD done by season, same number as warps
WSdeep diff x x x x not going to do, already have WSref by season and using deep stack not a gain
SS diff ~291k ~200 ~200 TBD done by season, same number as night stacks
night staticsky ~291k 110 40 TBD same number as night stacks (no filter groups)
ref staticsky 3025 4960 1700 TBD
deep staticsky 605 6300 2550 TBD
iq staticsky 665 4660 1600 TBD
night skycal ~291k 30 43 TBD same as for staticsky since single filter
ref skycal 15093 26 5 TBD
deep skycal 3008 31 8 TBD
iq skycal 3018 40 34 TBD
full-forced phot 2500k ? ? TBD
CNP chip/camera/warp ~162k/2695/270k ~115/410/60 150/430/125 finished 2695 exposures *60 imfiles/100? skycells CNP.PV3.20141104; CNP.<obsdate>.PV3.<process date>; (CNP.PV3?)
ref stk ~500? ~6.7k? ? TBD no seasons, so like 3PI WSdiff -- WS so Nin~30?
deep stk ~500? ~37k? ? finished, verifying Nin~500? Need to verify a functional deep stack since first version
iq stk ~500? 2200? ? TBD Nin~25?
WSref diff ~270k ~200? ? TBD
ref staticsky ~100? ~5k? ? TBD
deep staticsky ~100? ~6.3k? ? TBD
iq staticsky ~100? 4660 ? TBD
ref skycal ~500? ~30? ? TBD
deep skycal ~500? ~30? ? TBD
iq skycal ~500? ~30? ? TBD
full-forced phot ~270k ? ? TBD
M31 chip/camera/warp ~261k/4347/261k 129/624/62 190/313/30 TBD 4347 exposures *60 imfiles/87-96 skycells
background preserve/restore ~261k
night stk ~13.8k? ~1k? ? TBD treat similar to a normal MD field, 6-20? exposures in r,i
ref stk ~190x4? ~6.7k? ? TBD Nin~60-80, 4 seasons (only r,i), gzy not enough for deep so @ref?
deep stk ~475? ~37k? ? TBD Nin~500-max? by grizy, or only deep in i and others just ref Nin?
iq stk ~475? 2200? ? TBD Nin~25?
WSref diff ~261k ~200? ? TBD
SSref diff ~13.8k ~200? ? TBD
night staticsky ~13.8k 110? ? TBD
ref staticsky ~95*4 ~5000? ? TBD
deep staticsky ~95 ~6300? ? TBD
iq staticsky ~95 ~4660? ? TBD
night skycal ~475 ~30? ? TBD
ref skycal ~475*4 ~30? ? TBD
deep skycal ~475 ~30? ? TBD
iq skycal ~475 ~30? ? TBD
full-forced phot ~261k ? ? TBD
STS chip/camera/warp ~996k/16606/996k 210/1200/100 120/575/70 TBD 16606 exposures *60 imfiles/55-60 skycells?, 3->7 pointings
ref stk ~438*4? ~6.7k? ? TBD Nin~60-80?, 4 seasons for i only
deep stk ~438*2? ~37k? ? TBD Nin~500?, will have some r
iq stk ~438*2? ~2200? ? TBD Nin~25?, will have some r
WSref diff ~996k ~200? ? TBD
ref staticsky ~1752? ~110?/5k? ? TBD single filter so no extended source fits? or w/ r?
deep staticsky ~876? ~110?/5k? ? TBD
iq staticsky ~876? ~110?/5k? ? TBD
ref skycal ~1752? ~30? ? TBD
deep skycal ~876? ~30? ? TBD
iq skycal ~876? ~30? ? TBD
full-forced phot ~996k ? ? TBD
  • label/data_group, release name -- for gpc1 DB queries/datastore identification and specific postage-stamp server (PSS) association
  • ave/sig_dtime -- roughly the dtime_script for the jobs in the gpc1 DB
  • N job units -- depend on stage/task will be exposure, imfile or skycell

PV3 runtime estimates based on PV2

If run only on the 15 ippsXX processing nodes with high memory, 12 cores -- individual processing estimates only, some can be run ~same time like deep+iq/night/dif/staticsky/skycal possibly

  • MD: will be run one field at a time to limit the queued jobs
    • chip/cam/warp ~6 jobs/node ~90 jobs, ~42+3+27 ~72 days
    • night stack ~4 jobs/node ~60 jobs, ~48.2 d
    • ref stack ~4 jobs/node ~60 jobs, ~19.5 d (pv2 took ~2 weeks but also used extra nodes)
    • deep stack ~1 jobs/node ~15 jobs, ~88 days (pv2 took ~70d, may take longer for pv3 if don't let as much get clipped for Nin on edge skycells)
    • iq stack ~4 jobs/node ~60 jobs, ~1.4 d
    • WSref diff ~5 jobs/node 75 jobs, ~77.1 d
    • SSref diff ~5 jobs/node 75 jobs, ~9 d
    • all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, ~10.4, 0.9 d
    • total ~ 333.1 d, ~worst case but >60 dedicated processing nodes will likely be needed to get done w/in 3 months
  • CNP:
    • chip/cam/warp ~6 jobs/node ~90 jobs, !2.4+<1+4.3 ~8 d
    • ref stack ~4 jobs/node ~60 jobs, ~1 d
    • deep stack ~1 jobs/node ~15 jobs, ~14.3 d
    • iq stack ~4 jobs/node ~60 jobs, <1 d
    • WSref diff ~5 jobs/node 75 jobs, ~8 d
    • all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, <1 d
    • total ~ 32 d
  • M31: -- rough estimate based on typical MD times only for now --
    • chip/cam/warp ~6 jobs/node ~90 jobs, ~(4.3+4.3)+<1d+2.1 ~11 d -- chip done 2x for background, but process needs to be finalized still
    • night stack ~4 jobs/node ~60 jobs, ~2.7 d
    • ref stack ~4 jobs/node ~60 jobs, ~2.5 d -- overestimated, need to adjust for just ri?
    • deep stack ~1 jobs/node ~15 jobs, ~13.6 d -- not quite same number for gzy? then make ref and lower this estimate?
    • iq stack ~4 jobs/node ~60 jobs, <1 d
    • WSref diff ~5 jobs/node 75 jobs, ~8.1 d
    • SSref diff ~5 jobs/node 75 jobs, ~2.1 d
    • all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, ~1 d
    • total ~ 42 d
  • STS: -- rough based on typical MD times only for now --
    • chip/cam/warp ~6 jobs/node ~90 jobs, ~32.3+3.1+15.4 d ~51 d
    • ref stack ~4 jobs/node ~60 jobs, ~2.2 d? -- under/overestimated?
    • deep stack ~1 jobs/node ~15 jobs, ~25 d? -- assuming early 3 proj match 3 of the later 7 proj well
    • iq stack ~4 jobs/node ~60 jobs, ~0.4 d? -- same as for deep
    • WSref diff ~5 jobs/node 75 jobs, ~30.7 d
    • all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, ~1-2 d?
    • total ~ 110 d? very uncertain

Revised estimate with machines possible -- all machines being used for 3PI now so estimates now also limited to when nodes returned

  • MD:
    • deep stack -- now that chip-cam-warp finished, this it the limiting process group as only certain machines can be used (ipp067-095,ipps00-14,ippx037-044).
      • the special data nodes (ipp067-095) being used for nightly processing, this could still allow them to be used for deep stacks for ~10 out of 24 hrs
      • this would be an equivalent of ~34 nodes (10hr on ipp067-095 plus full use of ipps and ippxMD) for deep stacks to finish the remaining 9 MD fields in ~30-35 days
    • using the same equivalent ~34 nodes for processing the other MD data products while the bulk of the other nodes continue on 3PI, the following estimates become in somewhat of an order
      • reference stacks -- ~10 days after deep stacks finish
      • night stacks -- an extra 26 days after ref stacks finish
      • WS and SS diffims -- an extra 35-40 + 10 days after night stacks
      • iq stacks -- minimal, ~1-2 days with 15 nodes and will try to work into during other processing (some time may be possibly shaved off night or diffims if they can be run along side deep as well)
      • staticsky/skycal -- an extra 5 days after stacks and diffs
      • full-forced -- mostly unknown still
  • CNP: if the deep stack verifies okay, then the WSdiff are the next large process group (full-forced phot is an unknown) but can be run on mostly any type of node as well as the reference and iq stacks and have been folded into the times for MD above.
  • if by the time deep stacks are finished, 3PI may also be mostly finished and the remaining general MD products could be pushed through more quickly w/ any of the ippx+c nodes -- maybe 20 days after that in a best case (there is still an order dependence as reference and night stacks need to be finished before WS and SS respectively can be started)

PV3 issues to address

Issues to address, most before starting:

  • disk space limits (turned out to also include network limits) -- mostly solved
  • config reorganization or specific PV3 named ones to create -- all configs must have PV3 to avoid confusion -- IQ stack, WS+SS diff, staticsky+skycal, FF setup remaining
  • chip-camera-warp:
    • check if refcat oddities in field is a problem -- Eddie and Doug catalogs to compare to (TBD)
    • new refcat astrometry and zeropoint verification tests -- the new refcat demonstrated large astrometry and zeropoint variations when previous config used -- final verification needed with full fields finished
    • CNP tessellation check and exposure filling factor -- done, need plots for notes wiki
    • MD as a targeted survey and utilized throughout the different processing needs to have a summary of exposures changed/lost in PV3 -- PV3 vs PV2 and original processing comparison in quality faults and astrometry/zeropoint variations is critical so events published in transients work is reproduced -- TBD before final night+ref stack and diffs done
    • background restore M31 and masking for M31/STS -- test samples to be run and verified for final version, on hold for now 1/2015
  • stack
    • stack kernel files -- for deep stacks can be >400 and any one can cause a fault in the middle of processing if system is unstable.. for deep stacks may be helpful to disable those since not doing anything with conv products and shouldn't clean up deep stacks.. -- requires devel/code work and lower prio
    • stack quality fault -- sometime edge skycells fault w/ internal photometry/PSF error on the convolved stack and do not want to lose for staticsky or diffims --
    • night stack config, refstack edge loss -- see diffim for as well -- comparison to PV2 and earlier datasets -- TBD w/ final night+ref stacks
    • reference, deep stack input cuts -- need feedback on if 2009 and or early 2010 data is okay to use -- 2009 is not great data so not to be used, Jan/June 2010 seems okay (level of impact unknown) and has been used in deep stack
    • iq, reference, deep test stacks for CNP (M31,STS) and to verify Nin, input cuts, pixel rejection and minimum pixel limits --
  • staticsky/skycal
    • stack photometry sky oddities --
    • check refcat oddities in the MD fields for skycal re-zeropointing -- TBD
    • zeropt recalibration in skycal improvement and characterization (y-band at least may need own config) -- TBD
    • evaluate test samples for CNP (M31,STS) -- equivalent stacks in available filters vs deeper in primary filter, single filter only for STS/M31 so no extended source fits? -- TBD
  • diffim
    • verify similar photometry to previous reductions -- TBD
    • verify similar detections/coverage (no sig loss on edges) -- a key point for reference stack creation above -- TBD
    • evaluate test samples for CNP (M31,STS) -- TBD
  • full-forced photometry -- unknown, untested still for any MD type field but should be similar just more sources from deepstacks -- TBD

PV3 Processing Stages


Should be started ASAP, so that the warps are available for some early testing of the stacks and diffim processing.

Testing necessary for the M31, STS processing still -- priority for CNP as last reprocessing was ~2011.

All processing with tag ipp-mdpv3-20141104

(need to add summary issues and changes in processing/configs here)

Due to network and system disorder, the MD+CNP chip-camera-warp processing was primarily done and has now finished. Verification and summary information on the notes pages


Order of operations important -- while deep stacks take 1.5 months or so to finish, night and seasonal reference stacks must be finished before diffims can be started. Lighter jobs like difference images and iq stacks (even possibly reference stacks) can be run along with the deep stacks for some fields.

(should have own subsections for adding summary details on config/processing changes)

  • deep -- tag ipp-mdpv3-20141104
    • CNP --
    • MD09 --
  • seasonal reference -- tag ipp-mdpv3-20150326 on UH HPC cray system, MD03.notyr5 IPP systems
  • night -- tag ipp-mdpv3-20150326 on default IPP/Maui systems
  • iq -- tag ?
  • stack photometry (staticsky+skycal stage) -- tag ?

difference image

(should have own subsections for adding summary details on config/processing changes)

  • WSdiff -- seasonal reference stacks finished and mostly verified -- ippmd:ipp-20150312
  • SSdiff -- cannot start until night and seasonal reference stacks finish and verified --

forced warp photometry


Local IPP -- Processing Notes

Section for local processing details/notes -- not useful for general users

pantasks_server nodes for ippmd user -- will keep current for own sanity and may change often due to task shuffling

  • ipps00 -- ~ippmd/deepstack -- high memory stacks
  • ipps01 -- ~ippmd/stdscience -- chip-cam-warp (finished), diffim, staticsky+skycal
  • ipps02 -- ~ippmd/distribution -- reference and night stacks for transients group, possible skycal before loading into PSPS
  • ipps03 -- ~ippmd/stack -- lower memory, faster running iq, night and probably reference stacks
  • ipps04 -- ~ippmd/refstack -- for fixing missing md03 notyr5 stacks and CNP refstack
  • ippc29 -- ~ippmd/uhhpc_stack -- remote processing on uh cray, also used for test runs + debugging

host groups -- nodes that specifically have to be used for deep stack processing

  • m0 = ipps00--s04 (12 core, 148G, ~650GB tmp disk, very efficient machines)
  • m1 = ipps05--s14 ((12 core, 98G, ~650GB tmp disk, very efficient machines)
  • x0b,x1b = ippx037-x040, x041-x044 (12 core, 96G, 2TB tmp disk)
  • s4 = ipp067--071 (12 core, 132G) -- data nodes and used for nightly science processing so only 10hr/day available -- also for ipptopsps so avoid
  • s5 = ipp072--097 (12 core, 198G) -- data nodes and used for nightly science processing so only 10hr/day available -- also for ipptopsps so avoid
  • s6 = ipp100--104 (12 core, 198G) -- data nodes and used for nightly science processing so only 10hr/day available -- also for ipptopsps so avoid

data node targeting -- helps to have data going to same machine (need to update)

  • ~ippmd/src/ipp-mdpv3-20141104/ippTasks/ipphosts.mhpcc.config -- defaults to nightly processing allocation, after building ippmd tag, need to replace
    • ~ippmd/psconfig/ipp-mdpv3-20141104.lin64/share/pantasks/modules/ipphosts.mhpcc.config with
    • ~ippmd/src/ipp-mdpv3-20141104/ippTasks/ipphosts.mhpcc.config.ippmd or local copy changed on-the-fly
    • ~ippmd/psconfig/ipp-mdpv3-20141104.lin64/share/pantasks/modules/ipphosts.mhpcc.config.ippmd
  • stsci main targets for LAP/3PI -- MD have less file volume to move when finished
  • ipp067--081 are nearly full and need to have lighter allocations
  • ipp082-095 various levels of larger free space and should have higher allocations
  • ipp100-104 large amount of disk space, ipptopsps using heavily

processing loads/schedule groups --

  • CNP chip-cam-warp --
  • CNP deepstack --
  • CNP iq, refstack --
  • CNP WSdiff --
  • CNP staticsky,skycal --
  • CNP full-forced phot --
  • MD chip-cam-warp -- finished
  • MD deepstack --
  • MD refstack (multiple sets for out-of-season) --
  • MD WSdiff --
  • MD iq, night stack --
  • MD SSdiff --
  • MD staticsky, skycal --
  • MD full-forced phot

File group sizes -- it has been discussed that the conv will possibly be cleaned (normally won't go to stsci either) --

  • warps ~ /skycell ~
  • night stacks (compressed)~137M/skycell*291k ~39.9T
    • conv ~58M/skycell ~16.9T
  • iq stacks ~1450M/skycell*3325 ~4.8T
    • conv ~645M/skycell ~2.1T
  • ref stacks ~1472M/skycell*15141 ~22.3T (uhhpc ~650/skycell, did not return conv or kernels etc)
    • conv ~650/skycell ~9.8T
  • deep stacks ~1672-1805M/skycell*3033 ~5-5.5T -- large variation from kernels
    • conv ~665M/skycell ~2T