Summary for MD.PV3 (updates in progress)

• MediumDeepFields summary page -- quick links to all up to date refstacks
• MD.GR0 summary page -- summary of all reprocessing details and 2011/2012 refstack and some deepstack samples
• MD.PV1 summary page -- the data set that PV2 update reprocessing is based on
• MD.PV2 summary page -- update type reprocessing version (pixels remade, PV1 camera detections used), stack photometry on new stacks
• MD_stack_types summary page -- general guidelines for the stack types
• MD.pv2.iqdeepNtest.20140720 data sample page -- two skycells (edge, center annulus) in MD04 in grizy as a test case for stacks spanning ~15 input stack through an all exposure input stack

This is a overview summary of the MD.PV3 full pixel reprocessing. Start date and schedule a mess with the unstable processing 12/01/2014.

This processing will also involve other MD-like fields, including CNP (M31 and STS if time allows). Each field can be similar and different to the typical MD criteria for the stacks and diffims, summaries

• MD.PV3-MDnotes
• MD.PV3-CNPnotes

_General Summary Status:_

• 10/1 start chip-warp for MD should be possible, STS+M31 sample should be possible -- new refcat responding different w/ past configs, verification necessary
• 11/4 -- MD PV3 tag made for processing -- ipp-mdpv3-20141104
• 11/07 -- MD chip-warp started with MD04 -- label: MD##.PV3.20141104
• 11/11 -- CNP chip-warp started -- label: CNP.PV3.20141104
• 11/20 -- MD04 chip-warp finished
• 11/23 -- CNP chip-warp finished
• 12/16 -- CNP deepstack started -- label: CNP.PV3.deepstack.20141104
• 1/2/2015 -- all MD chip-warp ~finished
• 1/6 -- MD09 deepstack started
• 1/14 -- MD09 deepstack ~finished after multiple faults and problems due to overloaded system for LAP/3PI processing
• 1/15 all MD.PV3 processing suspended
• 2/24 -- limited number of high-mem nodes returned to MD for deep stack processing
• 4/01 -- setting up using UH HPC for seasonal refstacks, then night stacks -- refstacks running
• 4/26 -- UH HPC now dominated by 3PI-full forced jobs -- no nodes available
• night stacks processing on local machines -- problem with excess rejection and edge loses due to poor warp PSF models, will need development/code work to fix
• 6/04 -- night stack ~50% finished in bulk, still problem with excess edge loss and fields not complete until fixed -- timeframe estimated end of June with ref stack verification
• 08/22 -- 3PI mostly finished, MD to start up again soon
• 8/28 -- on hold again, once stsci data nodes disconnected and shipped, turns out some warp files are not available to continue MD processing

---

PV3 Summary Processing Products

From Ken's talk at PS1SC@STSCI meeting in June 2014, the products to produce include

• stack:
  • MDN -- night stack -- only made for normal MD fields
  • MD0 -- seasonal reference stack -- only where seasons exist (CNP yearly so no seasonal refstack)
  • MDD -- deep stack --
  • MDT -- IQ stack --
• diffim: (the diffims described there may need modifications)
  • warp-stack (WS) -- warp-ref only. No warp-deep (deep stack PSF can be a problem for diffims)
  • stack-stack (SS) -- night/stack-deep in talk but should be night-ref and typically for MD. M31 with multiple exposures per night may be possible. STS has upwards 100/night, so probably not.

field	stage/data product	N job units	ave_dtime(s)	sig_dtime(s)	status	comment	label; data_group; release name
MD	chip/camera/warp	~2500k/41717/2500k	130/610/85	130/500/65	finished	41717 exposures *60 imfiles/55-60 skycells	MD<f#>.PV3.20141104; MD<f#>.<obsdate>.PV3.<process date>; (MD<f#>.PV3?)
	night stk	~291k	860	500	TBD	typically 8 inputs, sometimes >20 in marginal nights, ~4849*55-60 skycells
	ref stk	15141	6700	1500	95% finished, verifying	Nin~80, 5 seasons
	deep stk	3033	37.5k*	13k	95% finished, verifiying	*using 8 theads, really depends on Nin. starting MD09 since doesn't require Jan/Jun 2010 start decision
	iq stk	3325	2215	310	TBD	edge skycells of course run faster in all stacks
	WSref diff	2500k	~200	~50	TBD	done by season, same number as warps
	WSdeep diff	x	x	x	x	not going to do, already have WSref by season and using deep stack not a gain
	SS diff	~291k	~200	~200	TBD	done by season, same number as night stacks
	night staticsky	~291k	110	40	TBD	same number as night stacks (no filter groups)
	ref staticsky	3025	4960	1700	TBD
	deep staticsky	605	6300	2550	TBD
	iq staticsky	665	4660	1600	TBD
	night skycal	~291k	30	43	TBD	same as for staticsky since single filter
	ref skycal	15093	26	5	TBD
	deep skycal	3008	31	8	TBD
	iq skycal	3018	40	34	TBD
	full-forced phot	2500k	?	?	TBD
CNP	chip/camera/warp	~162k/2695/270k	~115/410/60	150/430/125	finished	2695 exposures *60 imfiles/100? skycells	CNP.PV3.20141104; CNP.<obsdate>.PV3.<process date>; (CNP.PV3?)
	ref stk	~500?	~6.7k?	?	TBD	no seasons, so like 3PI WSdiff -- WS so Nin~30?
	deep stk	~500?	~37k?	?	finished, verifying	Nin~500? Need to verify a functional deep stack since first version
	iq stk	~500?	2200?	?	TBD	Nin~25?
	WSref diff	~270k	~200?	?	TBD
	ref staticsky	~100?	~5k?	?	TBD
	deep staticsky	~100?	~6.3k?	?	TBD
	iq staticsky	~100?	4660	?	TBD
	ref skycal	~500?	~30?	?	TBD
	deep skycal	~500?	~30?	?	TBD
	iq skycal	~500?	~30?	?	TBD
	full-forced phot	~270k	?	?	TBD
M31	chip/camera/warp	~261k/4347/261k	129/624/62	190/313/30	TBD	4347 exposures *60 imfiles/87-96 skycells
	background preserve/restore	~261k
	night stk	~13.8k?	~1k?	?	TBD	treat similar to a normal MD field, 6-20? exposures in r,i
	ref stk	~190x4?	~6.7k?	?	TBD	Nin~60-80, 4 seasons (only r,i), gzy not enough for deep so @ref?
	deep stk	~475?	~37k?	?	TBD	Nin~500-max? by grizy, or only deep in i and others just ref Nin?
	iq stk	~475?	2200?	?	TBD	Nin~25?
	WSref diff	~261k	~200?	?	TBD
	SSref diff	~13.8k	~200?	?	TBD
	night staticsky	~13.8k	110?	?	TBD
	ref staticsky	~95*4	~5000?	?	TBD
	deep staticsky	~95	~6300?	?	TBD
	iq staticsky	~95	~4660?	?	TBD
	night skycal	~475	~30?	?	TBD
	ref skycal	~475*4	~30?	?	TBD
	deep skycal	~475	~30?	?	TBD
	iq skycal	~475	~30?	?	TBD
	full-forced phot	~261k	?	?	TBD
STS	chip/camera/warp	~996k/16606/996k	210/1200/100	120/575/70	TBD	16606 exposures *60 imfiles/55-60 skycells?, 3->7 pointings
	ref stk	~438*4?	~6.7k?	?	TBD	Nin~60-80?, 4 seasons for i only
	deep stk	~438*2?	~37k?	?	TBD	Nin~500?, will have some r
	iq stk	~438*2?	~2200?	?	TBD	Nin~25?, will have some r
	WSref diff	~996k	~200?	?	TBD
	ref staticsky	~1752?	~110?/5k?	?	TBD	single filter so no extended source fits? or w/ r?
	deep staticsky	~876?	~110?/5k?	?	TBD
	iq staticsky	~876?	~110?/5k?	?	TBD
	ref skycal	~1752?	~30?	?	TBD
	deep skycal	~876?	~30?	?	TBD
	iq skycal	~876?	~30?	?	TBD
	full-forced phot	~996k	?	?	TBD

• label/data_group, release name -- for gpc1 DB queries/datastore identification and specific postage-stamp server (PSS) association
• ave/sig_dtime -- roughly the dtime_script for the jobs in the gpc1 DB
• N job units -- depend on stage/task will be exposure, imfile or skycell

PV3 runtime estimates based on PV2

If run only on the 15 ippsXX processing nodes with high memory, 12 cores -- individual processing estimates only, some can be run ~same time like deep+iq/night/dif/staticsky/skycal possibly

• MD: will be run one field at a time to limit the queued jobs
  • chip/cam/warp ~6 jobs/node ~90 jobs, ~42+3+27 ~72 days
  • night stack ~4 jobs/node ~60 jobs, ~48.2 d
  • ref stack ~4 jobs/node ~60 jobs, ~19.5 d (pv2 took ~2 weeks but also used extra nodes)
  • deep stack ~1 jobs/node ~15 jobs, ~88 days (pv2 took ~70d, may take longer for pv3 if don't let as much get clipped for Nin on edge skycells)
  • iq stack ~4 jobs/node ~60 jobs, ~1.4 d
  • WSref diff ~5 jobs/node 75 jobs, ~77.1 d
  • SSref diff ~5 jobs/node 75 jobs, ~9 d
  • all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, ~10.4, 0.9 d
  • total ~ 333.1 d, ~worst case but >60 dedicated processing nodes will likely be needed to get done w/in 3 months
• CNP:
  • chip/cam/warp ~6 jobs/node ~90 jobs, !2.4+<1+4.3 ~8 d
  • ref stack ~4 jobs/node ~60 jobs, ~1 d
  • deep stack ~1 jobs/node ~15 jobs, ~14.3 d
  • iq stack ~4 jobs/node ~60 jobs, <1 d
  • WSref diff ~5 jobs/node 75 jobs, ~8 d
  • all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, <1 d
  • total ~ 32 d
• M31: -- rough estimate based on typical MD times only for now --
  • chip/cam/warp ~6 jobs/node ~90 jobs, ~(4.3+4.3)+<1d+2.1 ~11 d -- chip done 2x for background, but process needs to be finalized still
  • night stack ~4 jobs/node ~60 jobs, ~2.7 d
  • ref stack ~4 jobs/node ~60 jobs, ~2.5 d -- overestimated, need to adjust for just ri?
  • deep stack ~1 jobs/node ~15 jobs, ~13.6 d -- not quite same number for gzy? then make ref and lower this estimate?
  • iq stack ~4 jobs/node ~60 jobs, <1 d
  • WSref diff ~5 jobs/node 75 jobs, ~8.1 d
  • SSref diff ~5 jobs/node 75 jobs, ~2.1 d
  • all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, ~1 d
  • total ~ 42 d
• STS: -- rough based on typical MD times only for now --
  • chip/cam/warp ~6 jobs/node ~90 jobs, ~32.3+3.1+15.4 d ~51 d
  • ref stack ~4 jobs/node ~60 jobs, ~2.2 d? -- under/overestimated?
  • deep stack ~1 jobs/node ~15 jobs, ~25 d? -- assuming early 3 proj match 3 of the later 7 proj well
  • iq stack ~4 jobs/node ~60 jobs, ~0.4 d? -- same as for deep
  • WSref diff ~5 jobs/node 75 jobs, ~30.7 d
  • all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, ~1-2 d?
  • total ~ 110 d? very uncertain

Revised estimate with machines possible -- all machines being used for 3PI now so estimates now also limited to when nodes returned

• MD:
  • deep stack -- now that chip-cam-warp finished, this it the limiting process group as only certain machines can be used (ipp067-095,ipps00-14,ippx037-044).
    • the special data nodes (ipp067-095) being used for nightly processing, this could still allow them to be used for deep stacks for ~10 out of 24 hrs
    • this would be an equivalent of ~34 nodes (10hr on ipp067-095 plus full use of ipps and ippxMD) for deep stacks to finish the remaining 9 MD fields in ~30-35 days
  • using the same equivalent ~34 nodes for processing the other MD data products while the bulk of the other nodes continue on 3PI, the following estimates become in somewhat of an order
    • reference stacks -- ~10 days after deep stacks finish
    • night stacks -- an extra 26 days after ref stacks finish
    • WS and SS diffims -- an extra 35-40 + 10 days after night stacks
    • iq stacks -- minimal, ~1-2 days with 15 nodes and will try to work into during other processing (some time may be possibly shaved off night or diffims if they can be run along side deep as well)
    • staticsky/skycal -- an extra 5 days after stacks and diffs
    • full-forced -- mostly unknown still

• CNP: if the deep stack verifies okay, then the WSdiff are the next large process group (full-forced phot is an unknown) but can be run on mostly any type of node as well as the reference and iq stacks and have been folded into the times for MD above.

• if by the time deep stacks are finished, 3PI may also be mostly finished and the remaining general MD products could be pushed through more quickly w/ any of the ippx+c nodes -- maybe 20 days after that in a best case (there is still an order dependence as reference and night stacks need to be finished before WS and SS respectively can be started)

---

PV3 issues to address

Issues to address, most before starting:

• disk space limits (turned out to also include network limits) -- mostly solved
• config reorganization or specific PV3 named ones to create -- all configs must have PV3 to avoid confusion -- IQ stack, WS+SS diff, staticsky+skycal, FF setup remaining
• chip-camera-warp:
  • check if refcat oddities in field is a problem -- Eddie and Doug catalogs to compare to (TBD)
  • new refcat astrometry and zeropoint verification tests -- the new refcat demonstrated large astrometry and zeropoint variations when previous config used -- final verification needed with full fields finished
  • CNP tessellation check and exposure filling factor -- done, need plots for notes wiki
  • MD as a targeted survey and utilized throughout the different processing needs to have a summary of exposures changed/lost in PV3 -- PV3 vs PV2 and original processing comparison in quality faults and astrometry/zeropoint variations is critical so events published in transients work is reproduced -- TBD before final night+ref stack and diffs done
  • background restore M31 and masking for M31/STS -- test samples to be run and verified for final version, on hold for now 1/2015
• stack
  • stack kernel files -- for deep stacks can be >400 and any one can cause a fault in the middle of processing if system is unstable.. for deep stacks may be helpful to disable those since not doing anything with conv products and shouldn't clean up deep stacks.. -- requires devel/code work and lower prio
  • stack quality fault -- sometime edge skycells fault w/ internal photometry/PSF error on the convolved stack and do not want to lose for staticsky or diffims --
  • night stack config, refstack edge loss -- see diffim for as well -- comparison to PV2 and earlier datasets -- TBD w/ final night+ref stacks
  • reference, deep stack input cuts -- need feedback on if 2009 and or early 2010 data is okay to use -- 2009 is not great data so not to be used, Jan/June 2010 seems okay (level of impact unknown) and has been used in deep stack
  • iq, reference, deep test stacks for CNP (M31,STS) and to verify Nin, input cuts, pixel rejection and minimum pixel limits --
• staticsky/skycal
  • stack photometry sky oddities --
  • check refcat oddities in the MD fields for skycal re-zeropointing -- TBD
  • zeropt recalibration in skycal improvement and characterization (y-band at least may need own config) -- TBD
  • evaluate test samples for CNP (M31,STS) -- equivalent stacks in available filters vs deeper in primary filter, single filter only for STS/M31 so no extended source fits? -- TBD
• diffim
  • verify similar photometry to previous reductions -- TBD
  • verify similar detections/coverage (no sig loss on edges) -- a key point for reference stack creation above -- TBD
  • evaluate test samples for CNP (M31,STS) -- TBD
• full-forced photometry -- unknown, untested still for any MD type field -- TBD

---

PV3 Processing Stages

chip--camera--warp

Should be started ASAP, so that the warps are available for some early testing of the stacks and diffim processing.

Testing necessary for the M31, STS processing still -- priority for CNP as last reprocessing was ~2011.

All processing with tag ipp-mdpv3-20141104

(need to add summary issues and changes in processing/configs here)

Due to network and system disorder, the MD+CNP chip-camera-warp processing was primarily done and has now finished. Verification and summary information on the notes pages

• MD.PV3-MDnotes
• MD.PV3-CNPnotes

---

stack

Order of operations important -- while deep stacks take 1.5 months or so to finish, night and seasonal reference stacks must be finished before diffims can be started. Lighter jobs like difference images and iq stacks (even possibly reference stacks) can be run along with the deep stacks for some fields.

(should have own subsections for adding summary details on config/processing changes)

• deep -- tag ipp-mdpv3-20141104
  • CNP --
  • MD09 --

• seasonal reference -- tag ipp-mdpv3-20150326 on UH HPC cray system

• night -- tag ipp-mdpv3-20150326 on default IPP/Maui systems

• iq -- tag ?

• photometry -- tag ?

---

difference image

(should have own subsections for adding summary details on config/processing changes)

• WSdiff -- cannot start until seasonal reference stacks finish and verified

• SSdiff -- cannot start until night and seasonal reference stacks finish and verified

---

Processing Notes

Section for local processing details/notes -- not for general users

pantasks_server nodes for ippmd user -- will keep current for own sanity and may change often due to task shuffling

• ipps00 -- ~ippmd/deepstack -- high memory stacks
• ipps01 -- ~ippmd/stdscience -- ---chip-cam-warp--- (finished), diffim, staticsky+skycal
• ipps02 -- ~ippmd/distribution -- reference and night stacks for transients group, possible skycal before loading into PSPS
• ipps03 -- ~ippmd/stack -- lower memory, faster running iq, night and probably reference stacks
• ippc29 -- ~ippmd/uhhpc_stack -- remote processing on uh cray, also used for test runs + debugging

host groups -- nodes that specifically have to be used for deep stack processing

• m0 = ipps00--s04 (12 core, 144G, very efficient machines)
• m1 = ipps05--s14 ((12 core, 96G, very efficient machines)
• x0b,x1b = ippx037-x040, x041-x044 (12 core, 96G)
• s4 = ipp067--071 (12 core, 132G) -- data nodes and used for nightly science processing so only 10hr/day available
• s5 = ipp072--095 (12 core, 198G) -- data nodes and used for nightly science processing so only 10hr/day available

data node targeting -- helps to have data going to same machine

• ~ippmd/src/ipp-mdpv3-20141104/ippTasks/ipphosts.mhpcc.config -- defaults to nightly processing allocation, after building ippmd tag, need to replace
  • ~ippmd/psconfig/ipp-mdpv3-20141104.lin64/share/pantasks/modules/ipphosts.mhpcc.config with
  • ~ippmd/src/ipp-mdpv3-20141104/ippTasks/ipphosts.mhpcc.config.ippmd or local copy changed on-the-fly
  • ~ippmd/psconfig/ipp-mdpv3-20141104.lin64/share/pantasks/modules/ipphosts.mhpcc.config.ippmd
• stsci main targets for LAP/3PI -- MD have less file volume to move when finished
• ipp067--081 are nearly full and need to have lighter allocations
• ipp082-095 various levels of larger free space and should have higher allocations

processing loads/schedule groups --

• CNP chip-cam-warp --
• CNP deepstack --
• CNP iq, refstack --
• CNP WSdiff --
• CNP staticsky,skycal --
• CNP full-forced phot --

• MD chip-cam-warp -- finished
• MD deepstack --
• MD refstack (multiple sets for out-of-season) --
• MD WSdiff --
• MD iq, night stack --
• MD SSdiff --
• MD staticsky, skycal --
• MD full-forced phot

File group sizes -- it has been discussed that the conv will possibly be cleaned (normally won't go to stsci either) --

• warps ~ /skycell ~
• night stacks (compressed)~137M/skycell*291k ~39.9T
  • conv ~58M/skycell ~16.9T
• iq stacks ~1450M/skycell*3325 ~4.8T
  • conv ~645M/skycell ~2.1T
• ref stacks ~1472M/skycell*15141 ~22.3T (uhhpc ~650/skycell, did not return conv or kernels etc)
  • conv ~650/skycell ~9.8T
• deep stacks ~1672-1805M/skycell*3033 ~5-5.5T -- large variation from kernels
  • conv ~665M/skycell ~2T

---