Summary for MD.PV3 (updates in progress)

• MediumDeepFields summary page -- quick links to all up to date refstacks
• MD.GR0 summary page -- summary of all reprocessing details and 2011/2012 refstacks and some deepstack samples
• MD.PV1 summary page -- the data set that PV2 update reprocessing is based on
• MD.PV2 summary page -- update type reprocessing

This is a summary of the MD.PV3 full pixel reprocessing. Start date and schedule TBD.

This processing will also involve other MD-like fields, including M31, STS and CNP. Criteria for the stacks and diffims are TBD. From Ken's talk at PS1SC@STSCI meeting in June 2014, the diffims described there may need to modifications:

• stack:
  • MDN -- night stack -- only made for normal MD fields?
  • MD0 -- seasonal reference stack
  • MDD -- deep stack
  • MDT -- iq stack
• diffim:
  • warp-stack (WS) -- (warp-ref) only, no (warp-deep) or only for CNP (i.e., is deep and ref the same basically -- difficult w/ possible iq issues)
  • stack-stack (SS) -- must add (night-ref) but no (stack-deep) and only for normal MD fields?

field	stage	N job units	ave_dtime(s)	sig_dtime(s)	status	comment	label/data_group, release name
MD	chip/camera/warp	~2500k/41717/2500k	130/610/85	130/500/65	TBD	41717 exposures *60 imfiles/55-60 skycells	MD<f#>.pv3.20141001?
	night stk	~291k	860	500	TBD	typically 8 inputs, sometimes >20 in marginal nights, ~4849*55-60 skycells
	ref stk	15141	6700	1500	TBD	Nin~80, 5 seasons
	deep stk	3033	37.5k*	13k	TBD	*using 8 theads, really depends on Nin
	iq stk	3325	2215	310	TBD	edge skycells of course run faster in all stacks
	WSref diff	2500k	~200	~50	TBD	done by season, same number as warps
	WSdeep diff	x	x	x	x	not going to do, already have WSref by season and using deep stack not a gain
	SS diff	~291k	~200	~50	TBD	done by season, same number as night stacks
	night staticsky	~291k	110	40	TBD	same number as night stacks (no filter groups)
	ref staticsky	3025	4960	1700	TBD
	deep staticsky	605	6300	2550	TBD
	iq staticsky	665	4660	1600	TBD
	night skycal	~291k	30	43	TBD	same as for staticsky since single filter
	ref skycal	15093	26	5	TBD
	deep skycal	3008	31	8	TBD
	iq skycal	3018	40	34	TBD
	full-forced phot	2500k	?	?	TBD
CNP	chip/camera/warp	~162k/2695/270k	~115/410/60	150/430/125	TBD	2695 exposures *60 imfiles/100? skycells
	ref stk	~500?	~6.7k?	?	TBD	Nin~80?, no seasons? make one as first/last month for ref stack so reduce to Nin~60?
	deep stk	~500?	~37k?	?	TBD	Nin~500?
	iq stk	~500?	?	?	TBD	Nin~25?
	WSref diff	~2695	~200?	?	TBD
	ref staticsky	~100?	~5k?	?	TBD
	deep staticsky	~100?	~6.3k?	?	TBD
	iq staticsky	~100?	?	?	TBD
	ref skycal	~500?	~30?	?	TBD
	deep skycal	~500?	~30?	?	TBD
	iq skycal	~500?	?	?	TBD
	full-forced phot	~270k	?	?	TBD
M31	chip/camera/warp	~261k/4347/261k	129/624/62	190/313/30	TBD	4347 exposures *60 imfiles/87-96 skycells
	background preserve/restore	~261k
	ref stk	~475x4?	~6.7k?	?	TBD	Nin~60-80, 4 seasons
	deep stk	~475?	~37k?	?	TBD	Nin~500-max? by grizy, or only deep in i and others just ref Nin?
	iq stk	~475?	?	?	TBD
	WSref diff	~261k	~200?	?	TBD
	ref staticsky	~95	5000?	?	TBD
	deep staticsky	~95	6300?	?	TBD
	iq staticsky	~95	?	?	TBD
	ref skycal	~475?	~30?	?	TBD
	deep skycal	~475?	~30?	?	TBD
	iq skycal	~475?	?	?	TBD
	full-forced phot	~261k	?	?	TBD
STS	chip/camera/warp	~996k/16606/996k	210/1200/100	120/575/70	TBD	16606 exposures *60 imfiles/55-60 skycells, 3->7 pointings
	ref stk	~6000?	?	?	TBD	Nin~60-80?, 4 seasons
	deep stk	~2100?	?	?	TBD
	iq stk	~2100?	?	?	TBD
	WSref diff	~996k	X	X	TBD
	ref staticsky	~?	?	?	TBD	single filter so no extended source fits?
	deep staticsky	~?	?	?	TBD
	iq staticsky	~?	?	?	TBD
	ref skycal	~	~37	~29	TBD
	deep skycal	~	~37	~29	TBD
	iq skycal	~	~37	~29	TBD
	full-forced phot	~996k	?	?	TBD

• label/data_group, release name -- for gpc1 DB queries/datastore identification and specific postage-stamp server (PSS) association
• ave/sig_dtime -- roughly the dtime_script for the jobs in the gpc1 DB
• N job units -- depend on stage/task will be exposure, imfile or skycell

If run only on the 15 ippsXX processing nodes with high memory, 12 cores -- individual processing estimates only, some can be run ~same time like deep+iq/night/dif/staticsky/skycal possibly

• MD: will be run one field at a time to limit the queued jobs
  • chip/cam/warp ~6 jobs/node ~90 jobs, ~42+3+27 ~72 days
  • night stack ~4 jobs/node ~60 jobs, ~48.2 d
  • ref stack ~4 jobs/node ~60 jobs, ~19.5 d (pv2 took ~2 weeks but also used extra nodes)
  • deep stack ~1 jobs/node ~15 jobs, ~88 days (pv2 took ~70d, may take longer for pv3 if don't let as much get clipped for Nin on edge skycells)
  • iq stack ~4 jobs/node ~60 jobs, ~1.4 d
  • WSref diff ~5 jobs/node 75 jobs, ~77.1 d
  • SSref diff ~5 jobs/node 75 jobs, ~9 d
  • all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, ~10.4, 0.9 d
  • total ~ 333.1 d, ~worst case but >60 dedicated processing nodes will likely be needed to get done w/in 3 months
• CNP:
  • chip/cam/warp ~6 jobs/node ~90 jobs,
  • ref stack ~4 jobs/node ~60 jobs,
  • deep stack ~1 jobs/node ~15 jobs,
  • iq stack ~4 jobs/node ~60 jobs,
  • WSref diff ~5 jobs/node 75 jobs,
  • SSref diff ~5 jobs/node 75 jobs,
  • all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs,
  • total ~
• M31:
  • chip/cam/warp ~6 jobs/node ~90 jobs,
  • ref stack ~4 jobs/node ~60 jobs,
  • deep stack ~1 jobs/node ~15 jobs,
  • iq stack ~4 jobs/node ~60 jobs,
  • WSref diff ~5 jobs/node 75 jobs,
  • SSref diff ~5 jobs/node 75 jobs,
  • all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs,
  • total ~
• STS:
  • chip/cam/warp ~6 jobs/node ~90 jobs,
  • ref stack ~4 jobs/node ~60 jobs,
  • deep stack ~1 jobs/node ~15 jobs,
  • iq stack ~4 jobs/node ~60 jobs,
  • WSref diff ~5 jobs/node 75 jobs,
  • SSref diff ~5 jobs/node 75 jobs,
  • all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs,
  • total ~

Issues to address before starting:

• disk space limits
• config reorg, or specific PV3 named ones to create
• chip-camera-warp:
  • check if refcat oddities in field problem here
  • background restore M31 and masking for M31/STS -- test samples to be run and verified for final version
  • CNP tessellation check
• stack
  • stack quality fault
  • night stack config, refstack edge loss -- see diffim as well
  • ref+deep stack input cuts -- need feedback on if 2009 and or early 2010 data is okay to use
  • iq+ref+deep test stacks for CNP,M31,STS and to verify Nin and input cuts -- no night stacks for these so iq necessary?
• staticsky/skycal
  • stack photometry sky oddities
  • check refcat oddities in the MD fields
  • zeropt recalib in skycal improvement and characterization (y-band at least needs own config)
  • test samples for CNP,M31,STS -- equivalent stacks in available filters vs deeper in primary filter, single filter only for STS/M31 so no extended source fits?
• diffim
  • verify similar photometry
  • verify similar detections/coverage (no sig loss on edges)
  • test samples for CNP,M31,STS -- ref vs deep stack for diffim, define seasons (appropriate for CNP)
• full-forced photometry -- unknown, untested still

_General Summary Status:_

• 10/1 start chip-warp for MD should be possible, STS+M31 sample should be possible

---

chip -- camera -- warp

Should be started ASAP, so that the warps are available for some early testing of the stacks and diffims.

Testing necessary for the M31, STS and CNP processing still.

---

stack

Order of operations important -- while deep stacks take 1.5 months or so to finish, night and seasonal reference stacks must be finished before diffims can be started. Lighter jobs like difference images and iq (even reference stacks) can be run along with the deep stacks for some fields.

night

seasonal reference

deep

iq

photometry

---

difference image

Cannot start until both night and seasonal reference stacks finish.

---