= Summary for MD.PV3 (updates in progress) = 
[[PageOutline]]

 * [wiki:MediumDeepFields] summary page -- quick links to all up to date refstacks
 * [wiki:MD.GR0] summary page -- summary of all reprocessing details and 2011/2012 refstack and some deepstack samples 
 * [wiki:MD.PV1] summary page -- the data set that PV2 update reprocessing is based on
 * [wiki:MD.PV2] summary page -- update type reprocessing

This is a summary of the MD.PV3 full pixel reprocessing. Start date and schedule a mess with the unstable processing 12/01/2014. 

This processing will also involve other MD-like fields, including CNP (M31 and STS if time allows). Each field can be similar and different to the typical MD criteria for the stacks and diffims, summary '''TBD'''.

___General Summary Status:___
 * '''10/1''' start chip-warp for MD should be possible, STS+M31 sample should be possible -- new refcat responding different w/ past configs, verification necessary
 * 11/4 -- MD PV3 tag made for processing -- ipp-mdpv3-20141104 
 * 11/07 -- MD chip-warp started with MD04 -- label: MD##.PV3.20141104
 * 11/11 -- CNP chip-warp started -- label: CNP.PV3.20141104
 * 11/20 -- MD04 chip-warp finished
 * 11/23 -- CNP chip-warp finished
 * 12/16 -- CNP deepstack started -- label: CNP.PV3.deepstack.20141104
 * 1/2/2015 -- all MD chip-warp ~finished
 * 1/6 -- MD09 deepstack started
 * 1/14 -- MD09 deepstack ~finished after multiple faults and problems due to overloaded system for LAP/3PI processing
 * '''1/15 all MD.PV3 processing suspended'''


----

== PV3 Summary Processing Products ==

From Ken's talk at PS1SC@STSCI meeting in June 2014, the products to produce include 
 * stack:
  * MDN -- night stack -- only made for normal MD fields 
  * MD0 -- seasonal reference stack -- only where seasons exist (CNP yearly so no seasonal refstack)
  * MDD -- deep stack -- 
  * MDT -- IQ stack -- 
 * diffim: (the diffims described there may need modifications)
  * warp-stack (WS) -- warp-ref only. No warp-deep (deep stack PSF can be a problem for diffims)
  * stack-stack (SS) -- night/stack-deep in talk but should be night-ref and typically for MD. M31 with multiple exposures per night may be possible. STS has upwards 100/night, so probably not. 


|| '''field''' || '''stage/data product''' || '''N job units''' || '''ave_dtime(s)''' || '''sig_dtime(s)''' || '''status''' || '''comment''' || '''label; data_group; release name''' ||
|| || || || || || || ||
|| MD || chip/camera/warp || ~2500k/41717/2500k || 130/610/85 || 130/500/65 || '''finished'' || 41717 exposures *60 imfiles/55-60 skycells || MD<f#>.PV3.20141104; MD<f#>.<obsdate>.PV3.<process date>; (MD<f#>.PV3?) ||
|| || || || || || || ||
|| || night stk || ~291k || 860 || 500 || TBD || typically 8 inputs, sometimes >20 in marginal nights, ~4849*55-60 skycells || ||
|| || ref stk || 15141 || 6700 || 1500 || TBD || Nin~80, 5 seasons ||
|| || deep stk || 3033 || 37.5k* || 13k || '''MD09 finished, verifiying''' || *using 8 theads, really depends on Nin. starting MD09 since doesn't require Jan/Jun 2010 start decision ||
|| || iq stk || 3325 || 2215 || 310 || TBD || edge skycells of course run faster in all stacks||
|| || || || || || || ||
|| || WSref diff || 2500k || ~200 || ~50 || TBD || done by season, same number as warps ||
|| || WSdeep diff || x || x || x || x || not going to do, already have WSref by season and using deep stack not a gain ||
|| || SS diff || ~291k || ~200 || ~200 || TBD || done by season, same number as night stacks ||
|| || || || || || || ||
|| || night staticsky || ~291k || 110 || 40 || TBD || same number as night stacks (no filter groups) ||
|| || ref staticsky || 3025 || 4960 || 1700 || TBD || ||
|| || deep staticsky || 605 || 6300 || 2550 || TBD || ||
|| || iq staticsky || 665 || 4660 || 1600 || TBD || || 
|| || || || || || || ||
|| || night skycal || ~291k || 30 || 43 || TBD || same as for staticsky since single filter ||
|| || ref skycal || 15093 || 26 || 5 || TBD || ||
|| || deep skycal || 3008 || 31 || 8 || TBD || ||
|| || iq skycal || 3018 || 40 || 34 || TBD || ||
|| || || || || || || ||
|| || full-forced phot || 2500k || ? || ? || TBD || ||
|| || || || || || || ||
|| || || || || || || ||
|| CNP || chip/camera/warp || ~162k/2695/270k || ~115/410/60 || 150/430/125 || '''finished''' || 2695 exposures *60 imfiles/100? skycells || CNP.PV3.20141104; CNP.<obsdate>.PV3.<process date>; (CNP.PV3?) ||
|| || || || || || || ||
|| || ref stk || ~500? || ~6.7k? || ? || TBD || no seasons, so like 3PI WSdiff -- WS so Nin~30? ||
|| || deep stk || ~500? || ~37k? || ? || '''finished, verifying''' || Nin~500? Need to verify a functional deep stack since first version||
|| || iq stk || ~500? || 2200? || ? || TBD || Nin~25? ||
|| || || || || || || ||
|| || WSref diff || ~270k || ~200? || ? || TBD || ||
|| || || || || || || ||
|| || ref staticsky || ~100? || ~5k? || ? || TBD || ||
|| || deep staticsky || ~100? || ~6.3k? || ? || TBD || ||
|| || iq staticsky || ~100? || 4660 || ? || TBD || ||
|| || || || || || || ||
|| || ref skycal || ~500? || ~30? || ? || TBD || || 
|| || deep skycal || ~500? || ~30? || ? || TBD || || 
|| || iq skycal || ~500? || ~30? || ? || TBD || || 
|| || || || || || || ||
|| || full-forced phot || ~270k || ? || ? || TBD || ||
|| || || || || || || ||
|| || || || || || || ||
|| M31 || chip/camera/warp || ~261k/4347/261k || 129/624/62 || 190/313/30 || TBD || 4347 exposures *60 imfiles/87-96 skycells  || || 
|| || background preserve/restore || ~261k ||
|| || || || || || || ||
|| || night stk || ~13.8k? || ~1k? || ? || TBD || treat similar to a normal MD field, 6-20? exposures in r,i ||
|| || ref stk || ~190x4? || ~6.7k? || ? || TBD || Nin~60-80, 4 seasons (only r,i), gzy not enough for deep so @ref? ||
|| || deep stk || ~475? || ~37k? || ? || TBD || Nin~500-max? by grizy, or only deep in i and others just ref Nin? ||
|| || iq stk || ~475? || 2200? || ? || TBD || Nin~25? ||
|| || || || || || || ||
|| || WSref diff || ~261k || ~200? || ? || TBD || ||
|| || SSref diff || ~13.8k || ~200? || ? || TBD || ||
|| || || || || || || ||
|| || night staticsky || ~13.8k || 110? || ? || TBD || ||
|| || ref staticsky || ~95*4 || ~5000? || ? || TBD || ||
|| || deep staticsky || ~95 || ~6300? || ? || TBD || ||
|| || iq staticsky || ~95 || ~4660? || ? || TBD || ||
|| || || || || || || ||
|| || night skycal || ~475 || ~30? || ? || TBD || || 
|| || ref skycal || ~475*4 || ~30? || ? || TBD || || 
|| || deep skycal || ~475 || ~30? || ? || TBD || || 
|| || iq skycal || ~475 || ~30? || ? || TBD || || 
|| || || || || || || ||
|| || full-forced phot || ~261k || ? || ? || TBD || ||
|| || || || || || || || 
|| || || || || || || ||
|| STS || chip/camera/warp || ~996k/16606/996k || 210/1200/100 || 120/575/70 || TBD || 16606 exposures *60 imfiles/55-60 skycells?, 3->7 pointings || ||
|| || || || || || || ||
|| || ref stk || ~438*4? || ~6.7k? || ? || TBD || Nin~60-80?, 4 seasons for i only ||
|| || deep stk || ~438*2? || ~37k? || ? || TBD || Nin~500?, will have some r ||
|| || iq stk || ~438*2? || ~2200? || ? || TBD || Nin~25?, will have some r ||
|| || || || || || || ||
|| || WSref diff || ~996k || ~200? || ? || TBD || ||
|| || || || || || || ||
|| || ref staticsky || ~1752? || ~110?/5k? || ? || TBD || single filter so no extended source fits? or w/ r? || 
|| || deep staticsky || ~876? || ~110?/5k? || ? || TBD || || 
|| || iq staticsky || ~876? || ~110?/5k? || ? || TBD || || 
|| || || || || || || ||
|| || ref skycal || ~1752? || ~30? || ? || TBD || ||
|| || deep skycal || ~876? || ~30? || ? || TBD || ||
|| || iq skycal || ~876? || ~30? || ? || TBD || ||
|| || || || || || || ||
|| || full-forced phot || ~996k || ? || ? || TBD || ||
|| || || || || || || ||
|| || || || || || || ||

 * label/data_group, release name -- for gpc1 DB queries/datastore identification and specific postage-stamp server (PSS) association
 * ave/sig_dtime -- roughly the dtime_script for the jobs in the gpc1 DB
 * N job units -- depend on stage/task will be exposure, imfile or skycell

=== PV3 runtime estimates based on PV2 ===

If run only on the 15 ippsXX processing nodes with high memory, 12 cores -- individual processing estimates only, some can be run ~same time like deep+iq/night/dif/staticsky/skycal possibly
 * MD: will be run one field at a time to limit the queued jobs
  * chip/cam/warp ~6 jobs/node ~90 jobs, ~42+3+27 ~72 days
  * night stack ~4 jobs/node ~60 jobs, ~48.2 d
  * ref stack ~4 jobs/node ~60 jobs, ~19.5 d (pv2 took ~2 weeks but also used extra nodes)
  * deep stack ~1 jobs/node ~15 jobs, ~88 days (pv2 took ~70d, may take longer for pv3 if don't let as much get clipped for Nin on edge skycells)
  * iq stack ~4 jobs/node ~60 jobs, ~1.4 d
  * WSref diff ~5 jobs/node 75 jobs, ~77.1 d
  * SSref diff ~5 jobs/node 75 jobs, ~9 d
  * all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, ~10.4, 0.9 d
  * total ~ 333.1 d, ~worst case but >60 dedicated processing nodes will likely be needed to get done w/in 3 months
 * CNP:
  * chip/cam/warp ~6 jobs/node ~90 jobs, !2.4+<1+4.3 ~8 d
  * ref stack ~4 jobs/node ~60 jobs, ~1 d 
  * deep stack ~1 jobs/node ~15 jobs, ~14.3 d
  * iq stack ~4 jobs/node ~60 jobs, <1 d
  * WSref diff ~5 jobs/node 75 jobs, ~8 d
  * all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, <1 d
  * total ~ 32 d
 * M31: -- rough estimate based on typical MD times only for now -- 
  * chip/cam/warp ~6 jobs/node ~90 jobs, ~(4.3+4.3)+<1d+2.1 ~11 d -- chip done 2x for background, but process needs to be finalized still 
  * night stack ~4 jobs/node ~60 jobs, ~2.7 d
  * ref stack ~4 jobs/node ~60 jobs, ~2.5 d -- overestimated, need to adjust for just ri?
  * deep stack ~1 jobs/node ~15 jobs, ~13.6 d -- not quite same number for gzy? then make ref and lower this estimate?
  * iq stack ~4 jobs/node ~60 jobs, <1 d 
  * WSref diff ~5 jobs/node 75 jobs, ~8.1 d
  * SSref diff ~5 jobs/node 75 jobs, ~2.1 d
  * all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, ~1 d
  * total ~ 42 d
 * STS: -- rough based on typical MD times only for now --
  * chip/cam/warp ~6 jobs/node ~90 jobs, ~32.3+3.1+15.4 d ~51 d
  * ref stack ~4 jobs/node ~60 jobs, ~2.2 d? -- under/overestimated?
  * deep stack ~1 jobs/node ~15 jobs, ~25 d? -- assuming early 3 proj match 3 of the later 7 proj well
  * iq stack ~4 jobs/node ~60 jobs,  ~0.4 d? -- same as for deep
  * WSref diff ~5 jobs/node 75 jobs, ~30.7 d
  * all staticsky + skycal ~4 and 8 jobs/node 60 and 120 jobs, ~1-2 d?
  * total ~ 110 d? very uncertain 


Revised estimate with machines possible -- 
 * MD: 
  * deep stack -- now that chip-cam-warp finished, this it the limiting process group as only certain machines can be used ()

 * CNP: if the deep stack verifies okay, then the WSdiff are the next large process group (full-forced phot is an unknown)
  *  

----

== PV3 issues to address ==

Issues to address, most before starting:
 * disk space limits (turned out to also include network limits) -- 
 * config reorganization or specific PV3 named ones to create -- all configs must have PV3 to avoid confusion
 * chip-camera-warp:
  * check if refcat oddities in field is a problem -- Eddie and Doug catalogs to compare still
  * new refcat astrometry and zeropoint verification tests -- 
  * CNP tessellation check -- 
  * background restore M31 and masking for M31/STS -- test samples to be run and verified for final version
 * stack  
  * stack quality fault -- sometime edge skycells fault w/ internal photometry/PSF error on the convolved stack and do not want to lose for staticsky or diffims -- 
  * night stack config, refstack edge loss -- see diffim as well -- 
  * ref+deep stack input cuts -- need feedback on if 2009 and or early 2010 data is okay to use -- 2009 is not great data so not to be used, Jan/June 2010 is still under consideration if the baffle is critical or not.
  * iq+ref+deep test stacks for CNP,M31,STS and to verify Nin, input cuts, pixel rejection and minimum pixel limits -- 
 * staticsky/skycal
  * stack photometry sky oddities -- 
  * check refcat oddities in the MD fields --
  * zeropt recalib in skycal improvement and characterization (y-band at least needs own config) -- 
  * test samples for CNP,M31,STS -- equivalent stacks in available filters vs deeper in primary filter, single filter only for STS/M31 so no extended source fits?
 * diffim
  * verify similar photometry to previous reductions -- 
  * verify similar detections/coverage (no sig loss on edges) -- really a key point for reference stack creation above as well
  * test samples for CNP,M31,STS -- 
 * full-forced photometry -- unknown, untested still for any MD type field

----


== PV3 Processing Stages ==

=== chip--camera--warp ===
Should be started ASAP, so that the warps are available for some early testing of the stacks and diffims.

Testing necessary for the M31, STS and CNP processing still -- priority for CNP as last reprocessing was ~2011

----

=== stack ===
Order of operations important -- while deep stacks take 1.5 months or so to finish, night and seasonal reference stacks must be finished before diffims can be started. Lighter jobs like difference images and iq stacks (even reference stacks) can be run along with the deep stacks for some fields.

==== night ====

==== seasonal reference ====

==== deep ====

==== iq ====

==== photometry ====

----


=== difference image ===

Cannot start until night and seasonal reference stacks finish for SSdiff.

----


== Processing Notes ==

Section for local processing details/notes -- not for general users

pantasks_server nodes for ippmd user -- will keep current for own sanity and may change often due to task shuffling
 * ipps00 -- ~ippmd/deepstack -- high memory stacks
 * ipps01 -- ~ippmd/stdscience -- ---chip-cam-warp--- (finished), diffim, staticsky+skycal
 * ipps02 -- ~ippmd/distribution -- reference and night stacks for transients group, possible skycal before loading into PSPS
 * ipps03 -- ~ippmd/stack -- lower memory, faster running iq, night and probably reference stacks 

host groups --
 * m0 = ipps00--s04 (12 core, 144G, very efficient machines)
 * m1 = ipps05--s14 ((12 core, 96G, very efficient machines)
 * x0b,x1b = ippx037-x040, x041-x044 (12 core, 96G) 
 * s4 = ipp067--071 (12 core, 132G) -- data nodes and used for nightly science processing so only 10hr/day available
 * s5 = ipp072--095 (12 core, 198G) -- data nodes and used for nightly science processing so only 10hr/day available 
 
data node targeting -- helps to have data going to same machine 
 * 
 * 
 * stsci main targets for LAP/3PI -- MD have less file volume to move when finished
 * ipp067--081 are nearly full and need to have lighter allocations
 * ipp082-095 various levels of larger free space and should have higher allocations

processing loads/schedule groups -- 
 * CNP chip-cam-warp --
 * CNP deepstack -- 
 * CNP iq, refstack -- 
 * CNP WSdiff --
 * CNP staticsky,skycal --
 * CNP full-forced phot -- 

 * MD chip-cam-warp -- finished
 * MD deepstack --
 * MD refstack (multiple sets for out-of-season) -- 
 * MD WSdiff --
 * MD iq, night stack --
 * MD SSdiff -- 
 * MD staticsky, skycal -- 
 * MD full-forced phot

----