Changes between Version 36 and Version 37 of MD.PV3

Timestamp:

Feb 4, 2015, 10:07:39 AM (11 years ago)

Author:

Mark Huber

Comment:

--

MD.PV3

@@ -9 +9 @@
  * [wiki:MD.pv2.iqdeepNtest.20140720] data sample page -- two skycells (edge, center annulus) in MD04 in grizy as a test case for stacks spanning ~15 input stack through an all exposure input stack
 
-This is a summary of the MD.PV3 full pixel reprocessing. Start date and schedule a mess with the unstable processing 12/01/2014. 
+This is a overview summary of the MD.PV3 full pixel reprocessing. Start date and schedule a mess with the unstable processing 12/01/2014. 
 
 This processing will also involve other MD-like fields, including CNP (M31 and STS if time allows). Each field can be similar and different to the typical MD criteria for the stacks and diffims, summaries 
@@ -178 +178 @@
 
 
-Revised estimate with machines possible -- 
+Revised estimate with machines possible -- '''all machines being used for 3PI now so estimates now also limited to when nodes returned'''
  * MD: 
-  * deep stack -- now that chip-cam-warp finished, this it the limiting process group as only certain machines can be used ()
-
- * CNP: if the deep stack verifies okay, then the WSdiff are the next large process group (full-forced phot is an unknown)
-  *  
+  * deep stack -- now that chip-cam-warp finished, this it the limiting process group as only certain machines can be used (ipp067-095,ipps00-14,ippx037-044).
+   * the special data nodes (ipp067-095) being used for nightly processing, this could still allow them to be used for deep stacks for ~10 out of 24 hrs
+   * this would be an equivalent of ~34 nodes (10hr on ipp067-095 plus full use of ipps and ippxMD) for deep stacks to finish the remaining 9 MD fields in ~30-35 days
+  * using the same equivalent ~34 nodes for processing the other MD data products while the bulk of the other nodes continue on 3PI, the following estimates become in somewhat of an order
+   * reference stacks --  ~10 days after deep stacks finish
+   * night stacks -- an extra 26 days after ref stacks finish 
+   * WS and SS diffims -- an extra 35-40 + 10 days after night stacks
+   * iq stacks -- minimal, ~1-2 days with 15 nodes and will try to work into during other processing (some time may be possibly shaved off night or diffims if they can be run along side deep as well)
+   * staticsky/skycal -- an extra 5 days after stacks and diffs
+   * full-forced -- mostly unknown still
+
+ * CNP: if the deep stack verifies okay, then the WSdiff are the next large process group (full-forced phot is an unknown) but can be run on mostly any type of node as well as the reference and iq stacks and have been folded into the times for MD above.
+
+ * if by the time deep stacks are finished, 3PI may also be mostly finished and the remaining general MD products could be pushed through more quickly w/ any of the ippx+c nodes -- maybe 20 days after that in a best case (there is still an order dependence as reference and night stacks need to be finished before WS and SS respectively can be started)
+
+
 
 ----
@@ -194 +206 @@
  * chip-camera-warp:
   * check if refcat oddities in field is a problem -- Eddie and Doug catalogs to compare to
-  * new refcat astrometry and zeropoint verification tests -- 
-  * CNP tessellation check -- 
-  * MD as a targeted and utilized throughout the survey needs to have a summary of exposures changed/lost in PV3
+  * new refcat astrometry and zeropoint verification tests -- the new refcat demonstrated large astrometry and zeropoint variations when previous config used
+  * CNP tessellation check and exposure filling factor
+  * MD as a targeted survey and utilized throughout the different processing needs to have a summary of exposures changed/lost in PV3 -- PV3 vs PV2 and original processing comparison in quality faults and astrometry/zeropoint variations is critical so events published in transients work is reproduced
   * background restore M31 and masking for M31/STS -- test samples to be run and verified for final version
  * stack  
@@ -224 +236 @@
 Should be started ASAP, so that the warps are available for some early testing of the stacks and diffim processing.
 
-Testing necessary for the M31, STS and CNP processing still -- priority for CNP as last reprocessing was ~2011. 
+Testing necessary for the M31, STS processing still -- priority for CNP as last reprocessing was ~2011.
+
+(need to add summary issues and changes in processing/configs here)
+
+Due to network and system disorder, the MD+CNP chip-camera-warp processing was primarily done and has now finished. Verification and summary information on the notes pages
+ * [wiki:MD.PV3-MDnotes]
+ * [wiki:MD.PV3-CNPnotes]
 
 ----
 
 === stack ===
-Order of operations important -- while deep stacks take 1.5 months or so to finish, night and seasonal reference stacks must be finished before diffims can be started. Lighter jobs like difference images and iq stacks (even reference stacks) can be run along with the deep stacks for some fields.
-
-==== night ====
-
-==== seasonal reference ====
-
-==== deep ====
-
-==== iq ====
-
-==== photometry ====
+Order of operations important -- while deep stacks take 1.5 months or so to finish, night and seasonal reference stacks must be finished before diffims can be started. Lighter jobs like difference images and iq stacks (even possibly reference stacks) can be run along with the deep stacks for some fields.
+
+(should have own subsections for adding summary details on config/processing changes)
+
+ * deep --
+  * CNP -- 
+  * MD09 --
+
+ * seasonal reference --
+
+ * night --
+
+ * iq -- 
+
+ * photometry --
+
 
 ----
@@ -246 +269 @@
 === difference image ===
 
-Cannot start until night and seasonal reference stacks finish for SSdiff.
+(should have own subsections for adding summary details on config/processing changes)
+
+ * WSdiff -- cannot start until seasonal reference stacks finish 
+
+ * SSdiff -- cannot start until night and seasonal reference stacks finish 
 
 ----