Changes between Version 16 and Version 17 of IPP_Progress_Report_20120206

Timestamp:

Feb 13, 2012, 3:14:42 PM (14 years ago)

Author:

eugene

Comment:

--

IPP_Progress_Report_20120206

@@ -5 +5 @@
 
 === Eugene Magnier ===
+
+I have been working on the reference photometry database, incorporating the ubercal zero points.  I have been running tests with a chunk of the sky in which I have generated a database, applied ubercal, run relphot, and have the synthetic photometry merged in.  This region has good (photometric) and bad (non-photometric) data, so it is a good test.  I am ready to examine the quality of the relative photometry, but I have been side-tracked by two other issues:
+
+First, I needed to modify the synthetic photometry database to match the 'depth' (density of the sky-based partition) of the 3pi database.  Unfortunately, trying to do this, I encountered some format handling bugs in the program, dvosplit, which is capable of doing this operation.  I fixed the bugs, and now dvosplit is more robust.  I now have a synth database in the right format to merge with the full 3pi database.  I have also launched the dvomerge for these two (still running as of 2/13).
+
+Second, I was frustrated by the long time to do large-scale database operations in DVO.  I wondered if the I/O rates were substantially worse that I would expect.  I ran some tests and discovered that, although there is some overhead loss to the DVO I/O layer, it looks like we are also largely limited by the disk I/O layer (probably seek time more than just bytes in or out).  I decided that the only way to get any significant speed up would be to parallelize the DVO I/O layer across many machines.  I have worked out how this can be done, and have written much of the supporting structure.  So far I have:
+ 
+ * dvodist : a new program which will distribute DVO database tables across the clusters in an appropriate fashion (and retrieve them to make a single directory database if desired)
+ * setphot : the program which applies ubercal (and other) zero points now has parallel capability (mostly tested and working, but it needs some hand-shaking work).
+ * relphot : the program which does relative photometry now has parallel capability coded, but not yet tested.
+
+One nice point is that the work on these programs illustrates all of the modes of parallelization needed by DVO, and gives example code chunks to be used in the rest of the system.  Thus, it will be fairly quick to parallelize most of the system.  Another nice point is that the database tables are distributed, but links are left behind, so the non-parallel programs can be run as before (with some performance hit going across the network for data).
 
 === Serge Chastel ===