Changes between Version 9 and Version 10 of IPP_Progress_Report_20100315

Timestamp:

Mar 23, 2010, 9:56:28 AM (16 years ago)

Author:

eugene

Comment:

--

IPP_Progress_Report_20100315

@@ -5 +5 @@
 
 === Eugene Magnier ===
+
+I spent most of the week working on re-structuring the DVO code to handle large files (>2^31 bytes = 2GB).  This task required changing a small number of basic data types in the Ohana code and modifying a large number of APIs to accept those changes.  Now the Ohana code is capable of handling FITS tables with sizes > 2GB as well as tables with more than 2^32 rows and images with axes larger than 2^32 pixels.  I also added gcc format checking macros to make this code more reliable.  
+
+I also spent some time updating a previously-existing program to merge two DVO databases either into a new third database, or to merge the first into the second.  We are going to use this program (creatively called dvomerge) to enable a new paradigm for the DVO ingest: nightly science will be ingested into a nightly dvo database, which will in turn be merged into a master database (or databases -- one per survey component).  This paradigm allows us to run consistency checks on the nightly database before ingesting it, and it will allow us to more-easily distribute the database.  It will be possible for remote sites to accept just the nightly DVO databases and to perform their own local merge, rather than attempting to rsync the full database.  The nightly databases should be in the range of 10-20GB, while the full-scale database is already >200GB for 3pi, and is expected to reach 20TB.  In order to test these upgrades, I fleshed out the dvo test suite, adding tap-based tests for addstar ingest, relphot, relastro, and dvomerge (these are in Ohana/src/addstar/test).
 
 === Heather Flewelling ===