Changes between Version 212 and Version 213 of ippToPsps

Timestamp:

Mar 22, 2012, 9:13:44 AM (14 years ago)

Author:

rhenders

Comment:

--

ippToPsps

@@ -75 +75 @@
 A full description of the {{{ippToPsps}}} database is detailed [wiki:ippToPsps_Database here]
 
-== dvoToMySQL ==
-
-For the special case where we have a 'locked' DVO database, i.e. one that is complete and will not grow any further, we can improve ippToPsps processing time by first generating MySQL tables that include all detections from DVO rather than just the detections-per-exposure (or stack) as above. For this we can use the dvoToMySQL tool, found [http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/browser/trunk/ippToPsps/jython/dvoToMySQL.py here]
-
-It may take a matter of days to convert a relatively small DVO database to MySQL, however, querying the MySQL database is hugely faster than accessing DVO directly, especially for regions of sky with a high density of detections such as the medium deep fields. (This was seen when loading MD4 prior to the Boston meeting in May 2011. DVO access per exposure was 40 minutes, whereas, once imported to MySQL, query time was roughly 30 seconds.)
+== DVO ==
+
+{{{ippToPsps}}} needs to access IPP DVO databases in order to attain various IDs that are assigned within DVO. This is done is two different ways, depending on survey.
+
+DVO has an API written in C that can be used to extract detection records for a give 'IMAGEID' found in cmf and smf file headers (see note below regarding IMAGEID confusion). The usual case is that {{{ippToPsps}}} uses this form of DVO access using a small C program called {{{dvograbber}}}, the source for which is in the {{{ippToPsps/src}}} directory.
+
+{{{ippToPsps}}} has a second way of accessing DVO, which is only intended to be used in the special case of surveys (like MD4), or areas of surveys, have very high coverege in a certain area of sky meaning the underlying DVO FITS grow to a very large large size. For such areas, it becomes unfeasible to access DVO via the C api as repeatedly reading FITS files as large as 2GB causes read tiumes of up to hours per frame.
+
+Instead, for these regions, {{{ippToPsps}}} can pre-ingest a region of DVO into a scratch MySQL database.
+
+It can take a long time to convert a relatively small DVO database to MySQL, however, querying the MySQL database is hugely faster than accessing DVO directly, especially for regions of sky with a high density of detections such as the medium deep fields. (This was seen when loading MD4 prior to the Boston meeting in May 2011. DVO access per exposure was 40 minutes, whereas, once imported to MySQL, query time was roughly 30 seconds.)
 
 '''Image ID confusion'''
 
-We access DVO via a combination of 'source ID' and 'image ID'. Both numbers come from the smf file. However, {{{IMAGE_ID}}} in the smf does not correspond to {{{IMAGE_ID}}} in DVO, instead, it maps to {{{EXTERN_ID}}} in the Images.dat file at the top-level of a given DVO database. The {{{IMAGE_ID}}} column of the same table maps instead to {{{IMAGE_ID}}} in the various 'cpm' (''measurements'') files contained within the subdirectories of the same DVO database.
+We access DVO via a combination of 'source ID' and 'image ID'. Both numbers come from the smf file. However, {{{IMAGEID}}} in the smf does not correspond to {{{IMAGE_ID}}} in DVO, instead, it maps to {{{EXTERN_ID}}} in the Images.dat file at the top-level of a given DVO database. The {{{IMAGE_ID}}} column of the same table maps instead to {{{IMAGE_ID}}} in the various 'cpm' (''measurements'') files contained within the subdirectories of the same DVO database.
 
 = {{{ippToPsps}}} programs =