Changes between Version 152 and Version 153 of ippToPsps
- Timestamp:
- Aug 3, 2011, 3:41:57 PM (15 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
ippToPsps
v152 v153 34 34 The tools chosen for {{{ippToPsps}}} are those considered to be the most effective to tackle the task at hand. As a result, {{{ippToPsps}}} is not consistant with the rest of the IPP code-base (which is, predominately, Perl and C), but since its role is essentially ''outside'' the IPP, this, to me, is not an issue. 35 35 36 === MySQL === 36 '''MySQL''' 37 37 38 38 Because we are dealing with table data that requires some intensive manipulation, it follows that a relational database be used. Relational databases are highly optimized to provide extremely fast query times, especially when indexing in incorporated. Thus we gain speed over the more obvious route of read-a-table-from-FITS-into-an-array-then-loop-through-each-value etc. … … 42 42 {{{ippToPsps}}} uses two MySQL databases, a ''scratch'' database, used to import tables and manipulate then before discarding them, and the ''ippToPsps'' database, which keeps track of which batches have been processed, published to PSPS etc. 43 43 44 === Jython and STILTS === 44 '''Jython and STILTS''' 45 45 46 46 {{{ippToPsps}}} is written in [http://www.jython.org/ Jython], this is in part to take full advantage of the [http://www.star.bris.ac.uk/~mbt/stilts/ STILTS] package, which enables very fast and efficient processing of astronomical catalog data tables. Since it supports FITS, VOTable, and SQL, it is a perfect fit for this project. It is also software maintained elsewhere, reducing this burden on us. … … 79 79 It may take a matter of days to convert a relatively small DVO database to MySQL, however, querying the MySQL database is hugely faster than accessing DVO directly, especially for regions of sky with a high density of detections, such as the medium deep fields. (This was seen when loading MD4 prior to the Boston meeting in May 2011. DVO access per exposure was 40 minutes, whereas, once imported to MySQL, query time was roughly 30 seconds.) 80 80 81 === Image ID confusion === 81 '''Image ID confusion''' 82 82 83 83 We access DVO via a combination of 'source ID' and 'image ID'. Both numbers come from the smf file. However, {{{IMAGE_ID}}} in the smf does not correspond to {{{IMAGE_ID}}} in DVO, instead, it maps to {{{EXTERN_ID}}} in the Images.dat file at the top-level of a given DVO database. The {{{IMAGE_ID}}} column of the same table maps instead to {{{IMAGE_ID}}} in the various 'cpm' (''measurements'') files contained within the subdirectories of the same DVO database. … … 229 229 230 230 231 == = Advantages over previous design ===231 == Advantages over previous design == 232 232 233 233 - no need for second datastore (not a big overhead, but it would require additional systems administration in an already complicated system)
