Changes between Version 302 and Version 303 of ippToPsps
- Timestamp:
- Apr 9, 2012, 1:30:55 PM (14 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
ippToPsps
v302 v303 160 160 The decision of when to pre-ingest DVO into MySQL is made in the {{{loader.py}}}. The {{{queue.py}}} program has already queued up smfs to load by box on the sky. {{{loader.py}}} uses the ratio of smfs files to size of DVO files on disk to determine if it is worthwhile pre-ingesting, or else importing DVO data the usual (C-interface) way. When the loader progresses to the next, neighboring box, the same ratio is calculated. If the decision is made again to pre-ingest from DVO, then any DVO regions overlapping both boxes are retained, while the remainder are purged before importing the new ones. Most of this functionality is handles in the various dvo classes. The Dvo base class can be found [browser:trunk/ippToPsps/jython/dvo.py here]. 161 161 162 DVO ingestion by the Dvo classes works a little like rsync in that file modified dates are maintained in the scratch database and if an out-of-date file is encountered it is re-ingested. This is particularly important for the high level DVO tables like Photcodes. SkyTable and Images.162 DVO ingestion by the Dvo classes works a little like rsync in that file modified dates are maintained in the scratch database and if an out-of-date file is encountered it is re-ingested. This is particularly important for the high level DVO tables like Photcodes. !SkyTable and Images. 163 163 164 164 '''Image ID confusion'''
