| | 21 | This week I began to tackle the problem of slow DVO access in regions of the sky where we have lots of coverage, such as the medium deep fields. In these regions the underlying DVO FITS files are huge and cause significant slow-down in loading to PSPS because the current method of data retrieval results in repeated access to the same files. In the past I have overcome this issue by ingesting entire DVO databases into a MySQL database. This is fine for a medium deep field, but impractical for a full-sky DVO that may, in the future, also contain MD fields. So, a more generic solution is required whereby I can pre-load a region of sky from DVO into a database, process all the camera and stack catalog files for that region, then move on to another region. For optimal efficiency, it's important that when moving onto a new bordering region I do not discard already-ingested data that may overlap both regions. It is also necessary to take file modified dates into account, a bit like rsync, so that recently changed files are re-ingested and unchanged ones are not discarded unnecessarily. |
| | 22 | |
| | 23 | Since I already had code to load entire DVO databases into MySQL for faster access I adapted it to ingest a portion at a time, defined by a box of RA/Dec. Steps included: |
| | 24 | |
| | 25 | * code to load SkyList.fits table into MySQL database |
| | 26 | * code to query the above table and get region files contained in RA/Dec box (could be whole sky, or anything smaller) |
| | 27 | * new Db table to maintain list of DVO files imported, including modified date and size and an index number (so we can delete associated detections later) |
| | 28 | * method to determine the total file-size of a DVO region in order to decide optimal time to load |
| | 29 | * rsync-style system that checks file modified dates so that we don't re-ingest unchanged files or ignore recently modified ones |
| | 30 | * process can be stopped and restarted as it remembers what has and hasn't been ingested |
| | 31 | * implemented a method to delete a DVO region from the Db, but retain Images.dat and SkyTable from DVO (to save time) |
| | 32 | |
| | 33 | Future work will be an algorithm to decide which region of the sky is optimal to load next, i.e. figuring out the size available smf/cmf files versus the size on disk in DVO - there is no point ingesting 30GB of DVO data if I only have 2 smf files to load to PSPS. |
| | 34 | |
| | 35 | * Other work: |
| | 36 | * made changes to gpc1 queries for getting stuff in DVO to reflect changes made by Heather a few weeks ago |
| | 37 | * some email help for Kent Wood regarding use of fast rather than slow queue |
| | 38 | * some mailing list admin including a new list for STSCI |
| | 39 | |
| | 40 | |
| | 41 | |
