| | 14 | |
| | 15 | |
| | 16 | * DVO -> Mysql program written and working: |
| | 17 | * loops through all DVO sub-dirs, pulls in all measurement and average tables, joins and inserts into master dvoDetection table |
| | 18 | * problems with duplicates in DVO - ignoring for now, will investigate later |
| | 19 | * various system problems: broke ipp005 by building big database in the root partition |
| | 20 | * Moved to ippc00, after rsync-ing DVO Db, but too slow, so back to ipp005 - Gavin moved Dbs to partition with some space |
| | 21 | * completed ingest of 167 million detection rows in about 30 hours. Query speed increased from 40 minutes for a full frame to around 20 seconds. |
| | 22 | |
| | 23 | * Testing of detection loading using new MySQL database: |
| | 24 | * path problems - csv dir not understood by ODM |
| | 25 | * DXLayer defaulting to 3PI rather than SCR |
| | 26 | * confusion over IMAGEID key: IMAGEID in smf files is not same as IMAGE_ID in DVO. Using SQL workaround. |
| | 27 | * crazy -999 "NULL" overflowing byte fields and breaking load |
| | 28 | * ImageMeta extension existing for empty Detection extension: fixed |
| | 29 | * duplicate detections in smf files breaking load: altered my tables to reject them and report to the log |
| | 30 | * nightly stacks in DVO do not have header values that were in original stack demo file. Changed code to query gpc1 for them instead |
| | 31 | * deep stacks not in addRun table (they were loaded manually) so I couldn't queue. Heather to the rescue... |
| | 32 | * found rows in StackApFlx and StackModelFit not in StackDetection: ignoring for now |
| | 33 | |
| | 34 | * actual loading: |
| | 35 | * required lots of hand-holding over the weekend, but necessary to get everything loaded in time. |
| | 36 | * encountered bug causing program to halt with a 'database user' prompt. Spent a lot of time diagnosing issue, then fixing it (STILTS functions not closing all their database connections) |
| | 37 | * investigated high numbers of bad flux, duplicate objIDs. Confirmed same result with old code. |
| | 38 | |
| | 39 | * made plot of magic mask fraction across lunations for Ken |
