Changeset 5397 for trunk/Ohana/src/addstar/doc/Changes.txt
- Timestamp:
- Oct 20, 2005, 8:21:35 AM (21 years ago)
- File:
-
- 1 edited
-
trunk/Ohana/src/addstar/doc/Changes.txt (modified) (1 diff)
Legend:
- Unmodified
- Added
- Removed
-
trunk/Ohana/src/addstar/doc/Changes.txt
r5386 r5397 1 1 2 2005.10.193 2 4 I have added the client / server mode, and tested it out to a 5 fair degree. It handles all of the available addstar modes, 6 including -ref and -cat. A handful of things still need to be 7 done, including pushing some of the logic to enforce sorting 8 into the dvo load_catalog functions. Also, I need to correct 9 the idiosyncratic problem with sky vs Myyyy for the skyprobe 10 data. There are a handful of other improvements that are 11 needed before addstar / DVO is ready for IPP release. 12 However, at this point, it is ready for internal use by the 13 grad students, but not yet ready for CFHT use with Elixir. 3 2005.10.20 : addstar-1.3 14 4 15 2005.10.07 5 This release incorportates many substantial improvements 6 needed to handle the panstarrs data problems: 7 - internal data representation now high precision 8 - multiple data storage formats (LONEOS, ELIXIR, PANSTARRS) 9 - multiple data storage modes (RAW, MEF, SPLIT) 10 - alternative matching scheme (-closest) 11 - sorted output tables now optional 12 - incremental updates possible for inserts 13 - client / server set now defined 16 14 17 I have finished the basic implementation of the update mode. 18 I have been able to demonstrate substantial improvements in 19 speed when the number of existing measurements dominates the 20 total number of measurements and the number of averages is 21 typically small compared to the number of measurements (ie, 22 most objects are real, detected in most images, and each new 23 image supplies many new measurements of objects which exist 24 and not many of objects which don't exist already). The speed 25 gain is significant in this context because the average table 26 is small compared to the measure table; since both update and 27 full-load methods require the complete average table, there is 28 no difference in the load time for the average table. 15 2005.08.15 : addstar-1.2 29 16 30 I was having some memory collision problems, and attempting to 31 use the ohana_allocate functions reminded me that the libFITS 32 functions were not supported under ohana_allocate. This was 33 unhelpful. I bit the bullet and split libohana into libohana 34 (base functions only, including ohana_allocate) and libdvo 35 (functions based on the libautocode structures). Doing this 36 allowed me to make libFITS depend on libohana (including 37 ohana_allocate). BUT, this forced me to change all LDFLAGS 38 entries in ohana to swap -lohana -lFITS for -lFITS -lohana, 39 and to add include <fitsio.h> in some cases. 17 This is a snapshot release before I begin serious work on the 18 code to handle alternate formats and so forth needed for the 19 panstarrs support. Minor updates since v1.1, mostly to fix 20 2MASS issues and to stay in sync with the libs. 40 21 41 2005.10.06 42 split / nosort / update 43 I have added a few new concepts to addstar recently: split 44 catalog files, nosort for the measurement table, and 45 update-only. 46 47 split mode 48 49 The split mode is quite straightforeward. In this mode, each 50 catalog is represented by a set of four files: *.cpt, *.cpm, 51 *.cpn, *.cps. Each file contains only one FITS table of the 52 data, along with basic header and empty matrix. Having 53 individual tables for each component of the database lets me 54 add entries without re-writing the entire table. This should 55 save on I/O operations in the long run. 56 57 The first file contains the table of averages, and is the file 58 normally identified in the table lookup functions. The header 59 of this file contains the names of the other table files 60 (paths relative to the directory containing the cpt file). 61 The names and extensions are specified in 'mkcatalog.c'; all 62 other functions use the defined filename references, rather 63 than expecting a naming convention. 64 65 The additional files contain the measures (cpm), missings 66 (cpn), and secfilt (cps) elements of the catalog tables. 67 68 To facilitate the handling of the additional filenames, file 69 pointers, and headers, Catalog was extended to include 70 pointers to the measure, missing, and secfilt files as 71 additional catalogs. When the data are loaded into memory, 72 these catalogs are locked (as usual), and file information is 73 stored in the individual Catalog entries; the data segements 74 are all loaded into the main catalog pointers (eg, measures 75 are loaded into catalog[0].measure, rather than 76 catalog[0].measure_catalog[0].measure). 77 78 The function 'load_catalog' auto-recognizes the SPLIT format 79 by looking for the header keyword MEASURE, identifying the 80 file containing the measures. The identification of the RAW 81 format and the SPLIT format are not cross-checked: if the 82 NAXIS keyword is set to 2, the file is assumed to be RAW, even 83 if the MEASURE keyword is present. Careful with this (though 84 there is no reason the main matrix should be used in a basic 85 database table). 86 87 nosort 88 89 the nosort option by itself provides a minor processing 90 speed-up by deferring the re-sorting of the measurement table 91 until after multiple addstar processes are run. addstar 92 should not require the measurements to be sorted, so this step 93 can be safetly deferred if only addstars are being performed. 94 the other DVO operations require the sorted table, so the sort 95 must be performed before they are run (either as part of the 96 catalog load, not implemented yet, or with a call to addstar 97 without the -nosort option set. the real goal of the nosort 98 option is to enable the -update concept in addstar, in which 99 only the new rows are written out; this will only work if 100 addstar can handle unsorted measures. 101 102 the nosort option required the addition of a 'sorted' element 103 in the Catalog structure to track if the data are sorted or 104 not. On load, this flag is set based on the value of the 105 header keyword SORTED; if the data is sorted during addstar, 106 the flag is appropriately set, otherwise it is set FALSE be 107 default. 108 109 The nosort option requires a function which can generate the 110 'next_meas' link sequence based on the measure table. there 111 is now a function called 'build_meas_link' which generates a 112 correct link list; there is also the pair of functions 113 'init_meas_links' and'init_miss_links' to generate the links 114 in the event that the table is sorted (should be must 115 quicker). 116 117 The 'missing' table is problematic: the LONEOS and ELIXIR 118 formats do not carry an averef entry, thus they do not have 119 enough information to define the links based only on the 120 missing table. This means we are forced to write out a sorted 121 missing table; the nosort option is invalid for the missing 122 table. One future upgrade path is to add the averef entry to 123 the PANSTARRS format and then only require the missing table 124 to be sorted if the format is old and does not support 125 -nosort. (Note also that, for the moment, the missing table 126 has only a single valid format). 127 128 In the process of defining the nosort option, I also cleaned 129 up a bit the find_matches functions to use clearer functions 130 for the links. 131 132 update 133 134 The 'update' process in principle allows addstar to 135 substantially reduce the amount of I/O it needs to perform by 136 only requiring addstar to write out new measures and new 137 average/secfilt entries. 138 139 The 'missing' table is problematic: since the format does not 140 support the 'nosort' option, it is not possible to use update 141 with the missing table. This means we are forced to write out 142 a complete, sorted missing table. This is currently 143 implemented in update_catalog_split by simply writing out the 144 complete missing table. In fact, this choice is still flawed 145 because the average table, since it is not written out in full 146 each time, is inconsistent with the missing table: the Nn 147 entries for each average, which identifies the number of 148 missing entries, are not updated. In practice, this means 149 that the -update option forces the use of the -missed option, 150 though at the moment, this is not forced or checked in any 151 way. 152 153 Note that the 'missed' table contains duplicate information 154 and can, in principle, be completely regenerated at any time. 155 This should be an addstar option: to re-construct the missing 156 table, potentially with constraints on the images which are 157 searched for matches. 158 159 2005.10.04 160 - moved measure/missing list manipulation to separate functions 161 - added concept of sorted / unsorted measure catalog 162 - defined build_meas_links and reorder_measure,missing 163 - some cleanup of both find_matches.c and find_matches_closest.c 164 165 2005.10.03: 166 - dropping GSCRegion *region entry from find_matches (unused!) 167 - adding function find_matches_closest (alternate matches) 168 169 2005.08.19: 170 changed load_photcode to handle CATMODE and CATFORMAT variations 171 - addstar.h: added CATMODE and CATFORMAT globals 172 - ConfigInit: read CATMODE and CATFORMAT from config 173 - gcatalog: set catalog.catmode from CATMODE 174 - mkcatalog: set CATFORMAT and CATMODE for new catalog 175 - wcatalog: set CATFORMAT for new catalog 176 177 using full photometry conversions in find_matches 178 added SetZeroPoint to gstars to enable phot conversions 179 180 2005.08.15: 181 cleanup of the minor Wall,Werror messages 182 183 2005.07.06 : current release is addstar-1.1 22 2005.07.06 : addstar-1.1 184 23 185 24 I have made a variety of fairly substantial changes since
Note:
See TracChangeset
for help on using the changeset viewer.
