Index: trunk/Ohana/src/addstar/doc/Changes.txt =================================================================== --- trunk/Ohana/src/addstar/doc/Changes.txt (revision 5386) +++ trunk/Ohana/src/addstar/doc/Changes.txt (revision 5397) @@ -1,185 +1,24 @@ -2005.10.19 - I have added the client / server mode, and tested it out to a - fair degree. It handles all of the available addstar modes, - including -ref and -cat. A handful of things still need to be - done, including pushing some of the logic to enforce sorting - into the dvo load_catalog functions. Also, I need to correct - the idiosyncratic problem with sky vs Myyyy for the skyprobe - data. There are a handful of other improvements that are - needed before addstar / DVO is ready for IPP release. - However, at this point, it is ready for internal use by the - grad students, but not yet ready for CFHT use with Elixir. +2005.10.20 : addstar-1.3 -2005.10.07 + This release incorportates many substantial improvements + needed to handle the panstarrs data problems: + - internal data representation now high precision + - multiple data storage formats (LONEOS, ELIXIR, PANSTARRS) + - multiple data storage modes (RAW, MEF, SPLIT) + - alternative matching scheme (-closest) + - sorted output tables now optional + - incremental updates possible for inserts + - client / server set now defined - I have finished the basic implementation of the update mode. - I have been able to demonstrate substantial improvements in - speed when the number of existing measurements dominates the - total number of measurements and the number of averages is - typically small compared to the number of measurements (ie, - most objects are real, detected in most images, and each new - image supplies many new measurements of objects which exist - and not many of objects which don't exist already). The speed - gain is significant in this context because the average table - is small compared to the measure table; since both update and - full-load methods require the complete average table, there is - no difference in the load time for the average table. +2005.08.15 : addstar-1.2 - I was having some memory collision problems, and attempting to - use the ohana_allocate functions reminded me that the libFITS - functions were not supported under ohana_allocate. This was - unhelpful. I bit the bullet and split libohana into libohana - (base functions only, including ohana_allocate) and libdvo - (functions based on the libautocode structures). Doing this - allowed me to make libFITS depend on libohana (including - ohana_allocate). BUT, this forced me to change all LDFLAGS - entries in ohana to swap -lohana -lFITS for -lFITS -lohana, - and to add include in some cases. + This is a snapshot release before I begin serious work on the + code to handle alternate formats and so forth needed for the + panstarrs support. Minor updates since v1.1, mostly to fix + 2MASS issues and to stay in sync with the libs. -2005.10.06 - split / nosort / update - I have added a few new concepts to addstar recently: split - catalog files, nosort for the measurement table, and - update-only. - - split mode - - The split mode is quite straightforeward. In this mode, each - catalog is represented by a set of four files: *.cpt, *.cpm, - *.cpn, *.cps. Each file contains only one FITS table of the - data, along with basic header and empty matrix. Having - individual tables for each component of the database lets me - add entries without re-writing the entire table. This should - save on I/O operations in the long run. - - The first file contains the table of averages, and is the file - normally identified in the table lookup functions. The header - of this file contains the names of the other table files - (paths relative to the directory containing the cpt file). - The names and extensions are specified in 'mkcatalog.c'; all - other functions use the defined filename references, rather - than expecting a naming convention. - - The additional files contain the measures (cpm), missings - (cpn), and secfilt (cps) elements of the catalog tables. - - To facilitate the handling of the additional filenames, file - pointers, and headers, Catalog was extended to include - pointers to the measure, missing, and secfilt files as - additional catalogs. When the data are loaded into memory, - these catalogs are locked (as usual), and file information is - stored in the individual Catalog entries; the data segements - are all loaded into the main catalog pointers (eg, measures - are loaded into catalog[0].measure, rather than - catalog[0].measure_catalog[0].measure). - - The function 'load_catalog' auto-recognizes the SPLIT format - by looking for the header keyword MEASURE, identifying the - file containing the measures. The identification of the RAW - format and the SPLIT format are not cross-checked: if the - NAXIS keyword is set to 2, the file is assumed to be RAW, even - if the MEASURE keyword is present. Careful with this (though - there is no reason the main matrix should be used in a basic - database table). - - nosort - - the nosort option by itself provides a minor processing - speed-up by deferring the re-sorting of the measurement table - until after multiple addstar processes are run. addstar - should not require the measurements to be sorted, so this step - can be safetly deferred if only addstars are being performed. - the other DVO operations require the sorted table, so the sort - must be performed before they are run (either as part of the - catalog load, not implemented yet, or with a call to addstar - without the -nosort option set. the real goal of the nosort - option is to enable the -update concept in addstar, in which - only the new rows are written out; this will only work if - addstar can handle unsorted measures. - - the nosort option required the addition of a 'sorted' element - in the Catalog structure to track if the data are sorted or - not. On load, this flag is set based on the value of the - header keyword SORTED; if the data is sorted during addstar, - the flag is appropriately set, otherwise it is set FALSE be - default. - - The nosort option requires a function which can generate the - 'next_meas' link sequence based on the measure table. there - is now a function called 'build_meas_link' which generates a - correct link list; there is also the pair of functions - 'init_meas_links' and'init_miss_links' to generate the links - in the event that the table is sorted (should be must - quicker). - - The 'missing' table is problematic: the LONEOS and ELIXIR - formats do not carry an averef entry, thus they do not have - enough information to define the links based only on the - missing table. This means we are forced to write out a sorted - missing table; the nosort option is invalid for the missing - table. One future upgrade path is to add the averef entry to - the PANSTARRS format and then only require the missing table - to be sorted if the format is old and does not support - -nosort. (Note also that, for the moment, the missing table - has only a single valid format). - - In the process of defining the nosort option, I also cleaned - up a bit the find_matches functions to use clearer functions - for the links. - - update - - The 'update' process in principle allows addstar to - substantially reduce the amount of I/O it needs to perform by - only requiring addstar to write out new measures and new - average/secfilt entries. - - The 'missing' table is problematic: since the format does not - support the 'nosort' option, it is not possible to use update - with the missing table. This means we are forced to write out - a complete, sorted missing table. This is currently - implemented in update_catalog_split by simply writing out the - complete missing table. In fact, this choice is still flawed - because the average table, since it is not written out in full - each time, is inconsistent with the missing table: the Nn - entries for each average, which identifies the number of - missing entries, are not updated. In practice, this means - that the -update option forces the use of the -missed option, - though at the moment, this is not forced or checked in any - way. - - Note that the 'missed' table contains duplicate information - and can, in principle, be completely regenerated at any time. - This should be an addstar option: to re-construct the missing - table, potentially with constraints on the images which are - searched for matches. - -2005.10.04 - - moved measure/missing list manipulation to separate functions - - added concept of sorted / unsorted measure catalog - - defined build_meas_links and reorder_measure,missing - - some cleanup of both find_matches.c and find_matches_closest.c - -2005.10.03: - - dropping GSCRegion *region entry from find_matches (unused!) - - adding function find_matches_closest (alternate matches) - -2005.08.19: - changed load_photcode to handle CATMODE and CATFORMAT variations - - addstar.h: added CATMODE and CATFORMAT globals - - ConfigInit: read CATMODE and CATFORMAT from config - - gcatalog: set catalog.catmode from CATMODE - - mkcatalog: set CATFORMAT and CATMODE for new catalog - - wcatalog: set CATFORMAT for new catalog - - using full photometry conversions in find_matches - added SetZeroPoint to gstars to enable phot conversions - -2005.08.15: - cleanup of the minor Wall,Werror messages - -2005.07.06 : current release is addstar-1.1 +2005.07.06 : addstar-1.1 I have made a variety of fairly substantial changes since