Index: /trunk/Ohana/src/addstar/doc/Changes.txt =================================================================== --- /trunk/Ohana/src/addstar/doc/Changes.txt (revision 5396) +++ /trunk/Ohana/src/addstar/doc/Changes.txt (revision 5397) @@ -1,185 +1,24 @@ -2005.10.19 - I have added the client / server mode, and tested it out to a - fair degree. It handles all of the available addstar modes, - including -ref and -cat. A handful of things still need to be - done, including pushing some of the logic to enforce sorting - into the dvo load_catalog functions. Also, I need to correct - the idiosyncratic problem with sky vs Myyyy for the skyprobe - data. There are a handful of other improvements that are - needed before addstar / DVO is ready for IPP release. - However, at this point, it is ready for internal use by the - grad students, but not yet ready for CFHT use with Elixir. +2005.10.20 : addstar-1.3 -2005.10.07 + This release incorportates many substantial improvements + needed to handle the panstarrs data problems: + - internal data representation now high precision + - multiple data storage formats (LONEOS, ELIXIR, PANSTARRS) + - multiple data storage modes (RAW, MEF, SPLIT) + - alternative matching scheme (-closest) + - sorted output tables now optional + - incremental updates possible for inserts + - client / server set now defined - I have finished the basic implementation of the update mode. - I have been able to demonstrate substantial improvements in - speed when the number of existing measurements dominates the - total number of measurements and the number of averages is - typically small compared to the number of measurements (ie, - most objects are real, detected in most images, and each new - image supplies many new measurements of objects which exist - and not many of objects which don't exist already). The speed - gain is significant in this context because the average table - is small compared to the measure table; since both update and - full-load methods require the complete average table, there is - no difference in the load time for the average table. +2005.08.15 : addstar-1.2 - I was having some memory collision problems, and attempting to - use the ohana_allocate functions reminded me that the libFITS - functions were not supported under ohana_allocate. This was - unhelpful. I bit the bullet and split libohana into libohana - (base functions only, including ohana_allocate) and libdvo - (functions based on the libautocode structures). Doing this - allowed me to make libFITS depend on libohana (including - ohana_allocate). BUT, this forced me to change all LDFLAGS - entries in ohana to swap -lohana -lFITS for -lFITS -lohana, - and to add include in some cases. + This is a snapshot release before I begin serious work on the + code to handle alternate formats and so forth needed for the + panstarrs support. Minor updates since v1.1, mostly to fix + 2MASS issues and to stay in sync with the libs. -2005.10.06 - split / nosort / update - I have added a few new concepts to addstar recently: split - catalog files, nosort for the measurement table, and - update-only. - - split mode - - The split mode is quite straightforeward. In this mode, each - catalog is represented by a set of four files: *.cpt, *.cpm, - *.cpn, *.cps. Each file contains only one FITS table of the - data, along with basic header and empty matrix. Having - individual tables for each component of the database lets me - add entries without re-writing the entire table. This should - save on I/O operations in the long run. - - The first file contains the table of averages, and is the file - normally identified in the table lookup functions. The header - of this file contains the names of the other table files - (paths relative to the directory containing the cpt file). - The names and extensions are specified in 'mkcatalog.c'; all - other functions use the defined filename references, rather - than expecting a naming convention. - - The additional files contain the measures (cpm), missings - (cpn), and secfilt (cps) elements of the catalog tables. - - To facilitate the handling of the additional filenames, file - pointers, and headers, Catalog was extended to include - pointers to the measure, missing, and secfilt files as - additional catalogs. When the data are loaded into memory, - these catalogs are locked (as usual), and file information is - stored in the individual Catalog entries; the data segements - are all loaded into the main catalog pointers (eg, measures - are loaded into catalog[0].measure, rather than - catalog[0].measure_catalog[0].measure). - - The function 'load_catalog' auto-recognizes the SPLIT format - by looking for the header keyword MEASURE, identifying the - file containing the measures. The identification of the RAW - format and the SPLIT format are not cross-checked: if the - NAXIS keyword is set to 2, the file is assumed to be RAW, even - if the MEASURE keyword is present. Careful with this (though - there is no reason the main matrix should be used in a basic - database table). - - nosort - - the nosort option by itself provides a minor processing - speed-up by deferring the re-sorting of the measurement table - until after multiple addstar processes are run. addstar - should not require the measurements to be sorted, so this step - can be safetly deferred if only addstars are being performed. - the other DVO operations require the sorted table, so the sort - must be performed before they are run (either as part of the - catalog load, not implemented yet, or with a call to addstar - without the -nosort option set. the real goal of the nosort - option is to enable the -update concept in addstar, in which - only the new rows are written out; this will only work if - addstar can handle unsorted measures. - - the nosort option required the addition of a 'sorted' element - in the Catalog structure to track if the data are sorted or - not. On load, this flag is set based on the value of the - header keyword SORTED; if the data is sorted during addstar, - the flag is appropriately set, otherwise it is set FALSE be - default. - - The nosort option requires a function which can generate the - 'next_meas' link sequence based on the measure table. there - is now a function called 'build_meas_link' which generates a - correct link list; there is also the pair of functions - 'init_meas_links' and'init_miss_links' to generate the links - in the event that the table is sorted (should be must - quicker). - - The 'missing' table is problematic: the LONEOS and ELIXIR - formats do not carry an averef entry, thus they do not have - enough information to define the links based only on the - missing table. This means we are forced to write out a sorted - missing table; the nosort option is invalid for the missing - table. One future upgrade path is to add the averef entry to - the PANSTARRS format and then only require the missing table - to be sorted if the format is old and does not support - -nosort. (Note also that, for the moment, the missing table - has only a single valid format). - - In the process of defining the nosort option, I also cleaned - up a bit the find_matches functions to use clearer functions - for the links. - - update - - The 'update' process in principle allows addstar to - substantially reduce the amount of I/O it needs to perform by - only requiring addstar to write out new measures and new - average/secfilt entries. - - The 'missing' table is problematic: since the format does not - support the 'nosort' option, it is not possible to use update - with the missing table. This means we are forced to write out - a complete, sorted missing table. This is currently - implemented in update_catalog_split by simply writing out the - complete missing table. In fact, this choice is still flawed - because the average table, since it is not written out in full - each time, is inconsistent with the missing table: the Nn - entries for each average, which identifies the number of - missing entries, are not updated. In practice, this means - that the -update option forces the use of the -missed option, - though at the moment, this is not forced or checked in any - way. - - Note that the 'missed' table contains duplicate information - and can, in principle, be completely regenerated at any time. - This should be an addstar option: to re-construct the missing - table, potentially with constraints on the images which are - searched for matches. - -2005.10.04 - - moved measure/missing list manipulation to separate functions - - added concept of sorted / unsorted measure catalog - - defined build_meas_links and reorder_measure,missing - - some cleanup of both find_matches.c and find_matches_closest.c - -2005.10.03: - - dropping GSCRegion *region entry from find_matches (unused!) - - adding function find_matches_closest (alternate matches) - -2005.08.19: - changed load_photcode to handle CATMODE and CATFORMAT variations - - addstar.h: added CATMODE and CATFORMAT globals - - ConfigInit: read CATMODE and CATFORMAT from config - - gcatalog: set catalog.catmode from CATMODE - - mkcatalog: set CATFORMAT and CATMODE for new catalog - - wcatalog: set CATFORMAT for new catalog - - using full photometry conversions in find_matches - added SetZeroPoint to gstars to enable phot conversions - -2005.08.15: - cleanup of the minor Wall,Werror messages - -2005.07.06 : current release is addstar-1.1 +2005.07.06 : addstar-1.1 I have made a variety of fairly substantial changes since Index: /trunk/Ohana/src/addstar/doc/notes.txt =================================================================== --- /trunk/Ohana/src/addstar/doc/notes.txt (revision 5396) +++ /trunk/Ohana/src/addstar/doc/notes.txt (revision 5397) @@ -16,4 +16,17 @@ with sorted measure tables; add this as a feature of the load_catalog API? + +2005.10.19 + + I have added the client / server mode, and tested it out to a + fair degree. It handles all of the available addstar modes, + including -ref and -cat. A handful of things still need to be + done, including pushing some of the logic to enforce sorting + into the dvo load_catalog functions. Also, I need to correct + the idiosyncratic problem with sky vs Myyyy for the skyprobe + data. There are a handful of other improvements that are + needed before addstar / DVO is ready for IPP release. + However, at this point, it is ready for internal use by the + grad students, but not yet ready for CFHT use with Elixir. 2005.10.14 @@ -122,4 +135,172 @@ from libohana. That is probably not a bad plan in any case... +2005.10.07 + + I have finished the basic implementation of the update mode. + I have been able to demonstrate substantial improvements in + speed when the number of existing measurements dominates the + total number of measurements and the number of averages is + typically small compared to the number of measurements (ie, + most objects are real, detected in most images, and each new + image supplies many new measurements of objects which exist + and not many of objects which don't exist already). The speed + gain is significant in this context because the average table + is small compared to the measure table; since both update and + full-load methods require the complete average table, there is + no difference in the load time for the average table. + + I was having some memory collision problems, and attempting to + use the ohana_allocate functions reminded me that the libFITS + functions were not supported under ohana_allocate. This was + unhelpful. I bit the bullet and split libohana into libohana + (base functions only, including ohana_allocate) and libdvo + (functions based on the libautocode structures). Doing this + allowed me to make libFITS depend on libohana (including + ohana_allocate). BUT, this forced me to change all LDFLAGS + entries in ohana to swap -lohana -lFITS for -lFITS -lohana, + and to add include in some cases. + +2005.10.06 + split / nosort / update + I have added a few new concepts to addstar recently: split + catalog files, nosort for the measurement table, and + update-only. + + split mode + + The split mode is quite straightforeward. In this mode, each + catalog is represented by a set of four files: *.cpt, *.cpm, + *.cpn, *.cps. Each file contains only one FITS table of the + data, along with basic header and empty matrix. Having + individual tables for each component of the database lets me + add entries without re-writing the entire table. This should + save on I/O operations in the long run. + + The first file contains the table of averages, and is the file + normally identified in the table lookup functions. The header + of this file contains the names of the other table files + (paths relative to the directory containing the cpt file). + The names and extensions are specified in 'mkcatalog.c'; all + other functions use the defined filename references, rather + than expecting a naming convention. + + The additional files contain the measures (cpm), missings + (cpn), and secfilt (cps) elements of the catalog tables. + + To facilitate the handling of the additional filenames, file + pointers, and headers, Catalog was extended to include + pointers to the measure, missing, and secfilt files as + additional catalogs. When the data are loaded into memory, + these catalogs are locked (as usual), and file information is + stored in the individual Catalog entries; the data segements + are all loaded into the main catalog pointers (eg, measures + are loaded into catalog[0].measure, rather than + catalog[0].measure_catalog[0].measure). + + The function 'load_catalog' auto-recognizes the SPLIT format + by looking for the header keyword MEASURE, identifying the + file containing the measures. The identification of the RAW + format and the SPLIT format are not cross-checked: if the + NAXIS keyword is set to 2, the file is assumed to be RAW, even + if the MEASURE keyword is present. Careful with this (though + there is no reason the main matrix should be used in a basic + database table). + + nosort + + the nosort option by itself provides a minor processing + speed-up by deferring the re-sorting of the measurement table + until after multiple addstar processes are run. addstar + should not require the measurements to be sorted, so this step + can be safetly deferred if only addstars are being performed. + the other DVO operations require the sorted table, so the sort + must be performed before they are run (either as part of the + catalog load, not implemented yet, or with a call to addstar + without the -nosort option set. the real goal of the nosort + option is to enable the -update concept in addstar, in which + only the new rows are written out; this will only work if + addstar can handle unsorted measures. + + the nosort option required the addition of a 'sorted' element + in the Catalog structure to track if the data are sorted or + not. On load, this flag is set based on the value of the + header keyword SORTED; if the data is sorted during addstar, + the flag is appropriately set, otherwise it is set FALSE be + default. + + The nosort option requires a function which can generate the + 'next_meas' link sequence based on the measure table. there + is now a function called 'build_meas_link' which generates a + correct link list; there is also the pair of functions + 'init_meas_links' and'init_miss_links' to generate the links + in the event that the table is sorted (should be must + quicker). + + The 'missing' table is problematic: the LONEOS and ELIXIR + formats do not carry an averef entry, thus they do not have + enough information to define the links based only on the + missing table. This means we are forced to write out a sorted + missing table; the nosort option is invalid for the missing + table. One future upgrade path is to add the averef entry to + the PANSTARRS format and then only require the missing table + to be sorted if the format is old and does not support + -nosort. (Note also that, for the moment, the missing table + has only a single valid format). + + In the process of defining the nosort option, I also cleaned + up a bit the find_matches functions to use clearer functions + for the links. + + update + + The 'update' process in principle allows addstar to + substantially reduce the amount of I/O it needs to perform by + only requiring addstar to write out new measures and new + average/secfilt entries. + + The 'missing' table is problematic: since the format does not + support the 'nosort' option, it is not possible to use update + with the missing table. This means we are forced to write out + a complete, sorted missing table. This is currently + implemented in update_catalog_split by simply writing out the + complete missing table. In fact, this choice is still flawed + because the average table, since it is not written out in full + each time, is inconsistent with the missing table: the Nn + entries for each average, which identifies the number of + missing entries, are not updated. In practice, this means + that the -update option forces the use of the -missed option, + though at the moment, this is not forced or checked in any + way. + + Note that the 'missed' table contains duplicate information + and can, in principle, be completely regenerated at any time. + This should be an addstar option: to re-construct the missing + table, potentially with constraints on the images which are + searched for matches. + +2005.10.04 + - moved measure/missing list manipulation to separate functions + - added concept of sorted / unsorted measure catalog + - defined build_meas_links and reorder_measure,missing + - some cleanup of both find_matches.c and find_matches_closest.c + +2005.10.03: + - dropping GSCRegion *region entry from find_matches (unused!) + - adding function find_matches_closest (alternate matches) + +2005.08.19: + changed load_photcode to handle CATMODE and CATFORMAT variations + - addstar.h: added CATMODE and CATFORMAT globals + - ConfigInit: read CATMODE and CATFORMAT from config + - gcatalog: set catalog.catmode from CATMODE + - mkcatalog: set CATFORMAT and CATMODE for new catalog + - wcatalog: set CATFORMAT for new catalog + + using full photometry conversions in find_matches + added SetZeroPoint to gstars to enable phot conversions + +2005.08.15: + cleanup of the minor Wall,Werror messages + 2005.03.07 : notes related to new version of addstar