Changeset 5397 for trunk/Ohana/src/addstar/doc/notes.txt

Timestamp:

Oct 20, 2005, 8:21:35 AM (21 years ago)

Author:

eugene

Message:

cleaned changes, notes

File:

• trunk/Ohana/src/addstar/doc/notes.txt (modified) (2 diffs)

trunk/Ohana/src/addstar/doc/notes.txt

@@ -16 +16 @@
     with sorted measure tables; add this as a feature 
     of the load_catalog API?
+
+2005.10.19
+
+        I have added the client / server mode, and tested it out to a
+        fair degree.  It handles all of the available addstar modes,
+        including -ref and -cat.  A handful of things still need to be
+        done, including pushing some of the logic to enforce sorting
+        into the dvo load_catalog functions.  Also, I need to correct
+        the idiosyncratic problem with sky vs Myyyy for the skyprobe
+        data.  There are a handful of other improvements that are
+        needed before addstar / DVO is ready for IPP release.
+        However, at this point, it is ready for internal use by the
+        grad students, but not yet ready for CFHT use with Elixir.
 
 2005.10.14
@@ -122 +135 @@
   from libohana.  That is probably not a bad plan in any case...
 
+2005.10.07
+
+        I have finished the basic implementation of the update mode.
+        I have been able to demonstrate substantial improvements in
+        speed when the number of existing measurements dominates the
+        total number of measurements and the number of averages is
+        typically small compared to the number of measurements (ie,
+        most objects are real, detected in most images, and each new
+        image supplies many new measurements of objects which exist
+        and not many of objects which don't exist already).  The speed
+        gain is significant in this context because the average table
+        is small compared to the measure table; since both update and
+        full-load methods require the complete average table, there is
+        no difference in the load time for the average table.  
+
+        I was having some memory collision problems, and attempting to
+        use the ohana_allocate functions reminded me that the libFITS
+        functions were not supported under ohana_allocate.  This was
+        unhelpful.  I bit the bullet and split libohana into libohana
+        (base functions only, including ohana_allocate) and libdvo
+        (functions based on the libautocode structures).  Doing this
+        allowed me to make libFITS depend on libohana (including
+        ohana_allocate).  BUT, this forced me to change all LDFLAGS
+        entries in ohana to swap -lohana -lFITS for -lFITS -lohana,
+        and to add include <fitsio.h> in some cases.
+
+2005.10.06
+        split / nosort / update
+        I have added a few new concepts to addstar recently: split
+        catalog files, nosort for the measurement table, and
+        update-only.
+
+        split mode
+
+        The split mode is quite straightforeward.  In this mode, each
+        catalog is represented by a set of four files: *.cpt, *.cpm,
+        *.cpn, *.cps.  Each file contains only one FITS table of the
+        data, along with basic header and empty matrix.  Having
+        individual tables for each component of the database lets me
+        add entries without re-writing the entire table.  This should
+        save on I/O operations in the long run.  
+
+        The first file contains the table of averages, and is the file
+        normally identified in the table lookup functions.  The header
+        of this file contains the names of the other table files
+        (paths relative to the directory containing the cpt file).
+        The names and extensions are specified in 'mkcatalog.c'; all
+        other functions use the defined filename references, rather
+        than expecting a naming convention.
+
+        The additional files contain the measures (cpm), missings
+        (cpn), and secfilt (cps) elements of the catalog tables.  
+
+        To facilitate the handling of the additional filenames, file
+        pointers, and headers, Catalog was extended to include
+        pointers to the measure, missing, and secfilt files as
+        additional catalogs.  When the data are loaded into memory,
+        these catalogs are locked (as usual), and file information is
+        stored in the individual Catalog entries; the data segements
+        are all loaded into the main catalog pointers (eg, measures
+        are loaded into catalog[0].measure, rather than
+        catalog[0].measure_catalog[0].measure).
+        
+        The function 'load_catalog' auto-recognizes the SPLIT format
+        by looking for the header keyword MEASURE, identifying the
+        file containing the measures.  The identification of the RAW
+        format and the SPLIT format are not cross-checked: if the
+        NAXIS keyword is set to 2, the file is assumed to be RAW, even
+        if the MEASURE keyword is present.  Careful with this (though
+        there is no reason the main matrix should be used in a basic
+        database table).
+
+        nosort 
+
+        the nosort option by itself provides a minor processing
+        speed-up by deferring the re-sorting of the measurement table
+        until after multiple addstar processes are run.  addstar
+        should not require the measurements to be sorted, so this step
+        can be safetly deferred if only addstars are being performed.
+        the other DVO operations require the sorted table, so the sort
+        must be performed before they are run (either as part of the
+        catalog load, not implemented yet, or with a call to addstar
+        without the -nosort option set.  the real goal of the nosort
+        option is to enable the -update concept in addstar, in which
+        only the new rows are written out; this will only work if
+        addstar can handle unsorted measures. 
+
+        the nosort option required the addition of a 'sorted' element
+        in the Catalog structure to track if the data are sorted or
+        not.  On load, this flag is set based on the value of the
+        header keyword SORTED; if the data is sorted during addstar,
+        the flag is appropriately set, otherwise it is set FALSE be
+        default.  
+
+        The nosort option requires a function which can generate the
+        'next_meas' link sequence based on the measure table.  there
+        is now a function called 'build_meas_link' which generates a
+        correct link list; there is also the pair of functions
+        'init_meas_links' and'init_miss_links' to generate the links
+        in the event that the table is sorted (should be must
+        quicker).
+
+        The 'missing' table is problematic: the LONEOS and ELIXIR
+        formats do not carry an averef entry, thus they do not have
+        enough information to define the links based only on the
+        missing table.  This means we are forced to write out a sorted
+        missing table; the nosort option is invalid for the missing
+        table.  One future upgrade path is to add the averef entry to
+        the PANSTARRS format and then only require the missing table
+        to be sorted if the format is old and does not support
+        -nosort.  (Note also that, for the moment, the missing table
+        has only a single valid format).
+
+        In the process of defining the nosort option, I also cleaned
+        up a bit the find_matches functions to use clearer functions
+        for the links.  
+        
+        update
+
+        The 'update' process in principle allows addstar to
+        substantially reduce the amount of I/O it needs to perform by
+        only requiring addstar to write out new measures and new
+        average/secfilt entries.
+
+        The 'missing' table is problematic: since the format does not
+        support the 'nosort' option, it is not possible to use update
+        with the missing table.  This means we are forced to write out
+        a complete, sorted missing table.  This is currently
+        implemented in update_catalog_split by simply writing out the
+        complete missing table.  In fact, this choice is still flawed
+        because the average table, since it is not written out in full
+        each time, is inconsistent with the missing table: the Nn
+        entries for each average, which identifies the number of
+        missing entries, are not updated.  In practice, this means
+        that the -update option forces the use of the -missed option,
+        though at the moment, this is not forced or checked in any
+        way.
+
+        Note that the 'missed' table contains duplicate information
+        and can, in principle, be completely regenerated at any time.
+        This should be an addstar option: to re-construct the missing
+        table, potentially with constraints on the images which are
+        searched for matches.  
+
+2005.10.04
+        - moved measure/missing list manipulation to separate functions
+        - added concept of sorted / unsorted measure catalog
+        - defined build_meas_links and reorder_measure,missing
+        - some cleanup of both find_matches.c and find_matches_closest.c
+
+2005.10.03:
+        - dropping GSCRegion *region entry from find_matches (unused!)
+        - adding function find_matches_closest (alternate matches)
+
+2005.08.19:
+        changed load_photcode to handle CATMODE and CATFORMAT variations
+        - addstar.h: added CATMODE and CATFORMAT globals
+        - ConfigInit: read CATMODE and CATFORMAT from config
+        - gcatalog: set catalog.catmode from CATMODE
+        - mkcatalog: set CATFORMAT and CATMODE for new catalog
+        - wcatalog: set CATFORMAT for new catalog
+        
+        using full photometry conversions in find_matches
+        added SetZeroPoint to gstars to enable phot conversions
+        
+2005.08.15:
+        cleanup of the minor Wall,Werror messages
+
 2005.03.07 : notes related to new version of addstar