Index: /trunk/Ohana/src/addstar/doc/Changes.txt
===================================================================
--- /trunk/Ohana/src/addstar/doc/Changes.txt	(revision 5233)
+++ /trunk/Ohana/src/addstar/doc/Changes.txt	(revision 5234)
@@ -1,2 +1,120 @@
+
+2005.10.06
+	split / nosort / update
+	I have added a few new concepts to addstar recently: split
+	catalog files, nosort for the measurement table, and
+	update-only.
+
+	split mode
+
+	The split mode is quite straightforeward.  In this mode, each
+	catalog is represented by a set of four files: *.cpt, *.cpm,
+	*.cpn, *.cps.  Each file contains only one FITS table of the
+	data, along with basic header and empty matrix.  Having
+	individual tables for each component of the database lets me
+	add entries without re-writing the entire table.  This should
+	save on I/O operations in the long run.  
+
+	The first file contains the table of averages, and is the file
+	normally identified in the table lookup functions.  The header
+	of this file contains the names of the other table files
+	(paths relative to the directory containing the cpt file).
+	The names and extensions are specified in 'mkcatalog.c'; all
+	other functions use the defined filename references, rather
+	than expecting a naming convention.
+
+	The additional files contain the measures (cpm), missings
+	(cpn), and secfilt (cps) elements of the catalog tables.  
+
+	To facilitate the handling of the additional filenames, file
+	pointers, and headers, Catalog was extended to include
+	pointers to the measure, missing, and secfilt files as
+	additional catalogs.  When the data are loaded into memory,
+	these catalogs are locked (as usual), and file information is
+	stored in the individual Catalog entries; the data segements
+	are all loaded into the main catalog pointers (eg, measures
+	are loaded into catalog[0].measure, rather than
+	catalog[0].measure_catalog[0].measure).
+	
+	The function 'load_catalog' auto-recognizes the SPLIT format
+	by looking for the header keyword MEASURE, identifying the
+	file containing the measures.  The identification of the RAW
+	format and the SPLIT format are not cross-checked: if the
+	NAXIS keyword is set to 2, the file is assumed to be RAW, even
+	if the MEASURE keyword is present.  Careful with this (though
+	there is no reason the main matrix should be used in a basic
+	database table).
+
+	nosort 
+
+	the nosort option by itself provides a minor processing
+	speed-up by deferring the re-sorting of the measurement table
+	until after multiple addstar processes are run.  addstar
+	should not require the measurements to be sorted, so this step
+	can be safetly deferred if only addstars are being performed.
+	the other DVO operations require the sorted table, so the sort
+	must be performed before they are run (either as part of the
+	catalog load, not implemented yet, or with a call to addstar
+	without the -nosort option set.  the real goal of the nosort
+	option is to enable the -update concept in addstar, in which
+	only the new rows are written out; this will only work if
+	addstar can handle unsorted measures. 
+
+	the nosort option required the addition of a 'sorted' element
+	in the Catalog structure to track if the data are sorted or
+	not.  On load, this flag is set based on the value of the
+	header keyword SORTED; if the data is sorted during addstar,
+	the flag is appropriately set, otherwise it is set FALSE be
+	default.  
+
+	The nosort option requires a function which can generate the
+	'next_meas' link sequence based on the measure table.  there
+	is now a function called 'build_meas_link' which generates a
+	correct link list; there is also the pair of functions
+	'init_meas_links' and'init_miss_links' to generate the links
+	in the event that the table is sorted (should be must
+	quicker).
+
+	The 'missing' table is problematic: the LONEOS and ELIXIR
+	formats do not carry an averef entry, thus they do not have
+	enough information to define the links based only on the
+	missing table.  This means we are forced to write out a sorted
+	missing table; the nosort option is invalid for the missing
+	table.  One future upgrade path is to add the averef entry to
+	the PANSTARRS format and then only require the missing table
+	to be sorted if the format is old and does not support
+	-nosort.  (Note also that, for the moment, the missing table
+	has only a single valid format).
+
+	In the process of defining the nosort option, I also cleaned
+	up a bit the find_matches functions to use clearer functions
+	for the links.  
+	
+	update
+
+	The 'update' process in principle allows addstar to
+	substantially reduce the amount of I/O it needs to perform by
+	only requiring addstar to write out new measures and new
+	average/secfilt entries.
+
+	The 'missing' table is problematic: since the format does not
+	support the 'nosort' option, it is not possible to use update
+	with the missing table.  This means we are forced to write out
+	a complete, sorted missing table.  This is currently
+	implemented in update_catalog_split by simply writing out the
+	complete missing table.  In fact, this choice is still flawed
+	because the average table, since it is not written out in full
+	each time, is inconsistent with the missing table: the Nn
+	entries for each average, which identifies the number of
+	missing entries, are not updated.  In practice, this means
+	that the -update option forces the use of the -missed option,
+	though at the moment, this is not forced or checked in any
+	way.
+
+	Note that the 'missed' table contains duplicate information
+	and can, in principle, be completely regenerated at any time.
+	This should be an addstar option: to re-construct the missing
+	table, potentially with constraints on the images which are
+	searched for matches.  
 
 2005.10.04
Index: /trunk/Ohana/src/addstar/doc/notes.txt
===================================================================
--- /trunk/Ohana/src/addstar/doc/notes.txt	(revision 5233)
+++ /trunk/Ohana/src/addstar/doc/notes.txt	(revision 5234)
@@ -1,10 +1,13 @@
 
-- measure <-> average matches and reordering
+2005.10.06
 
-  next[i] = i + 1
-  
-
-  
-
+  I am getting seg faults from find_matches_closest, apparently related to the
+  partial Measure load.  They occur when trying to realloc secfilt.  I
+  tried to debug using the ohana_allocate stuff, but it causes
+  problems because fitsio does not fall under the ohana memory
+  system.  This is fairly bad; I should probably deal with that issue
+  before pursuing the other stuff.  Do I have to make libfits depend
+  on libohana, in which case I should probably split off libdvo from
+  libohana.  That is probably not a bad plan in any case...
 
 2005.03.07 : notes related to new version of addstar
Index: /trunk/Ohana/src/addstar/include/addstar.h
===================================================================
--- /trunk/Ohana/src/addstar/include/addstar.h	(revision 5233)
+++ /trunk/Ohana/src/addstar/include/addstar.h	(revision 5234)
@@ -103,5 +103,7 @@
 int ONLY_MATCH;
 int CLOSEST;
-int REORDER;
+
+int NOSORT;
+int UPDATE;
 
 int MODE;
@@ -190,6 +192,6 @@
 int add_miss_link (Average *average, int *next, int Nmissing);
 int *build_measure_links (Average *average, int Naverage, Measure *measure, int Nmeasure);
-Measure *reorder_measure (Average *average, int Naverage, Measure *measure, int Nmeasure, int *next);
-Missing *reorder_missing (Average *average, int Naverage, Missing *missing, int Nmissing, int *next_miss);
+Measure *sort_measure (Average *average, int Naverage, Measure *measure, int Nmeasure, int *next);
+Missing *sort_missing (Average *average, int Naverage, Missing *missing, int Nmissing, int *next_miss);
 
 /** 
Index: /trunk/Ohana/src/addstar/src/args.c
===================================================================
--- /trunk/Ohana/src/addstar/src/args.c	(revision 5233)
+++ /trunk/Ohana/src/addstar/src/args.c	(revision 5234)
@@ -23,4 +23,6 @@
   fprintf (stderr, "  -replace              	  : replace time/photcode measurements (no duplication)\n");
   fprintf (stderr, "  -closest             	  : use closest-star algorith\n");
+  fprintf (stderr, "  -nosort             	  : don't re-sort the measure entries (improves speed)\n");
+  fprintf (stderr, "  -update             	  : only update the new rows (foreces -nosort)\n");
   fprintf (stderr, "  -image                	  : only insert image data\n");
   fprintf (stderr, "  -cal                  	  : perform zero-point calibration\n");
@@ -151,10 +153,19 @@
     remove_argument (N, &argc, argv);
   }
-  /* only add new rows (-update) or re-write complete measure table */
-  REORDER = TRUE;
+
+  /* don't re-sort the measure sequence */
+  NOSORT = FALSE;
+  if ((N = get_argument (argc, argv, "-nosort"))) {
+    NOSORT = TRUE;
+    remove_argument (N, &argc, argv);
+  }
+  /* only add new rows (-update) or re-write complete measure table (forces -nosort) */
+  UPDATE = FALSE;
   if ((N = get_argument (argc, argv, "-update"))) {
-    REORDER = FALSE;
-    remove_argument (N, &argc, argv);
-  }
+    UPDATE = TRUE;
+    NOSORT = TRUE;
+    remove_argument (N, &argc, argv);
+  }
+
   /* only add image potion to image table */
   ONLY_IMAGES = FALSE;
Index: /trunk/Ohana/src/addstar/src/build_links.c
===================================================================
--- /trunk/Ohana/src/addstar/src/build_links.c	(revision 5233)
+++ /trunk/Ohana/src/addstar/src/build_links.c	(revision 5234)
@@ -1,5 +1,5 @@
 # include "addstar.h"
 
-/* build the initial links assuming the table is re-ordered */
+/* build the initial links assuming the table is sorted */
 int *init_measure_links (Average *average, int Naverage, Measure *measure, int Nmeasure) {
 
@@ -23,5 +23,5 @@
 }
 
-/* build the initial links assuming the table is re-ordered */
+/* build the initial links assuming the table is sorted */
 int *init_missing_links (Average *average, int Naverage, Missing *missing, int Nmissing) {
 
@@ -94,14 +94,18 @@
       m = next[m];
     }
-    if (m == Nm) continue;
-    next[m] = Nm;
+    if (m == Nm) { 
+      average[averef].Nm = k + 1;
+    } else {
+      average[averef].Nm = k + 2;
+      next[m] = Nm;
+    }
   }
   return (next);
 }
 
-/* Missing does not carry enough information to reconsruct the links
+/* Missing does not carry enough information to reconstruct the links
    we must always save the missing table, if it exists */
 
-Measure *reorder_measure (Average *average, int Naverage, Measure *measure, int Nmeasure, int *next) {
+Measure *sort_measure (Average *average, int Naverage, Measure *measure, int Nmeasure, int *next) {
 
   int i, k, n, N;
@@ -124,5 +128,5 @@
 }
 
-Missing *reorder_missing (Average *average, int Naverage, Missing *missing, int Nmissing, int *next) {
+Missing *sort_missing (Average *average, int Naverage, Missing *missing, int Nmissing, int *next) {
 
   int i, k, n, N;
Index: /trunk/Ohana/src/addstar/src/find_matches.c
===================================================================
--- /trunk/Ohana/src/addstar/src/find_matches.c	(revision 5233)
+++ /trunk/Ohana/src/addstar/src/find_matches.c	(revision 5234)
@@ -8,5 +8,5 @@
   float dX, dY, dR;
   int *N1, *N2,  *next_meas, *next_miss;
-  int Nave, NAVE, Nmeas, NMEAS, Nmiss, NMISS, Nmatch;
+  int Nave, NAVE, Nmeas, NMEAS, Nmiss, NMISS, Nmatch, offset;
   Coords tcoords;
   int Nsecfilt, Nsec;
@@ -41,4 +41,5 @@
   NMEAS = Nmeas = catalog[0].Nmeasure;
   NMISS = Nmiss = catalog[0].Nmissing;
+  offset = catalog[0].Nmeas_off;
   
   /* project onto rectilinear grid with 1 arcsec pixels */
@@ -76,5 +77,5 @@
 
   /* set up pointers for linked list of measure, missing */
-  if (catalog[0].ordered) {
+  if (catalog[0].sorted) {
     next_meas = init_measure_links (catalog[0].average, Nave, catalog[0].measure, Nmeas);
   } else {
@@ -82,5 +83,5 @@
   }    
   next_miss = init_missing_links (catalog[0].average, Nave, catalog[0].missing, Nmiss);
-  /* missing MUST be written ordered, or not at all */
+  /* missing MUST be written 'sorted', or not at all */
 
   /* choose a radius for matches */
@@ -142,5 +143,5 @@
       catalog[0].measure[Nmeas].Mcal_PS     = image[0].Mcal_PS;
       catalog[0].measure[Nmeas].t           = image[0].tzero + 1e-4*stars[N].Y*image[0].trate;  /* trate is in 0.1 msec / row */
-      catalog[0].measure[Nmeas].averef      = n;
+      catalog[0].measure[Nmeas].averef      = n;              /* this must be an absolute sequence number, if partial average is loaded */
       catalog[0].measure[Nmeas].source      = stars[N].code;  /* photcode */
       catalog[0].measure[Nmeas].dophot      = stars[N].dophot;  
@@ -159,4 +160,5 @@
       Mval = (Nsec == -1) ? &catalog[0].average[n].M_PS : &catalog[0].secfilt[n*Nsecfilt+Nsec].M_PS;
       if (*Mval == NO_MAG) *Mval = Mcat;
+      /* in UPDATE mode, this value is not saved; use relphot to recalculate */
 
       /* adds the measurement to the calibration if appropriate color terms are found */
@@ -184,8 +186,13 @@
 	catalog[0].found[n] = Nmeas;
       }
+      /* Nm is updated, but not written out in -update mode (for existing entries)
+	 Nm is recalculated in build_meas_links if loaded table is not sorted */
       catalog[0].average[n].Nm ++;
       Nmeas ++;
 
-      update_coords (&catalog[0].average[n], &catalog[0].measure[0], next_meas);
+      if (!UPDATE) {
+	/* in UPDATE mode, newly calculated coordinates are not saved */
+	update_coords (&catalog[0].average[n], &catalog[0].measure[0], next_meas);
+      }
     }
     i++;
@@ -217,4 +224,5 @@
 
   /* incorporate unmatched image stars, if this star is in field of this catalog */
+  /* these new entries are all written out in UPDATE mode */ 
   for (i = 0; (i < Nstars) && !ONLY_MATCH; i++) {
     /* make sure there is space for next entry */
@@ -245,5 +253,5 @@
     catalog[0].average[Nave].Xm        = NO_MAG;
     catalog[0].average[Nave].Xg        = NO_MAG;
-    catalog[0].average[Nave].offset    = Nmeas;
+    catalog[0].average[Nave].offset    = Nmeas + offset;   /* this needs to be an absolute reference */
     catalog[0].average[Nave].missing   = -1;
     catalog[0].average[Nave].code      = 0;
@@ -261,5 +269,5 @@
     catalog[0].measure[Nmeas].Mcal_PS  	  = image[0].Mcal_PS;
     catalog[0].measure[Nmeas].t        	  = image[0].tzero + 1e-4*stars[N].Y*image[0].trate; /* trate is in 0.1 msec / row */
-    catalog[0].measure[Nmeas].averef   	  = Nave;
+    catalog[0].measure[Nmeas].averef   	  = Nave;           /* XXX EAM : must be absolute Nave if partial read */
     catalog[0].measure[Nmeas].source   	  = stars[N].code;  /* photcode */
     catalog[0].measure[Nmeas].dophot   	  = stars[N].dophot;  
@@ -296,4 +304,6 @@
     }
 
+    /* next[Nmeas] should always be -1 in this context (it is always the only
+       measurement for the star) */
     stars[N].found = Nmeas;
     next_meas[Nmeas] = -1;
@@ -316,11 +326,11 @@
   } */
 
-  if (REORDER) {
-    catalog[0].measure = reorder_measure (catalog[0].average, Nave, catalog[0].measure, Nmeas, next_meas);
-    catalog[0].ordered = TRUE;
+  if (NOSORT) {
+    catalog[0].sorted = FALSE;
   } else {
-    catalog[0].ordered = FALSE;
-  }
-  catalog[0].missing = reorder_missing (catalog[0].average, Nave, catalog[0].missing, Nmiss, next_miss);
+    catalog[0].sorted = TRUE;
+    catalog[0].measure = sort_measure (catalog[0].average, Nave, catalog[0].measure, Nmeas, next_meas);
+  }
+  catalog[0].missing = sort_missing (catalog[0].average, Nave, catalog[0].missing, Nmiss, next_miss);
   /* missing is REQUIRED to be sorted */
 
Index: /trunk/Ohana/src/addstar/src/find_matches_closest.c
===================================================================
--- /trunk/Ohana/src/addstar/src/find_matches_closest.c	(revision 5233)
+++ /trunk/Ohana/src/addstar/src/find_matches_closest.c	(revision 5234)
@@ -3,10 +3,10 @@
 void find_matches_closest (GSCRegion *region, Stars *stars, int Nstars, Catalog *catalog, Image *image, Image *overlap, int Noverlap) {
 
-  int i, j, n, N, J;
+  int i, j, n, N, J, Jmin;
   double X, Y, RADIUS, RADIUS2, Rmin, secz;
   float *X1, *Y1, *X2, *Y2;
   float dX, dY, dR;
   int *N1, *N2,  *next_meas, *next_miss;
-  int Nave, NAVE, Nmeas, NMEAS, Nmiss, NMISS, Nmatch, Jmin;
+  int Nave, NAVE, Nmeas, NMEAS, Nmiss, NMISS, Nmatch, offset;
   Coords tcoords;
   int Nsecfilt, Nsec;
@@ -41,4 +41,5 @@
   NMEAS = Nmeas = catalog[0].Nmeasure;
   NMISS = Nmiss = catalog[0].Nmissing;
+  offset = catalog[0].Nmeas_off;
   
   /* project onto rectilinear grid with 1 arcsec pixels */
@@ -76,5 +77,5 @@
 
   /* set up pointers for linked list of measure, missing */
-  if (catalog[0].ordered) {
+  if (catalog[0].sorted) {
     next_meas = init_measure_links (catalog[0].average, Nave, catalog[0].measure, Nmeas);
   } else {
@@ -82,5 +83,5 @@
   }    
   next_miss = init_missing_links (catalog[0].average, Nave, catalog[0].missing, Nmiss);
-  /* missing MUST be written ordered, or not at all */
+  /* missing MUST be written 'sorted', or not at all */
 
   /* choose a radius for matches */
@@ -174,4 +175,5 @@
     Mval = (Nsec == -1) ? &catalog[0].average[n].M_PS : &catalog[0].secfilt[n*Nsecfilt+Nsec].M_PS;
     if (*Mval == NO_MAG) *Mval = Mcat;
+    /* in UPDATE mode, this value is not saved; use relphot to recalculate */
 
     /* adds the measurement to the calibration if appropriate color terms are found */
@@ -183,4 +185,6 @@
     /* if we choose to calculate RA,DEC averages, see update_coords.c */
 
+    /* Nm is updated, but not written out in -update mode (for existing entries)
+       Nm is recalculated in build_meas_links if loaded table is not sorted */
     stars[N].found = Nmeas;
     catalog[0].found[n] = Nmeas;
@@ -189,4 +193,6 @@
     i++;
   }
+
+  ohana_memcheck (0);
 
   /** add reference for undetected catalog stars **/
@@ -215,4 +221,5 @@
 
   /** incorporate unmatched image stars, if this star is in field of this catalog **/
+  /* these new entries are all written out in UPDATE mode */ 
   for (i = 0; (i < Nstars) && !ONLY_MATCH; i++) {
     /* make sure there is space for next entry */
@@ -243,5 +250,5 @@
     catalog[0].average[Nave].Xm        = NO_MAG;
     catalog[0].average[Nave].Xg        = NO_MAG;
-    catalog[0].average[Nave].offset    = Nmeas;
+    catalog[0].average[Nave].offset    = Nmeas + offset;
     catalog[0].average[Nave].missing   = -1;
     catalog[0].average[Nave].code      = 0;
@@ -316,11 +323,11 @@
   } */
 
-  if (REORDER) {
-    catalog[0].measure = reorder_measure (catalog[0].average, Nave, catalog[0].measure, Nmeas, next_meas);
-    catalog[0].ordered = TRUE;
+  if (NOSORT) {
+    catalog[0].sorted = FALSE;
   } else {
-    catalog[0].ordered = FALSE;
-  }
-  catalog[0].missing = reorder_missing (catalog[0].average, Nave, catalog[0].missing, Nmiss, next_miss);
+    catalog[0].sorted = TRUE;
+    catalog[0].measure = sort_measure (catalog[0].average, Nave, catalog[0].measure, Nmeas, next_meas);
+  }
+  catalog[0].missing = sort_missing (catalog[0].average, Nave, catalog[0].missing, Nmiss, next_miss);
   /* missing is REQUIRED to be sorted */
 
@@ -342,5 +349,5 @@
   gettimeofday (&stop, NULL);
   dtime = DTIME (stop, start);
-  fprintf (stderr, "  match time %8.3f sec for %5d stars, %6d average\n", dtime, Nstars, Nave);
+  fprintf (stderr, "  match time %9.4f sec for %5d stars, %6d average\n", dtime, Nstars, Nave);
 
   return;
Index: /trunk/Ohana/src/addstar/src/gcatalog.c
===================================================================
--- /trunk/Ohana/src/addstar/src/gcatalog.c	(revision 5233)
+++ /trunk/Ohana/src/addstar/src/gcatalog.c	(revision 5234)
@@ -9,5 +9,9 @@
 
   /* read catalog header */
-  mode = LOAD_AVES | LOAD_MEAS | LOAD_MISS | LOAD_SECF;
+  if (UPDATE) {
+    mode = LOAD_AVES | LOAD_MEAS_META | LOAD_MISS | LOAD_SECF;
+  } else {
+    mode = LOAD_AVES | LOAD_MEAS | LOAD_MISS | LOAD_SECF;
+  }
   if (!load_catalog (catalog, mode, VERBOSE)) {
     fprintf (stderr, "ERROR: failure loading catalog\n");
@@ -16,6 +20,6 @@
 
   /* should this be moved into save_catalog?? */
-  status = fits_scan (&catalog[0].header, "ORDERED", "%t", 1, &catalog[0].ordered);
-  if (!status) catalog[0].ordered = TRUE;
+  status = fits_scan (&catalog[0].header, "SORTED", "%t", 1, &catalog[0].sorted);
+  if (!status) catalog[0].sorted = TRUE;
   /* XXX EAM - is this a good choice?  should the default be 'FALSE'? */
 
Index: /trunk/Ohana/src/addstar/src/mkcatalog.c
===================================================================
--- /trunk/Ohana/src/addstar/src/mkcatalog.c	(revision 5233)
+++ /trunk/Ohana/src/addstar/src/mkcatalog.c	(revision 5234)
@@ -111,4 +111,6 @@
   ALLOCATE (catalog[0].secfilt, SecFilt, 1);
   catalog[0].Naverage = catalog[0].Nmeasure = catalog[0].Nmissing = 0;
+  catalog[0].Nave_disk = catalog[0].Nmeas_disk = catalog[0].Nmiss_disk = 0;
+  catalog[0].Nmeas_off = 0;
   catalog[0].Nsecfilt = GetPhotcodeNsecfilt ();
 
Index: /trunk/Ohana/src/addstar/src/wcatalog.c
===================================================================
--- /trunk/Ohana/src/addstar/src/wcatalog.c	(revision 5233)
+++ /trunk/Ohana/src/addstar/src/wcatalog.c	(revision 5234)
@@ -7,9 +7,16 @@
 
   /* should this be moved into save_catalog?? */
-  fits_modify (&catalog[0].header, "ORDERED",  "%t", 1, catalog[0].ordered);
+  fits_modify (&catalog[0].header, "SORTED",  "%t", 1, catalog[0].sorted);
 
-  if (!save_catalog (catalog, VERBOSE)) {
-    fprintf (stderr, "ERROR: failure saving catalog\n");
-    exit (1);
+  if (UPDATE) {
+    if (!update_catalog (catalog, VERBOSE)) {
+      fprintf (stderr, "ERROR: failure updating catalog\n");
+      exit (1);
+    }
+  } else {
+    if (!save_catalog (catalog, VERBOSE)) {
+      fprintf (stderr, "ERROR: failure saving catalog\n");
+      exit (1);
+    }
   }
 
Index: /trunk/Ohana/src/libohana/include/dvo.h
===================================================================
--- /trunk/Ohana/src/libohana/include/dvo.h	(revision 5233)
+++ /trunk/Ohana/src/libohana/include/dvo.h	(revision 5234)
@@ -24,8 +24,9 @@
 
 /* catalog values to be loaded */
-# define LOAD_AVES 0x01
-# define LOAD_MEAS 0x02
-# define LOAD_MISS 0x04
-# define LOAD_SECF 0x08
+# define LOAD_AVES 	0x01
+# define LOAD_MEAS 	0x02
+# define LOAD_MISS 	0x04
+# define LOAD_SECF 	0x08 
+# define LOAD_MEAS_META 0x10
 
 /* invalid mag value */
@@ -128,5 +129,8 @@
   Missing *missing; 
   SecFilt *secfilt;
-  int Naverage, Nmeasure, Nmissing, Nsecfilt;
+  int Naverage, Nmeasure, Nmissing, Nsecfilt;   /* current number of each component */
+  int Nave_disk, Nmeas_disk, Nmiss_disk;        /* number of component on disk */
+  int Nmeas_off;			        /* dist seq of first loaded data value */
+  /* note the different counting for Nsecfilt */
 
   /* pointers to split data files */
@@ -139,5 +143,5 @@
   int catmode;				/* storage mode (raw, mef, split, mysql) */
   int catformat;			/* storage format (elixir, panstarrs, etc) */
-  int ordered;				/* is measure table average-ordered? */
+  int sorted;				/* is measure table average-sorted? */
   
   /* pointers for data manipulation */
@@ -177,4 +181,5 @@
 int load_catalog (Catalog *catalog, char mode, int VERBOSE);
 int save_catalog (Catalog *catalog, char VERBOSE);
+int update_catalog (Catalog *catalog, char VERBOSE);
 
 int FindMosaicForImage (Image *images, int Nimages, int entry);
@@ -291,4 +296,5 @@
 int load_catalog_split (Catalog *catalog, char mode, int VERBOSE);
 int save_catalog_split (Catalog *catalog, char VERBOSE);
+int update_catalog_split (Catalog *catalog, char VERBOSE);
 
 /*** DVO image db I/O Functions ***/
Index: /trunk/Ohana/src/libohana/src/dvo_catalog.c
===================================================================
--- /trunk/Ohana/src/libohana/src/dvo_catalog.c	(revision 5233)
+++ /trunk/Ohana/src/libohana/src/dvo_catalog.c	(revision 5234)
@@ -110,5 +110,10 @@
       break;
     case DVO_MODE_SPLIT:
-      update_catalog_split (catalog, VERBOSE);
+      /* new file needs to use save_catalog_split */
+      if (catalog[0].Nave_disk == 0) {
+	save_catalog_split (catalog, VERBOSE);
+      } else {
+	update_catalog_split (catalog, VERBOSE);
+      }
       break;
     default:
Index: /trunk/Ohana/src/libohana/src/dvo_catalog_split.c
===================================================================
--- /trunk/Ohana/src/libohana/src/dvo_catalog_split.c	(revision 5233)
+++ /trunk/Ohana/src/libohana/src/dvo_catalog_split.c	(revision 5234)
@@ -65,4 +65,6 @@
 
   measure = NULL;
+
+  /* (Full Load) */
   if (mode & LOAD_MEAS) {
     ALLOCATE (measure, Catalog, 1);
@@ -104,6 +106,33 @@
       fprintf (stderr, "Warning: mismatch between Nmeasure in PHU and Table headers (%d vs %d)\n", Nmeasure, catalog[0].Nmeasure);
     }
-    fits_free_header (&header);
-    fits_free_matrix (&matrix);
+    catalog[0].Nmeas_off = 0;
+    fits_free_header (&header);
+    fits_free_matrix (&matrix);
+  }
+
+  /* (Meta Load) */
+  if (mode & LOAD_MEAS_META) {
+    ALLOCATE (measure, Catalog, 1);
+
+    /* get split filename from main header (paths relative to cpt file) */
+    if (!fits_scan (&catalog[0].header, "MEASURE",  "%s", 1, string)) return (FALSE);
+    ALLOCATE (measure[0].filename, char, strlen(path) + strlen(string) + 2);
+    sprintf (measure[0].filename, "%s/%s", path, string);
+
+    /* lock & open catalog file */
+    if (lock_catalog (measure, catalog[0].lockmode) != 1) {
+      fprintf (stderr, "cannot access measure file %s\n", measure[0].filename);
+      exit (2);
+    }
+
+    /* read PHU */
+    if (!fits_load_header (measure[0].f, &measure[0].header)) {
+      if (VERBOSE) fprintf (stderr, "catalog file does not exist: %s\n", measure[0].filename);
+      return (FALSE);
+    }
+
+    /* set up the table size information */
+    catalog[0].Nmeasure  = 0;        /* no rows loaded */
+    catalog[0].Nmeas_off = Nmeasure; /* start of new entries */
   }
   catalog[0].measure_catalog = measure;
@@ -203,4 +232,9 @@
   catalog[0].secfilt_catalog = secfilt;
 
+  /* save the current number so we can do partial updates */
+  catalog[0].Nave_disk  = Naverage;
+  catalog[0].Nmeas_disk = Nmeasure;
+  catalog[0].Nmiss_disk = Nmissing;
+
   return (TRUE);
 }
@@ -275,4 +309,10 @@
     measure = catalog[0].measure_catalog;
 
+    /* XXX EAM : warn about this condition; add code to handle? */
+    if (catalog[0].Nmeas_off != 0) {
+      fprintf (stderr, "WARNING: LOAD_MEAS_META mixed with save??\n");
+      fprintf (stderr, "WARNING: this should not be allowed to happen!\n");
+    }
+
     /* rewind file pointers and truncate (file is still open) */
     fseek (measure[0].f, 0, SEEK_SET);
@@ -394,11 +434,14 @@
 int update_catalog_split (Catalog *catalog, char VERBOSE) {
 
-  int Nitems;
+  int i, Nx, Ny, Nlines;
+  int Nitems, Nskip, Nout, Ndisk, Nstart;
   Matrix matrix;
   Header header;
   FTable ftable;
+  VTable vtable;
   Catalog *measure, *missing, *secfilt;
 
   ftable.header = &header;
+  vtable.header = &header;
 
   if (catalog[0].Naverage == 0) {
@@ -409,5 +452,5 @@
   /* make sure header is consistent with data */
   fits_modify (&catalog[0].header, "NSTARS",   "%d", 1, catalog[0].Naverage);
-  fits_modify (&catalog[0].header, "NMEAS",    "%d", 1, catalog[0].Nmeasure);
+  fits_modify (&catalog[0].header, "NMEAS",    "%d", 1, catalog[0].Nmeasure + catalog[0].Nmeas_off);
   fits_modify (&catalog[0].header, "NMISS",    "%d", 1, catalog[0].Nmissing);
   fits_modify (&catalog[0].header, "NSECFILT", "%d", 1, catalog[0].Nsecfilt);
@@ -424,29 +467,29 @@
   }
 
-  /* in split mode, we can save only part of the data */ 
-
   /*** Average Table ***/
 
   if (catalog[0].average != NULL) {
-    ftruncate (fileno (catalog[0].f), catalog[0].header.size);
-
-    /* this is probably a NOP, do I have to keep it in? */
-    fits_create_matrix (&catalog[0].header, &matrix);
-    if (!fits_fwrite_matrix  (catalog[0].f, &matrix)) {
-      fprintf (stderr, "can't write primary matrix");
-      return (FALSE);
-    }
-    fits_free_matrix (&matrix);
+
+    /* skip past matrix (already at end of header) */
+    Nskip = fits_matrix_size (&catalog[0].header);
+    fseek (catalog[0].f, Nskip, SEEK_CUR);
+
+    /* how many lines to write out? */
+    Nout = catalog[0].Naverage - catalog[0].Nave_disk;
 
     /* write out Average table (convert to FITS table format) */
     AverageToFtable (&ftable, catalog[0].average, catalog[0].Naverage, catalog[0].catformat);
+    /* convert only output rows to vtable */
+    fits_table_to_vtable (&ftable, &vtable, catalog[0].Nave_disk, Nout);
+
     if (!fits_fwrite_Theader (catalog[0].f, &header)) {
       fprintf (stderr, "can't write table header");
       return (FALSE);
     }
-    if (!fits_fwrite_table (catalog[0].f, &ftable)) {
-      fprintf (stderr, "can't write table data");
-      return (FALSE);
-    }
+    if (!fits_fwrite_vtable (catalog[0].f, &vtable)) {
+      fprintf (stderr, "can't write table data");
+      return (FALSE);
+    }
+    fits_free_vtable (&vtable);
     fits_free_table (&ftable);
     fits_free_header (&header);
@@ -459,32 +502,45 @@
     measure = catalog[0].measure_catalog;
 
-    /* rewind file pointers and truncate (file is still open) */
-    fseek (measure[0].f, 0, SEEK_SET);
-    ftruncate (fileno (measure[0].f), 0);
-
-    /* write table PHU header */
-    if (!fits_fwrite_header  (measure[0].f, &measure[0].header)) {
-      fprintf (stderr, "can't write primary header");
-      return (FALSE);
-    }
-
-    /* this is probably a NOP, do I have to keep it in? */
-    fits_create_matrix (&measure[0].header, &matrix);
-    if (!fits_fwrite_matrix  (measure[0].f, &matrix)) {
-      fprintf (stderr, "can't write primary matrix");
-      return (FALSE);
-    }
-    fits_free_matrix (&matrix);
-
-    /* write out Measure table (convert to FITS table format) */
-    MeasureToFtable (&ftable, catalog[0].measure, catalog[0].Nmeasure, catalog[0].catformat);
+    /* skip past PHU header and matrix */
+    Nskip = measure[0].header.size + fits_matrix_size (&measure[0].header);
+    fseek (measure[0].f, Nskip, SEEK_SET);
+
+    Ndisk  = catalog[0].Nmeas_disk;
+    Nstart = catalog[0].Nmeas_disk - catalog[0].Nmeas_off; /* where is first new line? */
+    Nout   = catalog[0].Nmeasure - Nstart;                   /* how many lines to write out? */
+    Nlines = catalog[0].Nmeasure + catalog[0].Nmeas_off;   /* how many lines total in file */
+
+    /* convert to output format FITS table (only rows for output : 0 - Nout) */
+    MeasureToFtable (&ftable, &catalog[0].measure[Nstart], Nout, catalog[0].catformat);
+
+    fits_scan (&header, "NAXIS1", "%d", 1, &Nx);
+    fits_scan (&header, "NAXIS2", "%d", 1, &Ny);
+
+    /* convert all output rows to vtable */
+    ALLOCATE (vtable.row, int, MAX (1, Nout));
+    ALLOCATE (vtable.buffer, char *, MAX (1, Nout));
+    for (i = 0; i < Nout; i++) {
+      ALLOCATE (vtable.buffer[i], char, MAX (1, Nx));
+      memcpy (vtable.buffer[i], &ftable.buffer[i*Nx], Nx);
+      vtable.row[i] = i + Ndisk;
+    }
+
+    /* modify vtable to represent full disk table */
+    fits_modify (&header, "NAXIS2", "%d", 1, Nlines);
+    header.Naxis[1] = Nlines;
+
+    vtable.size = fits_matrix_size (&header);
+    vtable.Nrow = Nout;
+    vtable.pad = vtable.size - Nx*Ny;
+
     if (!fits_fwrite_Theader (measure[0].f, &header)) {
       fprintf (stderr, "can't write table header");
       return (FALSE);
     }
-    if (!fits_fwrite_table (measure[0].f, &ftable)) {
-      fprintf (stderr, "can't write table data");
-      return (FALSE);
-    }
+    if (!fits_fwrite_vtable (measure[0].f, &vtable)) {
+      fprintf (stderr, "can't write table data");
+      return (FALSE);
+    }
+    fits_free_vtable (&vtable);
     fits_free_table (&ftable);
     fits_free_header (&header);
@@ -492,4 +548,6 @@
 
   /*** Missing Table ***/
+  /* missing table CANNOT be written unsorted, thus it is always written 
+     out in full */
 
   if (catalog[0].missing != NULL) {
@@ -535,33 +593,27 @@
     secfilt = catalog[0].secfilt_catalog;
 
-    /* rewind file pointers and truncate (file is still open) */
-    fseek (secfilt[0].f, 0, SEEK_SET);
-    ftruncate (fileno (secfilt[0].f), 0);
-
-    /* write table PHU header */
-    if (!fits_fwrite_header  (secfilt[0].f, &secfilt[0].header)) {
-      fprintf (stderr, "can't write primary header");
-      return (FALSE);
-    }
-
-    /* this is probably a NOP, do I have to keep it in? */
-    fits_create_matrix (&secfilt[0].header, &matrix);
-    if (!fits_fwrite_matrix  (secfilt[0].f, &matrix)) {
-      fprintf (stderr, "can't write primary matrix");
-      return (FALSE);
-    }
-    fits_free_matrix (&matrix);
-
-    /* write out SecFilt table (convert to FITS table format) */
+    /* skip past PHU header and matrix */
+    Nskip = secfilt[0].header.size + fits_matrix_size (&secfilt[0].header);
+    fseek (secfilt[0].f, Nskip, SEEK_SET);
+
+    /* how many lines to write out? */
+    Nout  = catalog[0].Nsecfilt * (catalog[0].Naverage - catalog[0].Nave_disk);
+    Ndisk = catalog[0].Nsecfilt * catalog[0].Nave_disk;
+
+    /* convert to output format FITS table */
     Nitems = catalog[0].Naverage * catalog[0].Nsecfilt;
     SecFiltToFtable (&ftable, catalog[0].secfilt, Nitems, catalog[0].catformat);
+    /* convert only output rows to vtable */
+    fits_table_to_vtable (&ftable, &vtable, Ndisk, Nout);
+
     if (!fits_fwrite_Theader (secfilt[0].f, &header)) {
       fprintf (stderr, "can't write table header");
       return (FALSE);
     }
-    if (!fits_fwrite_table (secfilt[0].f, &ftable)) {
-      fprintf (stderr, "can't write table data");
-      return (FALSE);
-    }
+    if (!fits_fwrite_vtable (secfilt[0].f, &vtable)) {
+      fprintf (stderr, "can't write table data");
+      return (FALSE);
+    }
+    fits_free_vtable (&vtable);
     fits_free_table (&ftable);
     fits_free_header (&header);
@@ -578,2 +630,8 @@
 
 /* XXX EAM : this file needs work on the error exit conditions and memory leaks, esp under errors */
+
+/* XXX EAM : update is not efficient.  MeasureToFtable should only 
+   convert the new rows (Nmeas_disk to Nmeasure). the resulting
+   table represents the end rows of the vtable.  we need to define
+   the vtable based on the ftable, but with Ny = Nmeasure */  
+