Index: trunk/Ohana/src/dvomerge/doc/usage.txt
===================================================================
--- trunk/Ohana/src/dvomerge/doc/usage.txt	(revision 30615)
+++ trunk/Ohana/src/dvomerge/doc/usage.txt	(revision 30615)
@@ -0,0 +1,121 @@
+
+dvomerge
+
+dvomerge is used to merge dvo databases.  there are two main modes of operation:
+
+* merge two databases into a third, previously absent, database: dvomerge (input1) and (intput2) to (output)
+* merge one database into an existing database : dvomerge (input) into (output)
+
+Note that all dvomerge options may include the -region (RAmin) (RAmax) (DECmin) (DECmax) options
+  to restrict the merge to a portion of the sky.  
+
+The second mode of operation has two minor variants:
+
+* dvomerge (input) into (output) continue
+  the 'continue' option allows the dvomerge process to operate on a
+  fraction of the sky, in the assumption that an initial merge was
+  performed (with the 'into' mode).  The 'continue' is needed to tell
+  dvomerge that the images of the two databases have already been
+  merged; in this case, dvomerge will examine the image tables and
+  determine the mapping of image IDs from the state of the output
+  database. 
+
+* dvomerge (input) into (output) from (list)
+  the 'from' variant is equivalent to the 'continue' variant, but in
+  this case the catalog tables to be merged are specified in the list
+  file.  The file contains one catalog file per line, with the name
+  giving the directory and filename of the cpt files, without the
+  catdir component (eg, n0000/0001.01.cpt).  
+
+Threads : threading support is available for dvomerge .. into, except
+for the 'from' mode.  Add the option '-threads N' to specify the
+number of threads to be used.  Each thread handles a separate catalog
+table (read from disk, calculate the merge, write to disk).  With only
+a few threads, it is easy to saturate the disk I/O system, at which
+point the threading does not improve the speed of the merge process.
+
+Some other words of warning: 
+
+* dvomerge does not attempt to check for previous merges (successful
+  or not).  A re-run of dvomerge will simply duplicate the data
+  previously merged.  You must track your own results: which cpt files
+  were merged in a given run.
+
+* the region option selects the cpt files which touch the region.  to
+  select exactly the cpt files you want, reduce the region by a small
+  amount to avoid overlapping neighboring regions.  For example, if
+  you want to include only a specific cpt file with RA,DEC bounds of
+  0.0 - 1.0, 2.5 - 3.5, you might use -region 0.01 0.99 2.51 3.49
+
+* the cpt headers define their boundaries (RA0, RA1, DEC0, DEC1).
+
+* the merged dvo database is not sorted (indexed).  To use it with the
+  dvo shell or programs like relphot or relastro, it is necessary to
+  re-index with addstar -resort -D CATDIR (catdir).  (Note that
+  addstar -resort is also now able to use multiple threads: add the
+  -threads N option).
+
+dvorepair
+
+dvorepair can be used to fix damaged dvo databases.  Repair is
+possible because the measurement tables (cpm) and the image table
+together contain all of the information needed to regenerate the
+average (cpt) and secfilt (cps) tables.  
+
+dvorepair has several modes of operation:
+
+* dvorepair -images-vs-measures (catdir) (Ntol)
+
+  this mode scans the database and compares the number of detections
+  tied to a specific image (in the measure tables) with the number of
+  detections claimed by that image (in the average table).  Images
+  which have too few detections are reported to standard out.  The
+  value of Ntol specifies the number of detections which can be
+  dropped -- a few missing detections for a given image is not
+  catastrophic.  
+
+* dvorepair -delete-image-list (catdir) (list) 
+
+  this mode accepts the list of bad images as reported by the
+  -images-vs-measures mode.  It then scans the database and deletes
+  all detections associated with the deleted images.  It also repairs
+  the average (cpt) and secfilt (cps) tables so they are consistent
+  with the repaired measurement tables.  The repaired output tables do
+  not overwrite the existing tables; instead they are written with the
+  suffix ".fixed".  It is up to the user to find and replace the
+  original files with the .fixed version.
+
+  Note that the user may edit the output list from -images-vs-measures
+  and choose not to delete images as desired.  But be careful: the
+  format of the lines of the input file must match the original output
+  format.
+
+  Note that a repaired database is not indexed: you must run addstar
+  -resort to re-index the database.
+
+* dvorepair -fix-images (catdir) (list)
+
+  This mode accepts the same for list file as -delete-image-list, and
+  in this case, deletes the images from the image table (as opposed to
+  the detections).
+
+* dvorepair -fix-cpt (images) (list)
+
+  This mode accepts a list of corrupted cpt files and rebuilds the
+  average and secfilt tables from the measurements.  Instead of a
+  (catdir), the inputs are the full path or relative path to the image
+  table and a list of the full or relative paths to the cpt files
+  which are broken, ** with the .cpt extension removed**.
+
+  Again, repaired cpt, cps, cpm files are saved with the extension .fixed.
+
+dvoverify
+
+dvoverify will scan a dvo database or subset (by region) and verify
+the database integrity.  At the moment, the only check it makes is to
+verify that the tables are FITS compliant.  (ie, no check is made for
+internal consistency of the number of average, measure, or secfilt
+entries).
+
+
+ 
Index: trunk/Ohana/src/dvomerge/src/dvoverify.c
===================================================================
--- trunk/Ohana/src/dvomerge/src/dvoverify.c	(revision 30614)
+++ trunk/Ohana/src/dvomerge/src/dvoverify.c	(revision 30615)
@@ -9,6 +9,9 @@
 
 int VerifyTableFile (char *filename);
+int CheckCatalogIndexes (char *catdir, char *filename,  SkyRegion *region);
 
 # define DEBUG 0
+
+int VERBOSE;
 
 int main (int argc, char **argv) {
@@ -22,4 +25,10 @@
   SkyRegion UserPatch;
   // Catalog catalog;
+
+  VERBOSE = FALSE;
+  if ((N = get_argument (argc, argv, "-v"))) {
+    VERBOSE = TRUE;
+    remove_argument (N, &argc, argv);
+  }
 
   // restrict to a portion of the sky
@@ -83,6 +92,6 @@
     if (i % 1000 == 0) fprintf (stderr, ".");
 
-    sprintf (filename, "%s/%s.cpt", catdir, inlist[0].regions[i][0].name);
-    if (!VerifyTableFile (filename)) {
+    // sprintf (filename, "%s/%s.cpt", catdir, inlist[0].regions[i][0].name);
+    if (!VerifyTableFile (inlist[0].filename[i])) {
       Nbad ++;
     }
@@ -95,4 +104,8 @@
     sprintf (filename, "%s/%s.cpm", catdir, inlist[0].regions[i][0].name);
     if (!VerifyTableFile (filename)) {
+      Nbad ++;
+    }
+
+    if (!CheckCatalogIndexes(catdir, inlist[0].filename[i], inlist[0].regions[i])){
       Nbad ++;
     }
@@ -212,6 +225,125 @@
 }
 
-  // gfits_scan(&cpmHeaderTBL, "NAXIS1", "%d", 1, &NbytesPerRow);
-  // gfits_scan(&cpmHeaderTBL, "NAXIS2", "%d", 1, &Nrows);
+// CheckCatalogIndexes(catdir, inlist[0].regions[i][0].name);
+
+int CheckCatalogIndexes (char *catdir, char *filename,  SkyRegion *region) {
+
+  Catalog catalog;
+  int i, j, m, status;
+
+  status = TRUE;
+
+  // set the parameters which guide catalog open/load/create
+  catalog.filename  = filename;
+  catalog.catformat = dvo_catalog_catformat (CATFORMAT);  // set the default catformat from config data
+  catalog.catmode   = dvo_catalog_catmode (CATMODE);      // set the default catmode from config data
+  catalog.catflags  = LOAD_AVES | LOAD_MEAS;
+  catalog.Nsecfilt  = 0;
+  
+  // an error exit status here is a significant error (disk I/O or file access)
+  if (!dvo_catalog_open (&catalog, region, VERBOSE, "r")) {
+    fprintf (stderr, "ERROR: failure to open catalog file %s\n", catalog.filename);
+    return FALSE;
+  }
+
+  // Naves_disk == 0 implies an empty catalog file, skip empty catalogs
+  if (catalog.Naves_disk == 0) {
+    dvo_catalog_unlock (&catalog);
+    dvo_catalog_free (&catalog);
+    return TRUE;
+  }
+
+  // if the table is SORTED, then the following can be checked
+  // check the following:
+  // measure[j].averef -> average[averef]
+  // measure[j].objID = average[averef].objID
+  // measure[j].catID = average[averef].catID
+  // measure[j].measureOffset < Nmeasure
+  // \sum average[].Nmeasure = Nmeasure
+
+  // if the table is NOT SORTED, we have a subset of checks we can make
+  if (!catalog.sorted) {
+    fprintf (stderr, "!");
+    dvo_catalog_unlock (&catalog);
+    dvo_catalog_free (&catalog);
+    return TRUE;
+  }
+
+  int NmeasureTotal = 0;
+  int measureOffsetOK = TRUE;
+  for (i = 0; i < catalog.Naverage; i++) {
+    NmeasureTotal += catalog.average[i].Nmeasure;
+    if (VERBOSE && !(NmeasureTotal <= catalog.Nmeasure)) {
+      fprintf (stderr, "NmeasureTotal > catalog.Nmeasure: %d %d %d\n", i, catalog.average[i].Nmeasure, (int) catalog.Nmeasure);
+    }
+    measureOffsetOK &= (catalog.average[i].measureOffset < catalog.Nmeasure);
+    if (VERBOSE && !(catalog.average[i].measureOffset < catalog.Nmeasure)) {
+      fprintf (stderr, "measureOffset >= catalog.Nmeasure: %d %d %d\n", i, catalog.average[i].measureOffset, (int) catalog.Nmeasure);
+    }
+    measureOffsetOK &= (catalog.average[i].measureOffset + catalog.average[i].Nmeasure <= catalog.Nmeasure);
+    if (VERBOSE && !(catalog.average[i].measureOffset + catalog.average[i].Nmeasure <= catalog.Nmeasure)) {
+      fprintf (stderr, "measureOffset + Nmeasure > catalog.Nmeasure : %d %d %d\n", i, catalog.average[i].Nmeasure, (int) catalog.Nmeasure);
+    }
+  }
+
+  if (!measureOffsetOK) {
+    fprintf (stderr, "ERROR: catalog %s has an invalid measureOffset\n", catalog.filename);
+    status = FALSE;
+  }
+
+  if (NmeasureTotal != catalog.Nmeasure) {
+    fprintf (stderr, "ERROR: catalog %s has an invalid Nmeasure\n", catalog.filename);
+    status = FALSE;
+  }
+
+  if (!status) {
+    dvo_catalog_unlock (&catalog);
+    dvo_catalog_free (&catalog);
+    return (status);
+  }
+
+  int objIDsOK = TRUE;
+  int catIDsOK = TRUE;
+  int averefOK = TRUE;
+
+  for (i = 0; i < catalog.Naverage; i++) {
+    m = catalog.average[i].measureOffset;
+    for (j = 0; j < catalog.average[i].Nmeasure; j++) {
+      objIDsOK &= (catalog.average[i].objID == catalog.measure[m+j].objID);
+      catIDsOK &= (catalog.average[i].catID == catalog.measure[m+j].catID);
+      averefOK &= (catalog.measure[m+j].averef == i);
+    }
+  }
+    
+  if (!objIDsOK) {
+    fprintf (stderr, "ERROR: catalog %s has invalid obj IDs\n", catalog.filename);
+    status = FALSE;
+  }
+  if (!catIDsOK) {
+    fprintf (stderr, "ERROR: catalog %s has invalid cat IDs\n", catalog.filename);
+    status = FALSE;
+  }
+  if (!averefOK) {
+    fprintf (stderr, "ERROR: catalog %s has invalid averef values\n", catalog.filename);
+    status = FALSE;
+  }
+
+//  for (i = 0; i < catalog.Naverage; i++) {
+//    m = catalog.average[i].measureOffset;
+//    for (j = 0; i < catalog.Nmeasure; i++) {
+//      objIDsOK &= (catalog.average[i].objID == catalog.measure[m+j].objID);
+//      catIDsOK &= (catalog.average[i].catID == catalog.measure[m+j].catID);
+//      averefOK &= (catalog.measure[m+j].averef = i);
+//    }
+//  }
+
+  dvo_catalog_unlock (&catalog);
+  dvo_catalog_free (&catalog);
+
+  return status;
+}
+
+// gfits_scan(&cpmHeaderTBL, "NAXIS1", "%d", 1, &NbytesPerRow);
+// gfits_scan(&cpmHeaderTBL, "NAXIS2", "%d", 1, &Nrows);
     
   
Index: trunk/Ohana/src/dvomerge/src/help.c
===================================================================
--- trunk/Ohana/src/dvomerge/src/help.c	(revision 30614)
+++ trunk/Ohana/src/dvomerge/src/help.c	(revision 30615)
@@ -37,5 +37,6 @@
   fprintf (stderr, "  dvomerge (input) into (output)\n");
   fprintf (stderr, "  dvomerge (input) into (output) continue\n\n");
-  fprintf (stderr, "  dvomerge (input) into (output) from (list)\n\n");
+  fprintf (stderr, "  dvomerge (input) into (output) from (list)\n");
+  fprintf (stderr, "     dvomerge list implies 'continue' : list contains, eg, n1500/1688.00.cpt (one cpt per line)\n\n");
   fprintf (stderr, "  merge DVO databases\n");
   fprintf (stderr, "  optional flags:\n");