Index: branches/eam_branches/ipp-20121219/ippToPsps/doc/upgrade.txt
===================================================================
--- branches/eam_branches/ipp-20121219/ippToPsps/doc/upgrade.txt	(revision 35062)
+++ branches/eam_branches/ipp-20121219/ippToPsps/doc/upgrade.txt	(revision 35062)
@@ -0,0 +1,124 @@
+
+2013.01.24 : Status of the ippToPsps Upgrade
+
+Here are the modifications I have made to ippToPsps:
+
+* set up autoconfig, created a new script 'ippjython' so the code can run from any directory
+  - config/settings.xml : needs a more flexible definition of the database names
+  - on the MHPCC cluster, users need to set the env JAVA to point to the working version in ~ipp
+  - installation does not yet include the installation of the jython.jar file (needs to go into PSCONFIGDIR/share/jar)
+  - I shrunk the size of the SkyCell init table in IN/tables.vot (now tables.IN.vot)
+    - this table was extremely slow to load
+    - I'm not convinced this is used for anything other than init
+    - probably should be inserted into the db with native dvo code?
+  - ditto for ProjetionCell
+
+* created test suites to inject (a) fake data and (b) real gpc1 data into a test db
+  - choice of database should be different for test and for the operational system : see config/settings.xml above
+
+* replaced the stilts-based ingest of the SkyTable.dat file with a native dvo operation (dvopsps skytable)
+
+* added native dvo-based ingest of dvoDetections from dvo database (dvopsps detections)
+
+* added options to switch between native and stilts versions of dvo ingest
+
+* some modifications needed to support the pre-load option (needed regardless of choice of dvopsps or stilts)
+
+* disabled the build of dvograbber (requires libxml which may not be installed, and is basically deprecated)
+
+* allow detections to come from chips not named "XYnn" (eg, for nightly stacks and for tests)
+
+* other fixes
+  * fixed header keyword regex to support values of the form 1.0E+01
+  * fixed getClientsColumnBoolean to return False if no rows are returned
+  * do not crash if FWHM_MAJ, FW_MJ_UQ, PSFMODEL are not in header
+  * missing astrometry (or missing chip) causes chip to be skipped 
+  * added a KEY to dvoDetectionFull on objID, detectID
+  * removed uniqueness requirement for objID on dvoDetectionFull
+  * added capability for stacks to be in non-nebulous locations
+  * removed looped search for stack cmfs (since they now come from skycal with unique names)
+  * added ability to run loader once for a given config
+  * fix the sql used to update the cpm tables (for stilts-based detection ingest)
+  * drop ImageMeta_ tables before reading headers
+
+* some test settings to watch for
+  * ipptopsps and scratch dbs are set to test names by default (config/settings.xml, loader.py, dvo.py)
+  * config.force is set to True (ipptopspsdb.py)
+
+* concerns / fixes still needed:
+  * the rules for ingesting the dvo detections need to be adjusted to be consistent with the dvopsps paradigm
+  * it is not always clear when the scratch db or dvo tables are created
+    * I added a createDvoTables call in setupScratchDb.py, but unclear if it should be there.
+  * the names for the scratch DBs are in multiple places (config/settings.xml, loader.py, dvo.py)
+  * the code is somewhat fragile to unexpected conditions.  for
+    example, failure to select any inputs results in a java exception
+    instead of a clear error message.  there are other cases where
+    selects which return 0 lines cause crashes instead of informative
+    output.
+  * layout of smfs is fairly hard-wired and gpc1-specific (I've
+    removed some of this, but more generality would be good)
+
+  * need to enable -parallel option to dvopsps (config option?)
+  * add multi-row ingest to stilts
+  * need to fix test obj ingest
+  * file handles are not promptly closed -- is this a garbage
+    collection problem?
+
+* operational changes / thoughts
+  * ippToPsps modules are now run with the command 'ippjython (module.py) [options]' from any directory as desired.
+  * all P2 and ST batches use the dvopsps ingest method (assuming dvo.useStilts is left as False)
+
+    This second point affects the choice of config region & box somewhat.  
+
+    The script 'loader.py' loops over the region split into boxes
+    defined by the 'box size' argument of the config.  This loop
+    chooses both the gpc1 exposures (or stacks) to ingest, but also
+    the dvo detctions loaded into the mysql database.
+
+    In the past, the box size was chosen to be similar to a gpc1 footprint.  
+
+    Now (at least for the moment), dvopsps is run for each box,
+    ingesting that area (plus some buffer zone).  This says that a
+    given config should use a single box to avoid inefficiency with
+    dvopsps.
+
+    It is critical that the loaded region overlap all detections from
+    all gpc1 exposures in the current seletion.
+
+    If the region chosen is too small, then detections on the edges
+    may be missed (this probably forces the buffer zone to be 0.5*gpc1
+    diameter -- but NOTE that the width is not adjusted by cos(DEC),
+    and this can be a big error).
+
+    Also, if the region is too small, then the dvopsps ingest
+    operation is performed for too many overlapping areas, and this is
+    inefficient.
+
+    However, if the region is too big, we may run into non-linear
+    speed effects on the mysql database (either for ingest or for
+    queries using those detections to generate the batches).  
+
+    In my tests, the entire SAS region was successfully ingested into
+    mysql with dvopsps with no problem (** after I changed the code to
+    have a certain key added to the table definition **).  
+
+    *** I recommend running some tests with the galactic center to see
+        the impact of large number of sources, and to see if / where
+        there is a point of diminishing returns.  Ideally, we should
+        use a config region which is large fairly large, at least
+        comparable to SAS or larger.  
+
+* timing information for non-parallel dvo:
+  * SAS ingest to mysql with dvopsps (78M det) : ~100 min (~12k det / sec)
+  * SAS region smfs (read, join, write) : ~110 sec (~1.2k det / sec)
+
+  * extrapolating these to the full 3pi : 
+    * ingest with dvopsps : 29 days
+    * make batch for smf  : 290 days
+
+  ** we need to use the multi-row ingest for the smfs to improve that
+     portion of the batch creation, but it is not certain how much
+     improvement we will see from that!
+
+  ** clearly, if we use a single thread, we will be too slow, even with the above fix.
+     * we can run more than one loader with the same parallel dvo db into multiple mysql dbs