== Introduction == 

We have had problems with addstar + merging in the past, and we need to fix them.  There are several failure modes which have caused problems in the past, or which I imagine could cause problems:

 * the addstar machine (ipp004 and/or ipp005) fills up
 * there are nfs related issues (this may cause problems with dvomerge?)
 * if the merge gets corrupted, we do not stop and this causes further problems
 * we do not back up the minidvodbs often enough
 * we do not back up the merge 
 * we merge the minidvodbs again, causing database chaos

== The way addstar works now == 

  === Queueing ===

We have several scripts and survey tasks to queue up the camera stage and the staticsky stage for running addstar.  New addRuns are queued and grouped by a minidvodb_group. This minidvodb_group is used to prevent multiple camRuns for the same exp_id from being queued.  The first one in is the first one used. 

  === Ingesting addstar runs === 

Initially, addRuns are queued with a minidvodb_group, but no minidvodb_name. When addstar is running, it looks for the current (active state) minidvodb for a given minidvodb_group, and ingests that add_id into that minidvodb.  Once completed, the addRun is updated to reflect where the cam/staticsky was ingested.

  === Minidvodb Tasks ===

 * Creation 
   * When certain criteria are met, a new minidvodb is created. Currently a new minidvodb is created when the current active minidvodb has more than 500 add_ids or is older than 1 day. When a new minidvodb is created, the old active minidvodb is retired (set to a state of "to_be_merged".

 * Verification and Merging
   * When the tasks detect a new minidvodb in a state of "to_be_merged", the following happens:
     * relphot  -- run relphot on the minidvodb -- fail if it doesn't relphot '''EAM : this step is not needed'''
     * addstar -resort -- resort the minidvodb -- fail if it doesn't resort
     * verify that it is resorted -- fail if it isn't
     * dvomerge the minidvodb into the big dvodb -- if there is a failure here, it unfortunately blindly goes onto the next minidvodb to merge, which causes corruption.  

  === MinidvodbCopy ===
    
There are a set of tasks and db tables to make backup copies of the minidvodbs.  More details can be seen here (http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/MinidvodbCopy).  MinidvodbCopy takes a minidvodb_group, host, and destination path to copy minidvodbs somewhere.  I haven't run into problems with it yet. It has checks for diskspace - it doesn't transfer unless there is enough room.

  === Backups of the big dvodb ===

These are done by hand, and very sporadically.  We often merge hundreds of minidvodbs before making a backup. We often lose track of where the backups go (or they get modified and thus can't be used).  This is a horribly inefficient way to do this, and most recoveries take forever because we have to merge back together all the minidvodbs (this takes a month).

== What I would like to change == 

  I would like to split up the minidvodb tasks and tables to something like this:

  The minidvodb tasks/tables/scripts

  *  minidvodbRun
  *  minidvodbProcessed
  *  minidvodbCopy 
 
  These would be mostly unchanged, with the exception that they are NOT merged into the big database.  Once a minidvodb is relphot'd/resorted/verified, it is now ready for merging and copying a backup to ippmanoa. '''EAM : if we can make the ATRC machines available via rsync to the consortium, then maybe we should put them there (though for now, ippmanoa is fine'''

  Mergedvodb tasks/tables/scripts 

  * mergedvodbRun
  * mergedvodbProcessed
  * mergedvodbCopy 

  These would be for tracking/queueing/backing up the big dvodbs.  Some ideas:

   * no queuing up a merge if there is a previously failed merge for a given big dvodb (this makes sense- allows us to fix it).  
   * mergedvodbCopy - my idea is after each successful merge, do an rsync backup of this.  I don't know how long this takes, but we can investigate this
   * we can easily create one-off smaller dvodbs if necessary this way, and track it and have it in the database (handy for roy and probably others)

  I would also like to run some cron job that detects full disk space on ipp004/ipp005, stops the addstars if necessary, and emails me (or the group) to notify us. 



== How this would address the addstar problems ==


   *  the addstar machine (ipp004 and/or ipp005) fills up  -- solved with a cron job
   *  there are nfs related issues (this may cause problems with dvomerge?) -- we will have a backup of the old dvodb to recover from
   *  if the merge gets corrupted, we do not stop and this causes further problems -- we will now stop (and we have a backup).
   *  we do not back up the minidvodbs often enough -- we will back up minidvodbs on ippb03
   *  we do not back up the merge -- we will back this up now
   *  we merge the minidvodbs again, causing database chaos -- this would be done as a new mergedvodbRun (instead of hacking at the minidvodb tables)

   * what if we have some crazy corruption (like the wrong data in the db?) -- I don't have any good way for this, except to keep a spare older *good* dvodb to rebuild from.