
+ The data will be copied in batches which represent a single entry in
  a single stage.  This means, eg, a single chipRun, a single rawExp, or
  a single warpRun, etc.

+ We need hand-shaking between the source and target sites.

+ Data Store:

  product: distribution (restricted access)
    fileset: chipRun chip_id
      files....

+ remote copy system includes these stages:

  1) rc_getstages

     query the source datastore for stages to fetch
     + this may use the last-seen entry to query for only the new
       entries

     insert the new stages into rcDownloadStage

  2) rc_copystage

     for a single rcDownloadStage entry, download the complete fileset 

     * we can do this with the wget command to ensure the full set is
       downloaded, and allow for retries

     * the result files should be placed in a local temporary
       location.

     insert the downloaded data into the local database:

     * check the database version in the manifest
       - if it is not equal to the current version, raise an error

     * create nebulous entries for each of the files
       - for each file in the manifest, create the single nebulous
         entry and copy the file to the corresponding instance.
       - allow the remote site to replicate entries at this time
     
     * execute the SQL db update code 

     * update the rcDownloadStage entry

     * post an entry on our datastore to acknowledge success or
       failure

     * on success insert the stage info into our upload tables.

----------  

rc_manifest:

STAGE: chip
ID: nnnnn
DB_VERSION : version number of source database 
DB_DATA : filename.sql
NUMBER OF FILES : NNN
# neb name                              ==>  bundle name
neb://path/name/o4541g0012o.ota11.fits  ==>  o4541g0012o.ota11.fits
neb://path/name/o4541g0012o.ota12.fits  ==>  o4541g0012o.ota12.fits
neb://path/name/o4541g0012o.ota13.fits  ==>  o4541g0012o.ota13.fits

-----------

database tables:

rcDownloadStage:
  id
  ds_entry
  stage (needed here?)
  fileset uri
  state
  fault

rcDistributeStage
  id
  stage
  fileset uri
  destination
  state
  
------------

once the rcDistributeStage is built, we need a task to monitor for
successful download by the destination system.  Here are the tasks
needed:

1) build a book of remote destinations (static book loaded from
   config?)

2) for each remote destination, examine the remote datastore
   (PRODUCT name needed for download results)

   * we need to track the last seen entry 
   ** we need this information for several systems: summit copy,
      remote copy, distribute

   for each new entry, update rcDistributeStage with new state
   (success / failure)

------------

build the rcDistributeStage table entries:

new entries to be distributed should be added after the data is (a)
identified to be distributed, and (b) a distribution bundle is
generated.

* Does this replicate PStamp server tables / interfaces?

Steps to do (run for each of the distributed stages)

1) identify the entries in the stage to be distributed:
   * entry meets the needed condition (exists, is 'full', is magicked
   -- this last may be optional depending on the data source).  

   * entry is not in the rcDistributeStage

2) do work 
