== Contents of IPP Distribution File Sets ==
Distribution bundles are built and placed in file sets on the outgoing data store by the
server side of the ipp distribution system.

Each destination site runs the client sideis programmed to query a particular data store
product for new filesets.

'''Product List Format'''
The data store product list is the list of file sets in the given product. The product listing
contains information about the contents of the fileset. Example


{{{
# filesetID                |time registered     |type  |target_id|stage|stage_id|exp_name   |
o4741g0234o.fake.2799.11.1 |2009-04-15T01:22:12Z|notset|5        |fake |2799    |o4741g0234o|
o4741g0236o.chip.14519.33.1|2009-04-15T03:02:11Z|notset|1        |chip |14519   |o4741g0236o|
o4741g0235o.chip.14518.32.1|2009-04-15T03:02:12Z|notset|1        |chip |14518   |o4741g0235o|
o4741g0234o.chip.14517.31.1|2009-04-15T03:02:12Z|notset|1        |chip |14517   |o4741g0234o|
o4741g0237o.chip.14520.34.1|2009-04-15T03:02:13Z|notset|1        |chip |14520   |o4741g0237o|
o4741g0240o.chip.14522.36.1|2009-04-15T03:02:14Z|notset|1        |chip |14522   |o4741g0240o|
o4741g0239o.chip.14524.38.1|2009-04-15T03:02:14Z|notset|1        |chip |14524   |o4741g0239o|
o4741g0241o.chip.14523.37.1|2009-04-15T03:02:15Z|notset|1        |chip |14523   |o4741g0241o|
o4741g0238o.chip.14521.35.1|2009-04-15T04:03:19Z|notset|1        |chip |14521   |o4741g0238o|

}}}


Note: the fileset type for IPP distribution bundles has not yet been defined. (notset in this example)

'''Fileset Names'''

Runs that are the result of a particular exposure


    $exp_name.$stage.$stage_id.$dist_id.$prod_id

$exp_name is rawExp.exp_name
$stage is raw, chip, camera, fake, warp, or diff
$dist_id is distRun.dist_id       (This table is internal to the server side of the distribution system)
$prod_id is rcDSProduct.prod_id   (This table is internal to the server side of the distribution system)


Examples




    o4741g0239o.raw.36625.27.1
    o4741g0239o.chip.11955.2.1


For stackRun distributions the fileset name is

 
   $stack.$stack_id.$dist_id.$prod_id


for example


    stack.9554.42.2


The name of detrend filesets is TBD.


'''Contents of the filese'''t

1) An mdc file containing the database information for the run with the name

    dbinfo.$stage.$stage_id.mdc

for example

    dbinfo.chip.11955.mdc

This file is the output from the -exportrun mode of the stage's ippTool (regtool, chiptool, etc).
If the run is exported with the -clean option when the run is in full state 
the state and data state are changed from 'full' to 'cleaned'

XXX: Should we be doing something with the nebulous paths in this file here? Here I assume this
is the client's job.

2) A set of gzipped tar files containing the files for a given component of the run.

For stages raw, chip, camera, and fake the name of the file for each component is

    basename($component_path_base).$class_id.tgz

and for stages warp, diff, and stack the filenames are

    basename($component_path_base).tgz

For example the files for chip XY01 for chipRun 11955 the path_base is

    neb://anyhost/md01_20081002/o4741g0239o.36625/o4741g0239o.36625.ch.11955

so the name of the tarfile is

    o4741g0239o.36625.ch.11955.XY05.tgz

The filenames in the tarfile are the basenames of the nebulous paths. For example

        % tar tf  o4741g0239o.36625.ch.11955.XY05.tgz
        ./
        ./o4741g0239o.36625.ch.11955.XY05.ppImage.mdc
        ./o4741g0239o.36625.ch.11955.XY05.b2.fits
        ./o4741g0239o.36625.ch.11955.XY05.stats
        ./o4741g0239o.36625.ch.11955.XY05.cmf
        ./o4741g0239o.36625.ch.11955.XY05.trace
        ./o4741g0239o.36625.ch.11955.XY05.log
        ./o4741g0239o.36625.ch.11955.XY05.mdl.fits
        ./o4741g0239o.36625.ch.11955.XY05.b1.fits


'''Notes on the process of importing the data into the mirror system
'''
The client system runs a task that queries the ipp distribution data store product for new
filesets

        dsproductls --uri $uri --last_fileset $last_fileset_processed

For each fileset to be downloaded
        dsget each file in the fileset to local temporary storage

        parse the dbinfo file to find the workdirs and path_base
        foreach component
            edit paths as necessary for the mirror installation
            foreach file insert into nebulous
        import the dbinfo into the database using xxxtool -importrun

ISSUES/Question?
---------------