Notes on the IPP Postage Stamp Server

Overview
--------

This note describes the installation and operation of the IPP Postage Stamp Server.
We assume that IPP source for examination and that the user has the standard
ipp configuration environment set up.

This component of the IPP system is not intended for general distribution to ipp users
and does not have streamlined installation procedures at this time.

The PSS is a system which responds to requests to retrieve images from an IPP Image Server.
The images may be either complete images or subsets of images known as "Postage Stamps".

The internal mechanisms of the PSS are also used to service MOPS detectabiltiy
query requests.

Given this more general utility it should perhaps be called the "IPP Request Server".


Data Store
----------
The PSS uses the Data Store mechanism to find incoming requests and to 
provide results to the end user.

A DataStore is simply a web server with URLS that supports an interface
that allows collections of data called File Sets to be located and accessed.

A Data Store is organized into 3 levels. These are described in the following sections.

Data Store Root - a list of Products
-------------------------------------

Assume that a datastore is located on the server WEBSERVERHOST.
An HTTP get to the url http://WEBSERVERHOST/ds/index.txt would give a list of the
products in the Data Store.

    # productID  |Most recent |Time registered     |type     |Description                     |
    pstampresults|web204      |2008-10-16T00:35:16Z|psresults|Postage Stamp Results           |
    pstamprequests|TEST0012    |2008-08-29T03:26:40Z|psrequest|Postage Stamp Requests          |
    mops-pstamp-results|MOPSSTAMPREQ20080813T214918Z_00|2008-08-13T21:49:42Z|psresults|Postage Stamp Results for MOPS  |

Each line is divided into fields separated by a 'pipe' character | .

Data Store Product - a list of filesets
---------------------------------------

Similarly the url http://WEBSERVERHOST/ds/pstampresults/index.txt gives the list of filesets
in the product pstampresults:

    # filesetID|time registered     |type     |
    TEST0011   |2008-08-29T03:05:03Z|PSRESULTS|
    TEST0012   |2008-08-29T03:31:36Z|PSRESULTS|
    web190     |2008-10-15T02:29:11Z|PSRESULTS|
    web191     |2008-10-15T02:32:16Z|PSRESULTS|
    web192     |2008-10-15T02:38:50Z|PSRESULTS|
    web193     |2008-10-15T03:08:57Z|PSRESULTS|
    web194     |2008-10-15T23:32:11Z|PSRESULTS|
    web195     |2008-10-15T23:35:39Z|PSRESULTS|
    web196     |2008-10-15T23:52:40Z|PSRESULTS|
    web197     |2008-10-15T23:59:05Z|PSRESULTS|
    web198     |2008-10-16T00:04:17Z|PSRESULTS|
    web200     |2008-10-16T00:17:46Z|PSRESULTS|
    web204     |2008-10-16T00:35:16Z|PSRESULTS|

Note that the fileset ID for the latest fileset (web204) in the list appears in the root listing in the last
section.

The product list protocol accepts a query string which can be used to limit the listing to all filesets
registered after a given fileset.

For example, http://WEBSERVERHOST/ds/pstampresults/index.txt?web198 yields

    # filesetID|time registered     |type     |
    web200     |2008-10-16T00:17:46Z|PSRESULTS|
    web204     |2008-10-16T00:35:16Z|PSRESULTS|


File Set - a list of files
--------------------------
http://ippds.somwhere.net/ds/pstampresults/web205/index.txt

    # fileID          |bytes   |md5sum                          |type        |
    results.fits      |28800   |4e2048479e551844e31102a301d50740|table       |
    # fileID          |bytes   |md5sum                          |type        |chipname|
    1_1_o4748g0259o.ota01.fits|19840320|19a6233ab966cf3e13b7da8271dead67|chip        |XY01    |
    1_2_o4748g0259o.ota02.fits|20416320|625bf8198dd7aef7c892af03584ee0ed|chip        |XY02    |

    .... followed by 58 other files in this GPC1 exposure


The ipp includes a number of tools for accessing a data store. An introduction to these tools may
be seen by using the program perldoc.

    dsrootls
    dsproductls
    dsfilesetls
    dsget


Installing the IPP Data Store Server Implementation
---------------------------------------------------

The data store implementation consists of a set of cgi scripts which are
invoked by an apache web server. The indexes for the Data Store are stored in a mysql
database. The cgi scripts use DBI perl module to query the database. No database updates
are performed by the Data Store cgi scripts.

The files for the Data Store Server located in the IPP source tree in the directory
DataStoreServer (this directory may not be available in the distributed ipp
tarballs).

Mysql data base configuration
-----------------------------

To set up the database go to the DataStoreServer/scripts and start up mysql.

mysql> create database mydatabase;
mysql> use mydatabase;
mysql> source scripts/tabledefs.sql;

Note: if you already have a current IPP database that you would like to use
for the Postage Stamp Request tables, when skip you can skip the create step
and use that database.


Web Server Configuration
------------------------
The apache configuration needs to have the following entries added to
the appropriate configuration file. (Different apache distributions have
different locations for these files so it's up to the installer to
figure out where these configuration lines should go).

       <Directory "/DATASTOREDIR/dsroot">

          DirectoryIndex /ds-cgi/dsindex.cgi

          #Options None

          # turn on server side includes
          Options +Includes

          # files will be processed for server side includes if they have the
          # executable bit set
          XBitHack on

          AddType text/plain .txt
          AddOutputFilter INCLUDES .txt

          AllowOverride None

          Order allow,deny
          Allow from all
        </Directory>

        #
        # Location of the cgi scripts
        #

        <Directory "/DATASTOREDIR/ds-cgi">
          Options ExecCGI


          Order allow,deny
          Allow from all
        </Directory>

        Alias       /ds     /DATASTOREDIR/dsroot/
        ScriptAlias /ds-cgi /DATASTOREDIR/ds-cgi/

Here we have assumed that the data store cgi scripts will be installed in
/DATASTOREDIR/ds-cgi and the data store data files will be stored in
/DATASTOREDIR/dsroot.

To install the cgi scripts
    cd your_ipp_src_directory
    cd DataStoreServer
    scripts/installscripts --dsroot /DATASTOREDIR/dsroot --cgidir /DATASTOREDIR/ds-cgi

The file /DATASTOREDIR/ds-cgi/dsshellconfig is a shell script that sets
several environment variables that are required by the scripts.

This script should be editied to customize the values for the installation.
(This file will not be overwritten by subsequent invocations of
installscripts).

There are some scripts that the Postage Stamp Server uses that need to be
installed into the IPP build directory. These are not included in the current IPP
build. To build them invoke the following commands from the DataStoreServer
directory.
    psautogen
    make install


Testing the DataStore
---------------------
At this point an empty data store should be up and running. This can be tested
by going to the following url in a web browser
    http://WEBSERVERHOST/ds/index.txt

If things are working properlly you should see the following output

# productID  |Most recent |Time registered     |type     |Description                     |

If an errror occurs check the web server error log and begin debugging.

The following command may be used to create a test product in the data store

dsprodtool --dbname mydatabase --dsroot   /DATASTOREDIR/dsroot/ \
     --add testproduct --type psresults --description 'test data store product'

Now if you point to http://WEBSERVERHOST/ds/index.txt with a web browser you
should see 

# productID  |Most recent |Time registered     |type     |Description                     |
testproduct  |none        |2008-10-16T20:48:23Z|psresults|test data store product         |

(of course your timestamp will be different).

As a final test we will add a fileset to the data store.

echo this is a file > testfile.txt

echo 'testfile.txt|||text|' | dsreg --dbname mydatabase --dsroot /DATASTOREDIR/dsroot/ \
    --product testproduct --add testfileset --type PSRESULTS --list - --copy --datapath .

-list - tells dsreg to get the list of files for the fileset from stdin.
(Usually a file is used).  The input to the command gives the name of the file to be added to the
fileset, blank entries for size and md5sum, and the file type 'text'. 

Now if you go to http://WEBSERVERHOST/ds/testfileset/index.txt you should see
the fileset.

If you leave off the index.txt in the urls you will get a more web browser
friendly html pages with links.
For example
    http://WEBSERVERHOST/ds

To delete the test product
    dsprodtool --dbname mydatabase --dsroot /DATASTOREDIR/dsroot/ --del psresults --rm

Set up the product for the postage stamp results with

    dsprodtool --dbname mydatabase --dsroot /DATASTOREDIR/dsroot/ \
         --add pstampresults --type psresults --description 'Postage Stamp results'

and a product for requests

    dsprodtool --dbname mydatabase --dsroot /DATASTOREDIR/dsroot/ \
         --add pstamprequests --type psrequest --description 'Postage Stamp requests'

Postage Stamp Server Set up
---------------------------

Now we are ready to set up the Postage Stamp Server.

The programs that make up the PSS are controlled by a set of pantasks tasks. 
There are some configuration variables that must be added to the site.config file.

    # root directory of the data store
    # this must match the directory used above in the Data Store setup
    DATA_STORE_ROOT STR /DATASTOREDIR/dsroot

    # default data store product used for results 
    # (used by the web interface)
    PSTAMP_DATA_STORE_PRODUCT STR pstampresults

    # This directory is used for downloaded request files and other purposes
    # it can go anywhere. The permissions on the directory must be such that
    # that the web server program has permission to write there.
    PSTAMP_WORKDIR STR /SOMEWHERE/pstamp-work

PSS projects
------------
Each Postage stamp request selects images from a particular 'project'. 
Each project has a row in the table pstampProject has the following columns

    column      example
    -------------------
    name        simtest
    state       enabled
    dbname      ps_simtest
    dvodb       currently not used
    camera      SIMTEST
    telescope   SimScope
    need_magic  0

The column dbname containes the name of the database that is used to look up images. To
create a project use pstamptool. In this case the ps_simtest is a database
refererencing images made by a simtest run.

pstamptool -dbname mydatabase -addproject -name foo -imagedb ps_simtest -inst SIMTEST -telescope SimScope


Postage Stamp Request Tables
----------------------------

Postage Stamp requests are submitted in the form of a FITS binary table. The
format of this table is described in a document which can be found by going
to the page 

    PostageStampServer

on the IPP wiki.


The tables for the PSS are part of the regular IPP database
tables. The Data Store implementation has been designed to not require the IPP for it's
operation. 
So the Data Store's tables are not created as part of the ipp database setup.

However, the current PostageStamp Server implementation assumes that the data
store and postage stamp tables are in the same mysql database (this will be
fixes soon).

As will be explained below, the request database does not necessarily have to be the
same as the image database. 

XXXXXXXXXXXXXXXXXXXXXX The following sections are to be completed
Postage Stamp Server
--------------------
pantasks tasks
Database
DataStore for input and output
Web interface (limited functionality prototype only)


Request files
Link to Spec



Examples
--------

postage stamp for diffSkyfile and chipProcessedImfile at given coordinates for
a given diff.

get_image of a full gpc1 exposure




