Up: [wiki:IPPatTheROE IPP at the ROE]

= Downloading Data with IPP =

Ultimately, at the PS1 data distribution centre, there will be a data storage area allocated for us. This storage area will be populated (via symbolic links) with the data we require for our analysis. Bill Sweeney is in charge of the data distribution.

UPDATE: Our datastore is http://datastore.ipp.ifa.hawaii.edu/roe-ac-uk-md
The rest of the wiki has been updated accordingly.
-- EricTittley - 2009-11-17

On eday, we will maintain a mirror of the data in the data storage area.

But first, let's just learn how to set up basic downloads of data products.

Much of this follows: http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/RunningTheDistributionClient which, except for the links, is pretty much dead on.

== Setting up a local storage area and associated data base ==

Decide on a place to store the data
    * `set WORKDIR=/disk1/ps1/ert/IPP/PS1data`
    * `mkdir $WORKDIR`

Initialise the data base to hold the data metadata
    * `$SRCDIR/trunk/ippMonitor/scripts/dbadmin project localhost ipp PS1data`
    * `pxadmin -dbname PS1data -create-mirror`
    * `receivetool -dbname PS1data -definesource -source http://datastore.ipp.ifa.hawaii.edu -product roe-ac-uk-md -workdir $WORKDIR -comment "PS1 Data (local mirror)"`

== Ensure you can SSH to your local client ==

The `pantasks` controller needs to be able to `ssh` login-lessly to all nodes capable of processing data, ''including the local node''.

Ensure it works.
    * `ssh localhost`
    * `ssh hostname`

SSH'ing to the local host is particularly problematic on systems with shared home directories (like the IfA) where the `localhost` entry in `~/.ssh/known_hosts` can correspond to any machine on the network. It may need to be edited out.

== Start download using `pantasks` ==

Start `pantasks` then issue the following commands
    * `module pantasks.pro`
    * `module receive.pro`
    * `add.database PS1data`
    * `controller host add localhost`
    * `run`

For interest's sake, outside of `pantasks` you can examine info about the data base with:
    * `receivetool -dbname PS1data -list`

== What you get ==

You should get the directory
 * `$WORKDIR/gpc1/roe-ac-uk-md`
  * `gpc1` == Giga Pixel Camera 1
  * `ps1-200910-2` is the ''project''
  * The data is stored in the project directory.
 * `$WORKDIR/20091112`
  * `20091112` is the date you run the download on, so will vary

-- EricTittley - 2009-11-17