Up: IPP at the ROE

Downloading Data with IPP

Ultimately, at the PS1 data distribution centre, there will be a data storage area allocated for us. This storage area will be populated (via symbolic links) with the data we require for our analysis. Bill Sweeney is in charge of the data distribution.

On eday, we will maintain a mirror of the data in the data storage area.

But first, let's just learn how to set up basic downloads of data products.

Much of this follows: ​http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/RunningTheDistributionClient which, except for the links, is pretty much dead on.

Setting up a local storage area and associated data base

Decide on a place to store the data

• set WORKDIR=/disk1/ps1/ert/IPP/PS1data
• mkdir $WORKDIR

Initialise the data base to hold the data metadata

• $SRCDIR/trunk/ippMonitor/scripts/dbadmin project localhost ipp PS1data
• pxadmin -dbname PS1data -create-mirror
• receivetool -dbname PS1data -definesource -source http://datastore.ipp.ifa.hawaii.edu -product ps1-200910-2 -workdir $WORKDIR -comment "PS1 Data (local mirror)"

Note: product ps1-200910-2 exists, but contains far more than we want. Use it as a test point from which to download, but once downloading is shown to be working, cancel the job.

Ensure you can SSH to your local client

The pantasks controller needs to be able to ssh login-lessly to all nodes capable of processing data, including the local node.

Ensure it works.

• ssh localhost
• ssh hostname

SSH'ing to the local host is particularly problematic on systems with shared home directories (like the IfA) where the localhost entry in ~/.ssh/known_hosts can correspond to any machine on the network. It may need to be edited out.

Start download using pantasks

Start pantasks then issue the following commands

• module pantasks.pro
• module receive.pro
• add.database PS1data
• controller host add localhost
• run

For interest's sake, outside of pantasks you can examine info about the data base with:

• receivetool -dbname PS1data -list

-- EricTittley - 2009-11-06