Changes between Initial Version and Version 1 of IppDownloadData

Timestamp:

Nov 6, 2009, 2:17:06 PM (17 years ago)

Author:

dravg.uk.edinburgh

Comment:

First version, copied from the ROE's internal tWiki

IppDownloadData

@@ +1 @@
+Up: [wiki:IPPatTheROE IPP at the ROE]
+
+= Downloading Data with IPP =
+
+Ultimately, at the PS1 data distribution centre, there will be a data storage area allocated for us. This storage area will be populated (via symbolic links) with the data we require for our analysis. Bill Sweeney is in charge of the data distribution.
+
+On eday, we will maintain a mirror of the data in the data storage area.
+
+But first, let's just learn how to set up basic downloads of data products.
+
+Much of this follows: http://svn.pan-starrs.ifa.hawaii.edu/trac/ipp/wiki/RunningTheDistributionClient which, except for the links, is pretty much dead on.
+
+== Setting up a local storage area and associated data base ==
+
+Decide on a place to store the data
+    * `set WORKDIR=/disk1/ps1/ert/IPP/PS1data`
+    * `mkdir $WORKDIR`
+
+Initialise the data base to hold the data metadata
+    * `$SRCDIR/trunk/ippMonitor/scripts/dbadmin project localhost ipp PS1data`
+    * `pxadmin -dbname PS1data -create-mirror`
+    * `receivetool -dbname PS1data -definesource -source http://datastore.ipp.ifa.hawaii.edu -product ps1-200910-2 -workdir $WORKDIR -comment "PS1 Data (local mirror)"`
+
+Note: product `ps1-200910-2` exists, but contains far more than we want.  Use it as a test point from which to download, but once downloading is shown to be working, cancel the job.
+
+== Ensure you can SSH to your local client ==
+
+The `pantasks` controller needs to be able to `ssh` login-lessly to all nodes capable of processing data, ''including the local node''.
+
+Ensure it works.
+    * `ssh localhost`
+    * `ssh hostname`
+
+SSH'ing to the local host is particularly problematic on systems with shared home directories (like the IfA) where the `localhost` entry in `~/.ssh/known_hosts` can correspond to any machine on the network. It may need to be edited out.
+
+== Start download using `pantasks` ==
+
+Start `pantasks` then issue the following commands
+    * `module pantasks.pro`
+    * `module receive.pro`
+    * `add.database PS1data`
+    * `controller host add localhost`
+    * `run`
+
+For interest's sake, outside of `pantasks` you can examine info about the data base with:
+    * `receivetool -dbname PS1data -list`
+
+-- EricTittley - 2009-11-06