Changes between Initial Version and Version 1 of IppSimtest

Timestamp:

Nov 6, 2009, 12:20:10 PM (17 years ago)

Author:

dravg.uk.edinburgh

Comment:

First version, copied from the ROE's internal tWiki

IppSimtest

@@ +1 @@
+Up: [wiki:IPPatTheROE IPP at the ROE]
+
+= Running Simtest =
+
+As you can probably guess, Simtest is a suite of tests that, if nothing else, ensures you've got a workable IPP installed.
+
+Requisite to running Simtest, you will need to have built IPP (IppInstallation) and initialised the database (IppInitialisingDatabase).
+
+== Configuration files ==
+
+First, I will describe intallation/modification to configuration files. Probably I should have this step somewhere else, like in IPP installation.
+
+   1. `cp $LOCAL/ipp/default.lin64/share/ippconfig/ipprc.config ~/.ipprc`
+   2. Edit `~/.ipprc` as you see fit. Look particularly at the `PATH`.
+   3. `cp /home/ert/.ptolemyrc ~/`
+   4. Edit `~/.ptolemyrc` as you see fit. Look particularly at `CONFDIR`.
+   5. `mkdir ~/ippconfig`
+   6. `cp /home/ert/ippconfig/site.config ~/ippconfig/`
+   7. `Edit ~/ippconfig/site.config` appropriately. Take particular attention to `DATAPATH`, `DBSERVER`, `DBNAME`, `DBUSER`, `DBPASSWORD`, the last four of which you came up with when you initialised the database (IppInitialisingDatabase). Also check `PSASTRO.CATDIRS`. The directories don't need to exist, but they must be createable by you.
+
+The first step, installation of a `.ipprc` file, seems wholly unnecessary (since the information already exists in the and is available to the code in a predictable location). Perhaps a critial review of the configuration settings would make it unnecessary. But there you go.
+
+Through DVO, IPP has a provenance in Ohana, a collection of tools written for CFHT. Ohana tools look for user configuration in `~/.ptolemyrc`. There is no master copy. It seems to just get handed down from user to user. Take mine.
+
+`site.config` requires more attention. Anything under `"## Stuff from Chris Waters ##"` can actually be ignored for Simtest. But it might be needed in the future.
+
+== Preparing Simtest files ==
+
+Simtest isn't quite in sync with changes to IPP, so it needs a bit of tweaking.
+
+    * Edit `$LOCAL/ipp/default.lin64/share/ippconfig/simtest/psastro.config`
+    * Everywhere it says `*_SYNTH` change to `SYNTH.*` (eg `r_SYNTH` => `SYNTH.r`)
+
+== Ensure you can SSH to your local client ==
+
+The `pantasks` controller needs to be able to ssh to all nodes capable of processing data, including the local node.
+
+Ensure it works.
+
+    * `ssh localhost`
+    * `ssh hostname`
+
+SSH'ing to the local host is particularly problematic on systems with shared home directories (like the IfA) where the `localhost` entry in `~/.ssh/known_hosts` can correspond to any machine on the network. It may need to be edited out.
+
+== Running Simtest ==
+
+Simtest is run in the `pantasks` environment, sorta like running IDL `.pro` scripts in IDL.
+
+    * `mkdir /wherever/you/want/to/work/simtest`
+    * `cd /wherever/you/want/to/work/simtest`
+    * `psconfig default`
+    * `pantasks`
+    * pantasks: `module simtest.pro`
+    * pantasks: `simtest testdb localhost new`
+    * Answer YES to starting from scratch
+    * Supply `ipp`'s password: `aloha`
+    * Wait for the fireworks....
+
+`pantasks` will take a couple of minutes to generate the data on which to process. Then it will return to the `pantasks` prompt while the tasks are run in the background for the next couple of hours.
+
+    * Typing status will give the state of all the processes
+
+== Parallelisation ==
+
+In IPP, Parallelisation can either mean speeding up a single code by use of threading, or running multiple tasks at the same time on the same hardware (trivial paralellisation).
+
+Let's start with trivial parallelisation.
+
+In `pantasks`, the queue controller can be accessed via the controller command. To add a machine to the list of machines to which tasks can be assigned:
+
+    * pantasks: `controller host add` ''HostToAdd''
+
+If the host is multi-core, then you can add the host many times (for the rest of the examples, ''HostToAdd'' is `eday`):
+
+    * pantasks: `controller host add eday`
+    * pantasks: `controller host add eday`
+    * ...
+
+However, if the tasks are I/O intensive, then having all cores running separate tasks might not be optimal. You may just want to get the code to run in parallel with threads
+
+To let the controller know that the host can handle multiple threads:
+
+    * pantasks: `controller host add eday -threads 4`
+
+Prior to running `simtest` (which calls `controller` on its own), the number of threads to run is set by:
+
+    * pantasks: `$SIMTEST_THREADS = 4`
+
+Hosts can be added before or after the jobs are submitted.
+
+There is also a `$PARALLEL` variable, but I think it is just a boolean (0 or 1)
+
+Paul Price recommends for 8-way systems to set the number of threads to something like 4 and then let the node run 3 separate processes. Results may vary.
+
+-- EricTittley - 2009-11-06