Summary

Simtest: is a process for testing the IPP with simulated data. The simulated data is produced and injected into an SQL database by ipp_simulation_data.pl It can then be carried through the following ipp stages:

  • Registration
  • Detrend construction
  • Chip processing
  • Camera processing
  • Warp to skycell
  • Difference
  • Stack

Updated for ipp-2.4 / ipp-2.5 by S.Rodney


Annotated simtest walkthrough

Configuration:

  • Check ~/.ipprc (or $PS_SITE)
    • SITE and SYSTEM point at valid site.config and system.config files
    • PATH should contain your config directory
  • site.config: DATAPATH for SIMTEST, SIMWORK
    • the directory pointed to by SIMTEST is where your RAW simtest data will be created and stored. You can refer to it in command -line arguments as path://SIMTEST/
    • the SIMWORK directory is where the intermediate and final output files and all log files will be placed. Refer to it as path://SIMWORK/.
      • (The variable name used, e.g. SIMTEST or SIMWORK, is unimportant. Just be consistent in the program calls below)
    • DBSERVER, DBNAME, DBUSER, DBPASSWORD
      • these set the parameters used by programs to access the SQL database
  • Check ~/.ptolemyrc for:
    • CONFDIR
      • this is the directory containing your configuration files.

Initiating a database.

  • There are three ways:

1) If you don't care to use the ippMonitor webpage, you can create an empty database in mysql

> mysql mysql> CREATE DATABASE mydatabase; mysql> exit

then initiate the new database

> pxadmin -create -dbname mydatabase

2) If you are restarting an existing database project you can delete and rebuild an empty simtest database:

> pxadmin -recreate -dbname mydatabase

3) You can use dbadmin to create an empty database that will be recognized by ippMonitor (note: the dbadmin tool is a shell script that creates an SQL database and populates it with the tables needed by ippMonitor for user administration. The dbadmin script is probably not in your path, so find it in the ippMonitor source code directory and link to it or copy it somewhere. Call dbadmin with no options for syntax help.)

> cd /data/po02.0/ipp/ipp-2.1/ippMonitor > dbadmin
USAGE: dbadmin (options)

dbadmin init (dbserver) (dbuser) (dbpassword)

creates admin tables, basic db user -- requires root password

dbadmin client (dbserver) (dbuser) (dbpassword) (client)

add client name or regex

dbadmin user (dbserver) (dbuser) (username) (password)

create a new www user and password

dbadmin project (dbserver) (dbuser) (dbname)

create a new ipp project (database)

> dbadmin project alala ipp mydatabase

then initiate the new database

> pxadmin -create -dbname mydatabase

Make sim data

(and inject it into the database)

> ipp_simulation_data.pl --name test --dbname mydatabase --camera SIMTEST --telescope SimScope? --workdir path://SIMWORK/ --path path://SIMTEST/

wait until it finishes ...

Start pantasks

pantasks is the ipp parallel process manager for distributed computing across multiple nodes. start up pantasks from a second window

> pantasks

Welcome to pantasks - parallel task scheduler

load some pantasks commands

pantasks: module pantasks.pro

or, if you have a modified pantasks.pro file

pantasks: input /home/username/pantasks/pantasks.pro

Add some CPU 'controllers' to carry out the processing steps. Each controller you add will start an instance of the 'pclient' program, which receives processes from the pantasks 'pcontroller.' To add an individual controller host by hand:

pantasks: controller host add myhost

NOTE: You must have passphraseless ssh authentication set up from your pcontroller machine (the one running pantasks) to your pclient machines (the ones added with 'controller host add'). You may want to set up a module in your pantasks.pro file to quickly add a set of controllers with one command.

If you are using a database other than the one declared in your site.config file, use the DB variable. DB holds a list of databases. $DB:0 is the active database. The default database defined in your site.config file will only be used if $DB:n is 0.

pantasks: list DB -add mydatabase pantasks: echo $DB:0 mydatabase

The pantasks.pro module should contain a function called add.database to do this as well.

pantasks: add.database mydatabase

Registration

The newly injected simulated data is in the database, but we don't know what each image is yet, so we register all the new exposures in the database, getting their vital statistics and sorting them appropriately. In pantasks, start up the tasks list, turn off some switches and check the status

pantasks: module.tasks pantasks: all.off pantasks: register.on pantasks: status Task Status

AV Name Njobs Ngood Nfail Ntime Command -+ detrend.process.load 0 0 0 0 (dynamic) -+ detrend.process.run 0 0 0 0 (dynamic) -+ detrend.processexp.load 0 0 0 0 (dynamic) -+ detrend.processexp.run 0 0 0 0 (dynamic) -+ detrend.stack.load 0 0 0 0 (dynamic) -+ detrend.stack.run 0 0 0 0 (dynamic) -+ detrend.norm.load 0 0 0 0 (dynamic) -+ detrend.norm.run 0 0 0 0 (dynamic) -+ detrend.normexp.load 0 0 0 0 (dynamic) -+ detrend.normexp.run 0 0 0 0 (dynamic) -+ detrend.normstat.load 0 0 0 0 (dynamic) -+ detrend.normstat.run 0 0 0 0 (dynamic) -+ detrend.resid.load 0 0 0 0 (dynamic) -+ detrend.resid.run 0 0 0 0 (dynamic) -+ detrend.residexp.load 0 0 0 0 (dynamic) -+ detrend.residexp.run 0 0 0 0 (dynamic) -+ detrend.reject.load 0 0 0 0 (dynamic) -+ detrend.reject.run 0 0 0 0 (dynamic) -+ chip.imfile.load 0 0 0 0 (dynamic) -+ chip.imfile.run 0 0 0 0 (dynamic) -+ camera.exp.load 0 0 0 0 (dynamic) -+ camera.exp.run 0 0 0 0 (dynamic) -+ warp.exp.load 0 0 0 0 (dynamic) -+ warp.exp.run 0 0 0 0 (dynamic) -+ warp.skycell.load 0 0 0 0 (dynamic) -+ warp.skycell.run 0 0 0 0 (dynamic) -+ diff.skycell.load 0 0 0 0 (dynamic) -+ diff.skycell.run 0 0 0 0 (dynamic) -+ stack.skycell.load 0 0 0 0 (dynamic) -+ stack.skycell.run 0 0 0 0 (dynamic) ++ register.imfile.load 0 0 0 0 (dynamic) ++ register.imfile.run 0 0 0 0 (dynamic) ++ register.exp.load 0 0 0 0 (dynamic) ++ register.exp.run 0 0 0 0 (dynamic)

Start the registration step

pantasks: run

You can check the progress of the registration with status calls or on the web monitor: http://kiawe.ifa.hawaii.edu/ippMonitor

If you are watching the processes on your controller hosts (e.g. using 'top') then you should see pclient processes. The pantasks process itself is called pcontrol.

When all your images have finished registration you can halt the register.load processes, which are no longer needed. (or they can continue to run in the background.) You can turn off the registration switches.

pantasks: stop pantasks: register.off

NOTE: using 'halt' instead of 'stop' will stop the processes without harvesting jobs.

Detrend construction

Flip switches for the detrend step

pantasks: detrend.on

Back in a base shell, outside of pantasks, run the detrend tool. This creates a detRun table instance in the database, from which pantasks can get the info it needs to direct the detrend construction

> dettool -dbname mydatabase -definebyquery -det_type BIAS -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope? -filter NONE -select_exp_type BIAS

You should now be able to find an entry for this detrend run on the ippMonitor web page or with a MySQL query on the detRun table. When you see it there, start the pantasks scheduler in your pantasks window, window 2.

pantasks: run

When you see the bias in the ippMonitor under Detrend Frames or in MySQL under detFrames, then go back to window 1 for a sequence of dettools for DARKs, SHUTTERs, and FLATs. After running each dettool command from the command line, you should be able to find the run on the ippMonitor or in MySQL.

> dettool -dbname mydatabase -definebyquery -det_type DARK -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope? -filter NONE -select_exp_type DARK

Wait for the dark to finish before starting to make the shutter

> dettool -dbname mydatabase -definebyquery -det_type SHUTTER -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope? -filter r -select_exp_type FLAT -select_filter r

Wait for the shutter to finish before starting the flats (but both flats can be done at once)

> dettool -dbname mydatabase -definebyquery -det_type FLAT -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope? -filter r -select_exp_type FLAT -select_filter r

> dettool -dbname mydatabase -definebyquery -det_type FLAT -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope? -filter i -select_exp_type FLAT -select_filter i

When detrend creation is done, you can stop and switch off the detrend processing.

pantasks: stop pantasks: detrend.off

Chip and Camera

Start the chip and camera processing. This takes all the science images and does the detrending (dark sub, bias sub, shutter correction, flat fielding)

<pre>

pantasks: chip.on pantasks: camera.on pantasks: run

...

pantasks: stop

</pre>

Warp

Next we prepare for warping the science images into common skycells. i.e. putting them on a common tessellated tangent plane projection

If the skycells have not been defined, then the skycells program will make them. This creates a directory with catalogs of skycell definitions.

> skycells -D CATDIR ./skycells 8 -scale 0.2

no data in db ./skycells/Photcodes.dat directory ./skycells does not exist, creating...

IMPORTANT: be sure to point to this directory in your site.config file with, e.g.:

TESSELLATIONS METADATA

# Tessellation name with corresponding DVO catdir Skycells STR path://SIMTEST/skycells

END

Now we should be able to go back to the pantasks window and start warping:

<pre>

pantasks: warp.on pantasks: run

...

pantasks: stop

</pre>

The warped images will be stored in path://SIMWORK/tess_skycells/sky_skycell.<skycellnum>/ as '*.wrp[1-7].fits' with related mask ('.mask'), weight ('.wt'), and summary stats ('.stats'). The warping will automatically calculate the appropriate "<skycellnum>" for the given tessellation and location of image on the sky.

Difference

<pre>

pantasks: warp.off pantasks: diff.on pantasks: run

...

pantasks: stop

</pre>

These difference image will be stored in path://SIMWORK/tess_skycells/sky_skycell.0743078/ as '*.dif1.fits' with related mask ('.mask'), weight ('.wt'), and summary stats ('.stats').

Stack

<pre>

pantasks: diff.off pantasks: stack.on pantasks: run

...

pantasks: stop

</pre>

Lather, rinse, repeat.