Summary

Simtest: is a process for testing the IPP with simulated data. The simulated data is produced and injected into an SQL database by ipp_simulation_data.pl It can then be carried through the following ipp stages:

  • Registration
  • Detrend construction
  • Chip processing
  • Camera processing
  • Warp to skycell
  • Difference
  • Stack

Annotated simtest walkthrough

Configuration:

  • Check ~/.ipprc (or $PS_SITE) for:
    • DATAPATH for SIMTEST, SIMWORK
      • the directory pointed to by SIMTEST is where your RAW simtest data will be created and stored. You can refer to it in command -line arguments as path://SIMTEST/
      • the SIMWORK directory is where the intermediate and final output files and all log files will be placed. Refer to it as path://SIMWORK/.
      • (The variable name used, e.g. SIMTEST or SIMWORK, is unimportant. Just be consistent in the program calls below)
    • DBSERVER, DBNAME, DBUSER, DBPASSWORD
      • these set the parameters used by programs to access the SQL database
  • Check ~/.ptolemyrc for:
    • CONFDIR
      • this is the directory containing your configuration files.

Initiating a database.

  • There are three ways:

1) If you don't care to use the ippMonitor webpage, you can create an empty database in mysql

> mysql mysql> CREATE DATABASE mydatabase; mysql> exit

then initiate the new database

> pxadmin -create -dbname mydatabase

2) If you are restarting an existing database project you can delete and rebuild an empty simtest database:

> pxadmin -recreate -dbname mydatabase

3) You can use dbadmin to create an empty database that will be recognized by ippMonitor (note: the dbadmin tool is a shell script that creates an SQL database and populates it with the tables needed by ippMonitor for user administration. The dbadmin script is probably not in your path, so find it in the ippMonitor source code directory and link to it or copy it somewhere. Call dbadmin with no options for syntax help.)

> cd /data/po02.0/ipp/ipp-2.1/ippMonitor > dbadmin
USAGE: dbadmin (options)

dbadmin init (dbserver) (dbuser) (dbpassword)

creates admin tables, basic db user -- requires root password

dbadmin client (dbserver) (dbuser) (dbpassword) (client)

add client name or regex

dbadmin user (dbserver) (dbuser) (username) (password)

create a new www user and password

dbadmin project (dbserver) (dbuser) (dbname)

create a new ipp project (database)

> dbadmin project alala ipp mydatabase

then initiate the new database

> pxadmin -create -dbname mydatabase

Make sim data

(and inject it into the database)

> ipp_simulation_data.pl --name test --dbname mydatabase --camera SIMTEST --telescope SimScope? --workdir path://SIMWORK/ --path path://SIMTEST/

wait until it finishes ...

Start pantasks

pantasks is the ipp parallel process manager for distributed computing across multiple nodes. start up pantasks from a second window

> pantasks

Welcome to pantasks - parallel task scheduler

load some pantasks commands

pantasks: module pantasks.pro

or, if you have a modified pantasks.pro file

pantasks: input /home/username/pantasks/pantasks.pro

initialize controller connections, defined in your pantasks.pro file. (you must already have set up passphraseless ssh to any machine you wish to use as a controller)

pantasks: init.cluster.po starting controller connection Connected HostID: 0 HostID: 1 HostID: 2 HostID: 3 HostID: 4

To add an individual controller host by hand:

pantasks: controller host add myhost

If you are using a database other than the one declared in your .ipprc file, use the DB variable. DB holds a list of databases. $DB:0 is the active database. The default database defined in your .ipprc file will only be used if $DB:n is 0.

pantasks: list DB -add mydatabase pantasks: echo $DB:0 mydatabase

Registration

The newly injected simulated data is in the database, but we don't know what each image is yet, so we register all the new exposures in the database, getting their vital statistics and sorting them appropriately. In pantasks, start up the tasks list, turn off some switches and check the status

pantasks: module.tasks pantasks: chip.off pantasks: camera.off pantasks: warp.off pantasks: stack.off pantasks: detrend.off pantasks: diff.off pantasks: status
Scheduler is stopped Controller is stopped
Task Status

AV Name Njobs Ngood Nfail Ntime Command ++ register.imfile.load 0 0 0 0 (dynamic) ++ register.imfile.run 0 0 0 0 (dynamic) ++ register.exp.load 0 0 0 0 (dynamic) ++ register.exp.run 0 0 0 0 (dynamic) -+ chip.imfile.load 0 0 0 0 (dynamic) -+ chip.imfile.run 0 0 0 0 (dynamic) -+ camera.exp.load 0 0 0 0 (dynamic) -+ camera.exp.run 0 0 0 0 (dynamic) -+ dettool.process.load 0 0 0 0 (dynamic) -+ dettool.process.run 0 0 0 0 (dynamic) -+ dettool.processexp.load 0 0 0 0 (dynamic) -+ dettool.processexp.run 0 0 0 0 (dynamic) -+ dettool.stack.load 0 0 0 0 (dynamic) -+ dettool.stack.run 0 0 0 0 (dynamic) -+ dettool.norm.load 0 0 0 0 (dynamic) -+ dettool.norm.run 0 0 0 0 (dynamic) -+ dettool.normexp.load 0 0 0 0 (dynamic) -+ dettool.normexp.run 0 0 0 0 (dynamic) -+ dettool.normstat.load 0 0 0 0 (dynamic) -+ dettool.normstat.run 0 0 0 0 (dynamic) -+ dettool.resid.load 0 0 0 0 (dynamic) -+ dettool.resid.run 0 0 0 0 (dynamic) -+ dettool.residexp.load 0 0 0 0 (dynamic) -+ dettool.residexp.run 0 0 0 0 (dynamic) -+ dettool.reject.load 0 0 0 0 (dynamic) -+ dettool.reject.run 0 0 0 0 (dynamic) -+ warp.exp.load 0 0 0 0 (dynamic) -+ warp.exp.run 0 0 0 0 (dynamic) -+ warp.skycell.load 0 0 0 0 (dynamic) -+ warp.skycell.run 0 0 0 0 (dynamic) -+ diff.skycell.load 0 0 0 0 (dynamic) -+ diff.skycell.run 0 0 0 0 (dynamic) -+ stack.skycell.load 0 0 0 0 (dynamic) -+ stack.skycell.run 0 0 0 0 (dynamic) no defined jobs

Start the registration step

pantasks: run

You can check the progress of the registration with status calls or on the web monitor: http://kiawe.ifa.hawaii.edu/ippMonitor

When all your images have finished registration you can halt the register.load processes, which are no longer needed. (or they can continue to run in the background.)

pantasks: halt

Detrend construction

Flip switches for the detrend step

pantasks: register.off pantasks: detrend.on

Back in a base shell, outside of pantasks, run the detrend tool. This creates a detRun table instance in the database, from which pantasks can get the info it needs to direct the detrend construction

> dettool -definebyquery -det_type BIAS -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope? -filter NONE -select_exp_type BIAS

You should now be able to find an entry for this detrend run on the ippMonitor web page or with a MySQL query on the detRun table. When you see it there, start the pantasks scheduler in your pantasks window, window 2.

pantasks: run

When you see the bias in the ippMonitor under Detrend Frames or in MySQL under detFrames, then go back to window 1 for a sequence of dettools for DARKs, SHUTTERs, and FLATs. After running each dettool command from the command line, you should be able to find the run on the ippMonitor or in MySQL.

> dettool -definebyquery -det_type DARK -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope? -filter NONE -select_exp_type DARK

Wait for the dark to finish before starting to make the shutter

> dettool -definebyquery -det_type SHUTTER -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope? -filter r -select_exp_type FLAT -select_filter r

Wait for the shutter to finish before starting the flats (but both flats can be done at once)

> dettool -definebyquery -det_type FLAT -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope? -filter r -select_exp_type FLAT -select_filter r

> dettool -definebyquery -det_type FLAT -filelevel FPA -workdir path://SIMWORK/ -inst SIMTEST -telescope SimScope? -filter i -select_exp_type FLAT -select_filter i

Check on the status in pantasks

pantasks: status

Scheduler is running Controller is running

Task Status

AV Name Njobs Ngood Nfail Ntime Command -+ warp.exp.load 0 0 0 0 (dynamic) -+ warp.exp.run 0 0 0 0 (dynamic) -+ warp.skycell.load 0 0 0 0 (dynamic) -+ warp.skycell.run 0 0 0 0 (dynamic) -+ stack.skycell.load 0 0 0 0 (dynamic) -+ stack.skycell.run 0 0 0 0 (dynamic) -+ diff.skycell.load 0 0 0 0 (dynamic) -+ diff.skycell.run 0 0 0 0 (dynamic) -+ register.imfile.load 31 31 0 0 regtool -+ register.imfile.run 74 74 0 0 register_imfile.pl -+ register.exp.load 36 36 0 0 regtool -+ register.exp.run 74 74 0 0 register_exp.pl ++ dettool.process.load 314 314 0 0 dettool ++ dettool.process.run 75 75 0 0 detrend_process_imfile.pl ++ dettool.processexp.load 314 314 0 0 dettool ++ dettool.processexp.run 75 75 0 0 detrend_process_exp.pl ++ dettool.stack.load 314 314 0 0 dettool ++ dettool.stack.run 5 5 0 0 detrend_stack.pl ++ dettool.norm.load 314 314 0 0 dettool ++ dettool.norm.run 5 5 0 0 detrend_norm_apply.pl ++ dettool.normexp.load 314 314 0 0 dettool ++ dettool.normexp.run 5 5 0 0 detrend_norm_exp.pl ++ dettool.normstat.load 314 314 0 0 dettool ++ dettool.normstat.run 5 5 0 0 detrend_norm_calc.pl ++ dettool.resid.load 314 314 0 0 dettool ++ dettool.resid.run 75 75 0 0 detrend_resid.pl ++ dettool.residexp.load 314 314 0 0 dettool ++ dettool.residexp.run 75 75 0 0 detrend_reject_imfile.pl ++ dettool.reject.load 312 312 0 0 dettool ++ dettool.reject.run 5 5 0 0 detrend_reject_exp.pl

no defined jobs

Chip and Camera

When detrend creation is done, start the chip and camera processing. This takes all the science images and does the detrending (dark sub, bias sub, shutter correction, flat fielding)

pantasks: stop pantasks: detrend.off pantasks: chip.on pantasks: camera.on pantasks: run

...

pantasks: stop

Warp

Next we prepare for warping the science images into common skycells. i.e. putting them on a common tessellated tangent plane projection

If the skycells have not been defined, then the skycells program will make them. This creates a directory with catalogs of skycell definitions.

> skycells -D CATDIR /data/ipp002.0/price/skycells 8 -scale 0.2

no data in db /data/ipp002.0/price/skycells/Photcodes.dat directory /data/ipp002.0/price/skycells does not exist, creating...

IMPORTANT: be sure to point to this directory in your camera.config file with, e.g.:

TESSELLATIONS METADATA

# Tessellation name with corresponding DVO catdir Skycells STR path://SIMTEST/skycells

END

As of ipp-2.1 we still have to do these final stages more or less by hand. In later releases there will be a perl script to craft the sql queries that define a filter set for a given skycell, etc.

Here is an example sql query to get the cam_id's of all the r-band exposures:

> mysql -h alala -u ipp -pipp simtest mysql> select camProcessedExp.cam_id, rawExp.exp_tag from camProcessedExp JOIN chipProcessedExp USING(chip_id) JOIN rawExp USING(exp_tag) where rawExp.filter = 'r'; +--------+-------------+ | cam_id | exp_tag | +--------+-------------+ | 1 | test0060.61 | | 2 | test0063.64 | | 3 | test0062.63 | | 4 | test0064.65 | | 5 | test0065.66 | | 8 | test0061.62 | | 13 | test0066.67 | +--------+-------------+ 7 rows in set (0.00 sec)

Now we use warptool (by hand at the moment) to define the warp runs. There is a single run for each image being warped. Warptool assigns a sequential warp_id each time it is called, but as yet it knows nothing about the data being warped. In this example we are defining 7 warp runs that will later be assigned to the 7 r-band images listed above:

> warptool -definerun -mode warp -workdir path://SIMWORK/ warp_id S64 1 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:24:19Z

> warptool -definerun -mode warp -workdir path://SIMWORK/ warp_id S64 2 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:25:10Z

> warptool -definerun -mode warp -workdir path://SIMWORK/ warp_id S64 3 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:25:12Z

> warptool -definerun -mode warp -workdir path://SIMWORK/ warp_id S64 4 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:25:13Z

> warptool -definerun -mode warp -workdir path://SIMWORK/ warp_id S64 5 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:25:14Z

> warptool -definerun -mode warp -workdir path://SIMTEST/ warp_id S64 6 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:25:16Z

> warptool -definerun -mode warp -workdir path://SIMWORK/ warp_id S64 7 mode STR warp state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T00:25:17Z

Now we can use warptool to add the input exposure to each run. The input exposures are identified by their cam_id's, which we collected in mysql above.

> warptool -addinputexp -warp_id 1 -cam_id 1 > warptool -addinputexp -warp_id 2 -cam_id 2 > warptool -addinputexp -warp_id 3 -cam_id 3 > warptool -addinputexp -warp_id 4 -cam_id 4 > warptool -addinputexp -warp_id 5 -cam_id 5 > warptool -addinputexp -warp_id 6 -cam_id 8 > warptool -addinputexp -warp_id 7 -cam_id 13

Next, we set the state of each warp run from 'stop' (the default) to 'run' so that pantasks can process them.

> warptool -updaterun -warp_id 1 -state run > warptool -updaterun -warp_id 2 -state run > warptool -updaterun -warp_id 3 -state run > warptool -updaterun -warp_id 4 -state run > warptool -updaterun -warp_id 5 -state run > warptool -updaterun -warp_id 6 -state run > warptool -updaterun -warp_id 7 -state run

Finally, go back to your pantasks window and start it up:

pantasks: warp.on pantasks: run

...

pantasks: stop

Difference

The process is similar for difftool and stacktool. We run difftool once to define a run:

> difftool -definerun -workdir path://SIMTEST/ -skycell_id skycell.0743078 -tess_id skycells diff_id S64 1 state STR reg workdir STR path://SIMTEST/ dvodb STR NULL registered UTC 2007-05-03T02:12:53Z skycell_id STR skycell.0743078 tess_id STR skycells

Run difftool a second (and third) time to add the skycells we are differencing after running warptool, the warped image files are identified by the warp_id that was assigned by warptool above):

> difftool -addinputskyfile -diff_id 1 -warp_id 1 -skycell_id skycell.0743078 -tess_id skycells > /data/ipp002.0/price>difftool -addinputskyfile -diff_id 1 -warp_id 2 -skycell_id skycell.0743078 -tess_id skycells -template

And a final time to set the state to 'run':

> /data/ipp002.0/price>difftool -updaterun -diff_id 1 -state run

Then go back to pantasks and fire it up:

pantasks: warp.off pantasks: diff.on pantasks: run

...

pantasks: stop

Stack

Same deal for stacktool. When you define the run, give it a list of all the warp_id's that you want to stack and add.

> stacktool -definerun -skycell_id skycell.0743078 -tess_id skycells -workdir path://SIMTEST/ -warp_id 1 -warp_id 2 -warp_id 3 -warp_id 4 -warp_id 5 -warp_id 6 -warp_id 7

Maybe have to set the state here?

Then tell pantasks to go

pantasks: diff.off pantasks: stack.on pantasks: run

...

pantasks: stop

Lather, rinse, repeat.