Back to ippToPsps main page

The ippToPsps scratch database

Overview

All instances of loader.py need use of a local 'scratch' database in which to import FITS tables, manipulate then export.

Setting up a loading machine

Each instance of loader.py uses a 'scratch' MySQL database on localhost while processing. So, to do any loading, a local MySQL server should be up-and-running with a database created as the ipp user for use by ippToPsps. The default name for this is ipptopsps_scratch.

Because ippToPsps uses this local database as a scratch database, there are large number of inserts and deletions, meaning large log files. For this reason, it is important to configure MySQL to store its data on a local partition, and not /var as is the default. For example, on ipp005:

/var/lib/mysql

was moved to:

/export/ipp005.0/mysql

In addition, it is prudent to decrease the default MySQL log-retention period from 90 days to 1 day. This is done is in

/etc/mysql/my.cnf 

on the line

expire_logs_days                                = 90

You will need (if it is not done already) to create an ipp user. Connect to the local MySQL server as root then to the following.

GRANT ALL PRIVILEGES ON *.* TO 'ipp'@'localhost' IDENTIFIED BY 'ipp' WITH GRANT OPTION;

Now disconnect then reconnect as ipp and create the scratch database.

CREATE DATABASE ipptopsps_scratch;

Finally, you need to setup the scratch database with the initialization tables and stored procedures necessary for processing. To do this, use the setupScratchDb.py program, like this:

./run.sh setupScratchDb.py someConfig nameOfScratchDb

The choice of config here is not important, so use any.

At the time of writing, scratch databases are set up on the following IPP hosts:

ipp005
ipp006
ipp007
ipp033
ipp034
ipp045

ipp005 and ipp045 in fact have three scratch databases each, so three clients can potentially be run on these machines. This is particularly useful on ipp005 as this is where the DVO databases are currently stored, so clients running on the same machine have a much higher throughput. (ipp045 used to host a backup 3pi DVO database.)

Stored procedure for psf likelihoods

It is necessary, hopefully temporarily, that ippToPsps calculates psf, extended and cosmic ray likelihoods for each detection. Fot this it has been necessary to implement a stored procedure within each scratch database in order to perform the calculations. This procedure is installed by a methos on the ScratchDb class, here.