[[PageOutline]]

[wiki:ippToPsps Back to ippToPsps main page]

= The {{{ippToPsps}}} scratch database =

== Overview ==

All instances of {{{loader.py}}} need use of a local 'scratch' database in which to import FITS tables, manipulate then export.

You can have more than one scratch database per machine. The naming convention for scratch databases on a single machine is:

{{{
ipptopsps_scratch
ipptopsps_scratch2
ipptopsps_scratch3
}}}

Instances of {{{loader.py}}} will cycle through all of the above until a 'free' scratch database is found. It is obviously important that two loaders do not use the same scratch database at the same time.

== Setting up a loading machine ==

Each instance of {{{loader.py}}} uses a 'scratch' MySQL database on {{{localhost}}} while processing. So, to do any loading, a local MySQL server should be up-and-running with a database created as the {{{ipp}}} user for use by {{{ippToPsps}}}. The default name for this is {{{ipptopsps_scratch}}}.

Because {{{ippToPsps}}} uses this local database as a scratch database, there are large number of inserts and deletions, meaning large log files. For this reason, it is important to configure MySQL to store its data on a local partition, and not {{{/var}}} as is the default. For example, on {{{ipp005}}}:

{{{
/var/lib/mysql
}}}

was moved to:

{{{
/export/ipp005.0/mysql
}}}

In addition, it is prudent to decrease the default MySQL log-retention period from 90 days to 1 day. This is done is in

{{{
/etc/mysql/my.cnf 
}}}

on the line

{{{
expire_logs_days                                = 90
}}}

You will need (if it is not done already) to create an {{{ipp}}} user. Connect to the local MySQL server as root then to the following.

{{{
GRANT ALL PRIVILEGES ON *.* TO 'ipp'@'localhost' IDENTIFIED BY 'ipp' WITH GRANT OPTION;
}}}

Now disconnect then reconnect as {{{ipp}}} and create the scratch database.

{{{
CREATE DATABASE ipptopsps_scratch;
}}}

Finally, you need to setup the scratch database with the initialization tables and stored procedures necessary for processing. To do this, use the {{{setupScratchDb.py}}} program, like this:

{{{
./run.sh setupScratchDb.py someConfig nameOfScratchDb
}}}

The choice of config here is not important, so use any.

At the time of writing, scratch databases are set up on the following IPP hosts:

{{{
ipp005
ipp006
ipp007
ipp033
ipp034
ipp045
}}}

{{{ipp005}}} and {{{ipp045}}} in fact have ''three'' scratch databases each, so three clients can potentially be run on these machines. This is particularly useful on {{{ipp005}}} as this is where the DVO databases are currently stored, so clients running on the same machine have a much higher throughput. ({{{ipp045}}} used to host a backup 3pi DVO database.)


== Stored procedure for psf likelihoods ==

It is necessary, hopefully temporarily, that {{{ippToPsps}}} calculates psf, extended and cosmic ray likelihoods for each detection. Fot this it has been necessary to implement a stored procedure within each scratch database in order to perform the calculations. This procedure is installed by a method on the !ScratchDb class, [browser:trunk/ippToPsps/jython/scratchdb.py here].