Changes between Version 3 and Version 4 of ippToPsps_ScratchDatabase

Timestamp:

Apr 3, 2012, 9:17:08 AM (14 years ago)

Author:

rhenders

Comment:

--

ippToPsps_ScratchDatabase

@@ -10 +10 @@
 All instances of {{{loader.py}}} need use of a local 'scratch' database in which to import FITS tables, manipulate then export.
 
+
+== Setting up a loading machine ==
+
+Each instance of {{{loader.py}}} uses a 'scratch' MySQL database on {{{localhost}}} while processing. So, to do any loading, a local MySQL server should be up-and-running with a database created as the {{{ipp}}} user for use by {{{ippToPsps}}}. The default name for this is {{{ipptopsps_scratch}}}.
+
+Because {{{ippToPsps}}} uses this local database as a scratch database, there are large number of inserts and deletions, meaning large log files. For this reason, it is important to configure MySQL to store its data on a local partition, and not {{{/var}}} as is the default. For example, on {{{ipp005}}}:
+
+{{{
+/var/lib/mysql
+}}}
+
+was moved to:
+
+{{{
+/export/ipp005.0/mysql
+}}}
+
+In addition, it is prudent to decrease the default MySQL log-retention period from 90 days to 1 day. This is done is in
+
+{{{
+/etc/mysql/my.cnf 
+}}}
+
+on the line
+
+{{{
+expire_logs_days                                = 90
+}}}
+
+You will need (if it is not done already) to create an {{{ipp}}} user. Connect to the local MySQL server as root then to the following.
+
+{{{
+GRANT ALL PRIVILEGES ON *.* TO 'ipp'@'localhost' IDENTIFIED BY 'ipp' WITH GRANT OPTION;
+}}}
+
+Now disconnect then reconnect as {{{ipp}}} and create the scratch database.
+
+{{{
+CREATE DATABASE ipptopsps_scratch;
+}}}
+
+Finally, you need to setup the scratch database with the initialization tables and stored procedures necessary for processing. To do this, use the {{{setupScratchDb.py}}} program, like this:
+
+{{{
+./run.sh setupScratchDb.py someConfig nameOfScratchDb
+}}}
+
+The choice of config here is not important, so use any.
+
+At the time of writing, scratch databases are set up on the following IPP hosts:
+
+{{{
+ipp005
+ipp006
+ipp007
+ipp033
+ipp034
+ipp045
+}}}
+
+{{{ipp005}}} and {{{ipp045}}} in fact have ''three'' scratch databases each, so three clients can potentially be run on these machines. This is particularly useful on {{{ipp005}}} as this is where the DVO databases are currently stored, so clients running on the same machine have a much higher throughput. ({{{ipp045}}} used to host a backup 3pi DVO database.)
+
+
 == Stored procedure for psf likelihoods ==