Changes between Version 165 and Version 166 of ippToPsps

Timestamp:

Oct 5, 2011, 1:59:37 PM (15 years ago)

Author:

rhenders

Comment:

--

ippToPsps

@@ -85 +85 @@
 = Using the software =
 
+{{{ippToPsps}}} contains numerous separate programs that work together to process and load IPP data (in the form of batches) to the datastore, monitor the progress of those batches and then perform the necessary cleanup of the various temporary files once those batches are safely merged into the PSPS database.
+
 == Setting up a loading machine ==
 
@@ -113 +115 @@
 }}}
 
-== Running {{{ippToPsps}}} ==
-
-{{{ippToPsps}}} is configured for use entirely by a config file (see above). So, running the software is simply a case of editing a config and passing it as an argument to the program, like this:
+== Loading ==
+
+Loading data means running the {{{load.py}}} program while passing it the relevant configuration file (see above). So, running the software is simply a case of editing a config and passing it as an argument to the program, like this:
 
 {{{
 cd trunk/ippToPsps/jython
-./run.sh ipptopsps.py someConfig.xml
+./run.sh load.py someConfig.xml
 }}}
 
 The {{{./run.sh}}} prefix above is necessary to invoke the correct Java virtual machine, while including the relevant jar files in the CLASSPATH (all included in the {{{trunk/ippToPsps/Jars}}} subdir)
 
-== Creating and publishing an 'IN' batch ==
+=== Creating and publishing an 'IN' batch ===
 
 The same program is used to create IN batches, but requires one extra argument, like this:
 
 {{{
-./run.sh ipptopsps.py someConfig.xml init
+./run.sh load.py someConfig.xml init
 }}}
 
 This will produce an IN batch, publish it, then stop (the IN-batch MySQL tables will also be updateded in the local scratch database).
 
-== Queuing ==
-
-When run, {{{ippToPsps}}} first queues up all the exposures or stacks that it needs to process, then works through them sequentially, doing detections first then stacks. The master list comes from the gpc1 {{{addRun}}} table, which lists all items currently merged into the DVO database we are using. From this list, {{{ippToPsps}}} subtracts items that have already been successfully loaded to the datastore already, and queues up the remainder.
+=== Queuing ===
+
+When run, {{{load.py}}} first queues up all the exposures or stacks that it needs to process, then works through them sequentially, doing detections first then stacks. The master list comes from the gpc1 {{{addRun}}} table, which lists all items currently merged into the DVO database we are using. From this list, {{{ippToPsps}}} subtracts items that have already been successfully loaded to the datastore already, and queues up the remainder.
+
+Because it is possible, and usual, to run multiple versions of {{{load.py}}} in order to speed up loading time, the methods to queue and begin processing a new batch form a critical section. This simply means that the 'batch' table in the {{{ippToPsps}}} database is locked by a client looking for a new item to process, then released afterwards. 
 
 An important factor in the queuing of items is the current ''epoch'', which is set in the config file.
 
-== Epochs ==
-
-Each config file requires an entry in the {{{options/epoch}}} field.  An ''epoch'' in the context of {{{ippToPsps}}} is the date that we count as ''the beginning''. If we loaded IPP data to PSPS once and only once, this would not be necessary: we would queue up all available exposures and stacks and simply load them. However, the early stages of the project have required multiple re-loads of data while the IPP perfects the science, necessitating that the same exposures and stacks are loaded more than once. Because {{{ippToPsps}}} queues available items by taking into account those that have already been published to PSPS, we have to give it an epoch date from which to accurately determine those exposure that have been loaded or not. Generally speaking, the epoch is reset every time we delete the main PSPS database.
-
-== Test mode ==
+=== Epochs ===
+
+Each config file requires an entry in the {{{options/epoch}}} field.  An ''epoch'' in the context of {{{ippToPsps}}} is the date that we count as ''the beginning''. If we loaded IPP data to PSPS once and only once, this would not be necessary: we would queue up all available exposures and stacks and simply load them. However, the early stages of the project have required multiple re-loads of data while the IPP perfects the science, necessitating that the same exposures and stacks are loaded more than once. Because {{{load.py}}} queues available items by taking into account those that have already been published to PSPS, we have to give it an epoch date from which to accurately determine those exposure that have been loaded or not. Generally speaking, the epoch is reset every time we delete the main PSPS database.
+
+=== Test mode ===
 
 One field in the config, under the ''options'' section, is ''testMode''. This is a boolean, i.e. can be either 1 or 0. When switched on, each batch type will be run in it's own test mode. For detections, this means: