Changes between Version 321 and Version 322 of ippToPsps

Timestamp:

Apr 10, 2012, 1:17:13 PM (14 years ago)

Author:

rhenders

Comment:

--

ippToPsps

@@ -201 +201 @@
 == loader.py ==
 
-{{{loader.py}}} is the program that actually processes and publishes data to PSPS. You can run one or more instances of {{{loader.py}}} as the code has been designed so taht they will run concurrently without causing any problems. Note, however, that clients on machines where the DVO database is not on a local disk will run more slowly due to the increased burden of accessing DVO over the network. Also, {{{loader.py}}} will only run on a host with a scratch database setup. Instructions for this are [wiki:ippToPsps#Settingupaloadingmachine here]. '''Note''' that {{{loader.py}}} will look for an available scratch database that is not in use by anyone else, if none are found then it will quit.
-
-The first thing a {{{loader.py}}} instance does is look to the queue tables in the {{{ippToPsps}}} database for a list of items to process. It first attempt to secure a 'stripe' of RA (from the [wiki:ippToPsps_Database#Thestripetable stripe table]) that is not being processed by any other {{{loader.py}}} clients. If all available stripes have already been reserved, then it chooses the stripe with the largest number of pending items.
+{{{loader.py}}} is the program that actually processes and publishes data to PSPS. You can run one or more instances of {{{loader.py}}} as the code has been designed so that they will run concurrently without causing any problems. Note, however, that clients on machines where the DVO database is not on a local disk will run more slowly due to the increased burden of accessing DVO over the network. Also, {{{loader.py}}} will only run on a host with a scratch database setup. Instructions for this are [wiki:ippToPsps#Settingupaloadingmachine here]. '''Note''' that {{{loader.py}}} will look for an available scratch database that is not in use by anyone else, if none are found then it will quit. Example usage is:
+
+{{{
+./run.sh loader.py new3pi
+}}}
+
+The first thing a {{{loader.py}}} instance does is look to the queue tables in the {{{ippToPsps}}} database for a list of items to process. It first attempts to secure a 'stripe' of RA (from the [wiki:ippToPsps_Database#Thestripetable stripe table]) that is not being processed by any other {{{loader.py}}} clients. If all available stripes have already been reserved, then it chooses the stripe with the largest number of pending items.
 
 With a stripe chosen, {{{loader.py}}} gathers a list of box_ids (each stripe is sub-divided into boxes). It works its way through each box, processing all pending items. Batches are processed like this:
@@ -218 +222 @@
 
 Because it is possible, and usual, to run multiple versions of {{{loader.py}}} (in order to speed up loading time), the method to begin processing a new batch is part of a critical section. This simply means that the 'batch' table in the {{{ippToPsps}}} database is locked by a client looking for a new item to process, then released afterwards, ensuring that two separate {{{loader.py}}} instances do not attempt to process the same item. 
+
+it is usual to run instances of {{{loader.py}}} in a screen session.
 === Test mode ===