Changes between Version 240 and Version 241 of ippToPsps

Timestamp:

Apr 2, 2012, 6:01:48 PM (14 years ago)

Author:

rhenders

Comment:

--

ippToPsps

@@ -144 +144 @@
 The first thing a {{{loader.py}}} instance does is look to the queue tables in the {{{ippToPsps}}} database for a list of items to process. It first attempt to secure a 'stripe' of RA (from the [wiki:ippToPsps_Database#Thestripetable stripe table]) that is not being processed by any other {{{loader.py}}} clients. If all available stripes have already been reserved, then it chooses the stripe with the largest number of pending items.
 
-With a stripe chosen, {{{loader.py}}} gathers a list of box_ids (each stripe is sub-divided into boxes). It works its way through each box, processing all pending items.
+With a stripe chosen, {{{loader.py}}} gathers a list of box_ids (each stripe is sub-divided into boxes). It works its way through each box, processing all pending items. Batches are processed like this:
+
+ * all relevant FITS tables from a given {{{smf}}} or {{{cmf}}} or DVO database are read into temporary tables in the scratch MySQL database
+ * empty MySQL tables for PSPS output are created, also in the 'scratch' database. These tables match the shape of the final PSPS database tables exactly.
+ * all relevant columns from the temporary IPP tables are copied into the PSPS tables, discarding duplicates where necessary
+ * for P2 and ST batches, the DVO database is accessed and all IDs are read for the detections we are interested in
+ * any remaining NULL values are replaced with -999, as required by the PSPS loader
+ * PSPS tables are exported to a FITS file
+ * FITS file is published, complete with a batch 'manifest' file, to the datastore
+
+The reading and export of FITS tables is done using STILTS. For import, we can specify which columns we wish to import from the IPP smf and cmf files (we don't need everything).
 
 Because it is possible, and usual, to run multiple versions of {{{loader.py}}} (in order to speed up loading time), the method to begin processing a new batch is part of a critical section. This simply means that the 'batch' table in the {{{ippToPsps}}} database is locked by a client looking for a new item to process, then released afterwards, ensuring that two separate {{{loader.py}}} instances do not attempt to process the same item. 
-
 === Test mode ===