Changes between Version 36 and Version 37 of ippToPsps

Timestamp:

Mar 18, 2010, 12:18:10 PM (16 years ago)

Author:

rhenders

Comment:

--

ippToPsps

@@ -18 +18 @@
 ||'''Batch name''' || '''PSPS name''' || '''Description''' || '''IPP Source''' ||
 || Initialisation || IN || Metadata for the other batches eg filter ID, survey ID etc || generated from XML config ||
-|| Detection || P2 || Single exposure detections || generated from one {{{smf}}} file per exposure ||
+|| Detection || P2 || Single exposure detections || generated from one {{{smf}}} file per exposure plus associated DVO database ||
 || Difference || OB || difference image detections|| generated from one {{{cmf}}} file per skycell per exposure ||
 || Stack || ST || stack image detections || generated from... ||
 
+
+= Configuration =
+
+Due to the potential for changes in both input and output for {{{ippToPsps}}}, the code is heavily configurable. Config files are in an XML format as this affords the most flexibility. {{{ippToPsps}}} is pointed to a config directory, under which subdirectories for each batch type hold the various XML config files.
+
+== Table shapes ==
+
+All FITS tables mirror PSPS database tables. Since the schema may change, {{{ippToPsps}}} reads table shapes from an XML config. This config can be regenerated from the master PSPS schema using a script ({{{pspsSchema2xml.pl}}}) in the scripts directory.
+
+== Initialisation data ==
+
+The table shapes of the initialisation batch are handled as above. The actual initialisation data (lists of filters etc), which is liable to change, is held in an XML config and used by {{{ippToPsps}}} to populate the tables in the FITS file. This data is also used when generating other batch types, detections for example, as look-up tables for setting survey ID etc.
+
+== IPP to PSPS mappings ==
+
+Most data to be loaded into the FITS tables comes from IPP {{{smf}}} or {{{cmf}}} files. For many columns, there is a direct mapping between these files and the PSPS database column. These mappings are detailed in a config.
+
+
 = Architecture =
 
 == ippToPsps ==
 
-{{{ippToPsps}}} is a C program within the IPP build. When given the correct arguments it will generate a single FITS for the specified product (above). The program is run from a perl script, which itself generates a list of exposure IDs based on arguments provided by the user (label etc). An instance of {{{ippToPsps}}} is run per exposure ID. Upon completion, the calling script bundles the resultant FITS files up as a ''job'', then publishes it to the datastore, ready for collection by the DXLayer.
-
-= Configuration =
-
-Due to the potential for changes in both input and output for {{{ippToPsps}}}, the code is heavily configurable. Config files are in an XML format as this affords the most flexibility. {{{ippToPsps}}} is pointed to a config directory, under which subdirectories for each batch type hold the various XML config files.
-
-== Table shapes ==
-
-All FITS tables mirror PSPS database tables. Since the schema may change, {{{ippToPsps}}} reads table shapes from an XML config. This config can be regenerated from the master PSPS schema using a script ({{{pspsSchema2xml.pl}}}) in the scripts directory.
-
-== Initialisation data ==
-
-The table shapes of the initialisation batch are handled as above. The actual initialisation data (lists of filters etc), which is liable to change, is held in an XML config and used by {{{ippToPsps}}} to populate the tables in the FITS file. This data is also used when generating other batch types, detections for example, as look-up tables for setting survey ID etc.
-
-== IPP to PSPS mappings ==
-
-Most data to be loaded into the FITS tables comes from IPP {{{smf}}} or {{{cmf}}} files. For many columns, there is a direct mapping between these files and the PSPS database column. These mappings are detailed in a config.
-
-= DXLayer =
-
-...
-
-= Loader =
-
-...
+{{{ippToPsps}}} is a C program within the IPP build. When given the correct arguments it will generate a single FITS for the specified product (above). The program is run from a Perl script, which itself generates a list of exposure IDs based on arguments provided by the user (label etc). An instance of {{{ippToPsps}}} is run per exposure ID. Upon completion, the calling script bundles the resultant FITS files up as a ''job'', then publishes it to the datastore, ready for collection by the DXLayer.
+
+== DXLayer ==
+
+The DXLayer polls the datastore waiting for new batches. Upon receipt of a new batch, the FITS files are converted to a csv format, suitable for ingest by the ODM.
+
+== ODM Loader ==
+
+Performs validation on incoming data based on metadata previously loaded as an initialisation batch (see above). If validation is successfully, new batches are merged into the PSPS database.
 
 = Batch types =