Changes between Version 1 and Version 2 of DVO_Parallel

Timestamp:

May 30, 2012, 3:26:24 PM (14 years ago)

Author:

eugene

Comment:

--

DVO_Parallel

@@ -3 +3 @@
 DVO now has a parallel mode.  This page is meant to describe at a high level how this works in software and user guidance for working with a parallelized DVO database.
 
-=== Parallel Tables and the HostTable ===
+=== Parallel Tables and the \HostTable ===
 
 A parallelized DVO database has the Image and other top-level tables stored on a single machine (top-level host) in the primary CATDIR, while all of the average, measure, secfilt, missing (optional) tables are stored in a parallel set of directories on multiple machines.  Parallel operations are initiated on the top-level host machine, with the database of interest specified as usual with the catdir command (in dvo shell) or the CATDIR config variable.  Queries which need to interact with the parallel tables are sent to a remote client program.  The remote client may do some work or simply return the results of a query (presumably with the total amount of data substantially reduced).  The DVO program on the top-level host collects the results and uses them if needed for further operations locally.
 
-The remote machines and directories are managed with the file CATDIR/HostTable.dat.  This file contains one line for each remote host.  Line which start with '#' are comments and ignored, as are empty lines.  Each host listed in the table is associated with a host ID (integer > 0) and a directory where the remote data is stored.  A single real computer may be represented more than once, with different directories for each entry and different host ID values.
+The remote machines and directories are managed with the file CATDIR/HostTable.dat.  This file contains one line for each remote host.  Line which start with '#' are comments and ignored, as are empty lines.  Each host listed in the table is associated with a host ID (integer > 0) and a directory where the remote data is stored.  A single real computer may be represented more than once, with different directories for each entry and different host ID values.  Here is an example of a section of a Host Table:
+{{{
+# ID  Hostname  Catdir
+   4  ipp004    /data/ipp004.0/eugene/catdirs/3pi.20111229
+#  5  ipp005    /data/ipp005.0/eugene/catdirs/3pi.20111229 -- over used by dvo
+   6  ipp006    /data/ipp006.0/eugene/catdirs/3pi.20111229
+}}}
 
 The distribution of the remote files is managed with the program {{{dvodist}}} and tracked in the SkyTable table.  This table includes fields to identify the (primary) host id, an optional backup host ID, and a set of flags to track if the given table is available from the primary or backup location (or from the top-level host).