Changeset 422 for trunk/doc/design/design.tex

Timestamp:

Apr 12, 2004, 7:28:11 PM (22 years ago)

Author:

Paul Price

Message:

Hacking the Pixel Server section to simplify it and explain it more.
Changed the names and acronymns and removed one component which didn't
seem to be necessary. Attempted to flesh out the basic operations.

File:

• trunk/doc/design/design.tex (modified) (8 diffs)

trunk/doc/design/design.tex

@@ -1 @@
-%%% $Id: design.tex,v 1.3 2004-04-13 04:05:15 price Exp $
+%%% $Id: design.tex,v 1.4 2004-04-13 05:28:11 price Exp $
 %\documentclass[panstarrs,psreport]{panstarrs}
 \documentclass[panstarrs]{panstarrs}
@@ -325 +325 @@
 stages on separate machines and monitors their progress.  An example
 of the controller functionality is ``Run the phase 2 processing on
-exposure number 1234''.
+exposure number 1234 using machines 1,3,5,7,9''.
 
 \subsubsection{Scheduler}
@@ -333 +333 @@
 images as they are received.  An example of the scheduler
 functionality is ``I've just received exposure number 1234; run phase
-1--4 controllers on these''.
+1--4 controllers on exposure 1234''.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -348 +348 @@
 The IPP Pixel Server (IPS) is a repository for all image pixel data
 required by the IPP.  Images may reside in the IPS for different
-periods depending on their use and type.  Images stored by the IPS
+periods depending on their use and type.  Data stored by the IPS
 include the raw images, the calibration images, intermediate
 processing stage images as needed, final processed images, difference
-images, and image subsections.  The IPS must retain images as long as
-they are needed, up to the lifetime of the project.  In order to
-achieve the I/O requirements, the IPS may maintain the pixel data
-distributed across the processor nodes in an organized fashion, i.e.\
-associating specific machines with specific OTAs.  The IPS interacts
-with the IPP Metadata Database to allow other systems or subsystems to
-identify the available images meeting specified criteria.  IPS
-specifications are described in the IPS subsystem specification.
+images, and image subsections, \tbd{along with the associated
+metadata}.  The IPS must retain images as long as they are needed, up
+to the lifetime of the project.  In order to achieve the I/O
+requirements, the IPS may maintain the pixel data distributed across
+the processor nodes in an organized fashion, i.e.\ associating
+specific machines with specific OTAs.  The IPS interacts with the IPP
+Metadata Database to allow other systems or subsystems to identify the
+available images meeting specified criteria.  IPS specifications are
+described in the IPS subsystem specification.
 
 In addition to storing the pixel data, the IPS is responsible for
@@ -369 +370 @@
 
 \begin{enumerate}
-\item IPP Pixel Data Scheduler
-\item IPP Pixel Data Locality Optimizer
-\item IPP Pixel Data Database
-\item IPP Pixel Data Retrieval Agent
-\item IPP Pixel Data Query Library
-\item IPP Pixel Data I/O Library
-\end{enumerate}
-
-\subparagraph{IPP Pixel Data Scheduler (IPP-PDS)}
-
-The IPP Pixel Data Scheduler coordinates the movement of image data
-onto {\em local} storage for processing by the IPP System and executes
-batch image data management tasks.  By ``local storage'' is meant
-storage accessible from a particular local machine (i.e.\ either on a
-disk physically connected to the machine, or a disk mounted over the
-network).
-
-The IPP Pixel Data Scheduler has four basic modes of operation:
+\item IPP Pixel Server Scheduler (IPSS)
+\item IPP Pixel Server Data Locality Optimizer (IPSDLO)
+\item IPP Pixel Server Database (IPSD)
+\item IPP Pixel Server Node Agent (IPSNA)
+\item IPP Pixel Server I/O Library (IPSIOL)
+\end{enumerate}
+
+\subparagraph{IPP Pixel Server Scheduler (IPSS)}
+
+The IPSS coordinates the movement of image data and executes batch
+image data management tasks.  The IPSS has four basic modes of
+operation:
 
 \begin{itemize}
-\item Copy external data: The IPP-PDS generates {\em retrieve data}
-  tasks which are executed on nodes specified by the IPP-DLO.  This
-  mode will be used frequently to copy data from the Summit Pixel
-  Server to the IPP nodes for processing.
-\item Delete data: The IPP-PDS looks up the location of the data in
-  the IPP Pixel Data Database and generates {\em delete data} tasks
-  which are executed on the appropriate nodes.  This mode will be used
-  on a regular basis to clean old data that is no longer required.
-\item Replicate data: The IPP-PDS generates {\em retrieve data} tasks
-  which are executed on nodes specified either by the ``replicate
-  data'' command, or by the IPP-DLO.  This mode differs from the
-  ``copy external data'' mode in that it copies data already within
-  the IPP-PDS.  This mode will be used to backup and rearrange data.
-\item Move data: the IPP-PDS executes a replication followed by a
+\item Retrieve external data: The IPSS generates {\em retrieve data}
+  tasks which are executed by the IPSNAs on nodes specified by the
+  IPSDLO.  This mode will be used frequently to copy data from the
+  Summit Pixel Server to the IPP nodes for processing.
+\item Delete data: The IPSS looks up the location of the data in the
+  IPP Pixel Data Database and generates {\em delete data} tasks which
+  are executed by the IPSNAs on the appropriate nodes.  This mode will
+  be used on a regular basis to clean old data that is no longer
+  required.
+\item Replicate data: The IPSS generates {\em copy data} tasks which
+  are executed by the IPSNAs on nodes specified either by the
+  ``replicate data'' command, or by the IPPDLO.  This mode differs
+  from the ``copy external data'' mode in that it copies data already
+  within the IPSS.  This mode will be used to backup and rearrange
+  data.
+\item Move data: the IPSS executes a replication followed by a
   deletion.  This mode will be used to reorganise the storage.
 \end{itemize}
 
-It is not intended that the IPP-PDS will be used by the nodes in the
+It is not intended that the IPSS will be used by the nodes in the
 course of processing --- it is only for bulk data management.  ``Copy
 external data'' mode will be used frequently to retrieve data from the
@@ -415 +412 @@
 
 
-\subparagraph{IPP Pixel Data Locality Optimizer (IPP-PDLO)}
-
-The IPP Pixel Data Locality Optimizer is a data task filter.  Data
-tasks generated by the IPP Pixel Data Scheduler are passed through the
-IPP Pixel Data Locality Optimizer which may assign tasks to specific
-nodes.  This component is a merely a plug-in and may be bypassed
-depending upon the operating mode of the IPP Pixel Data Scheduler.
-
-\subparagraph{IPP Pixel Data Database (IPP-PDD)}
-
-The IPP Pixel Data Database contains image data locations \tbd{and the
-associated metadata}.
-
-The IPP-PDD will contain at least:
-
+\subparagraph{IPP Pixel Server Data Locality Optimizer (IPPDLO)}
+
+Data tasks generated by the IPSS are passed through the IPSDLO which
+assigns write tasks to specific nodes.  This allows the location of
+the data to be optimized so that it resides on the node which will
+process it.
+
+\subparagraph{IPP Pixel Server Database (IPSD)}
+
+The IPSD contains image data locations \tbd{and the associated
+metadata}.  The IPSD will contain at least:
 \begin{itemize}
 \item The location of image data and its associated metadata that is
-available for retrieval from the Summit Pixel Server.
+  available for retrieval from the Summit Pixel Server.
 \item The location of image data and its associated metadata that is
-yet to be processed by the IPP System.
+  yet to be processed by the IPP System.
 \item The location of calibration data and its associated metadata for
-processing within the IPP System.
+  processing within the IPP System.
 \item The location of reduced image data and its associated metadata as
-generated by the IPP System.
+  generated by the IPP System.
 \item The location of difference image data and its associated metadata as
-generated by the IPP System.
+  generated by the IPP System.
 \item The location of stacked image data and its associated metadata as
-generated by the IPP System.
+  generated by the IPP System.
 \item A history of data management commands and actions.
 \end{itemize}
 
-\subparagraph{IPP Pixel Data Retrieval Agent (IPP-PDRA)}
-
-The IPP Pixel Data Retrieval Agent acquires image data from a
-specified location, possibly the Summit Pixel Server(s), and stores it
-at a specified location.  The IPP-PDRA is independent of the
-underlying storage medium by using the IPP Pixel Data I/O Library.
-
-
-\subparagraph{IPP Pixel Data I/O Library (IPP-PDIOL)}
-
-The PDIOL is the workhorse of the Pixel Server system.  It is a
-library for retrieving files from and storing files to Uniform
-Resource Identifiers (URIs), which can be used on the nodes to access
-the pixel data.  It will be able to:
+\subparagraph{IPP Pixel Server Node Agent (IPSNA)}
+
+The IPSNA runs on a node to perform the operations required by the IPSS
+and IPSIOL.  This includes:
 \begin{itemize}
-\item Locate new and reduced data for an exposure.
-\item Locate the appropriate calibration data for an exposure.
-\item Add the storage location and metadata of new data.
+\item Retrieve data from an external source (e.g.\ the Summit Pixel
+  Server) to a local disk as requested by the IPSS.
+\item Copy data from one of the other nodes to a local disk as
+requested by the IPPS.
+\item Delete data from a local disk as requested by the IPSS or
+  through the IPSIOL.
+\item Respond to requests for data made by nodes through the IPSIOL.
+\end{itemize}
+
+\tbd{The Agent does not wear a suit, nor does it know kung fu.}
+
+\subparagraph{IPP Pixel Server I/O Library (IPSIOL)}
+
+The IPSIOL is the workhorse of the IPP Pixel Server system.  It is a
+library for reading and writing pixel data to the IPP Pixel Server.
+It will generally be used on the nodes, although the IPSS will also
+make use of it.  The IPSIOL will be able to:
+\begin{itemize}
+\item Lookup the location of new and reduced data for an exposure.
+\item Lookup the location of the appropriate calibration data for an
+  exposure.
+\item Open a file at the location returned by a lookup.
+\item Write new data and metadata to a specified location.
 \item Update the storage location and/or metadata of any data.
 \item Remove the storage location of data and metadata that has been
@@ -470 +473 @@
 
 
-
 \paragraph{Pixel Data Flow}
 
-\subparagraph{Acquisition}
+Below we sketch out the intended sequence of events for common
+operations.
+
+\subparagraph{Acquisition of data from the Summit Pixel Server}
 
 \begin{enumerate}
 \item The Summit Pixel Server sends a ``new data notification'' to the
-IPP Pixel Data Data Scheduler.
-\item The IPP Pixel Data Data Scheduler generates a {\em retrieve
-data} task which is filtered through the IPP Pixel Data Locality
-Optimizer, which possibly assigns the task to a specific node or group
-of nodes.
-\item The IPP Pixel Data Scheduler farms out the various copy tasks to
-the nodes, which spawn IPP Pixel Data Retrieval Agents.
-\item The IPP Pixel Data Retrieval Agents downloads the image data
+IPSS.
+\item The IPSS generates the {\em retrieve data} tasks which are to be
+executed on specific nodes (i.e.\ those which will reduce the raw
+data).
+\item Each specified node spawn IPSDRAs which downloads the image data
 from the Summit Pixel Server to the disk physically mounted on the
 node.
-\item The node reports the finished task to the IPP Pixel Data Scheduler.
-\item The IPP Pixel Data Scheduler updates the IPP Pixel Data Database to
-the new storage location.
-\item The IPP Pixel Data Scheduler notifies the IPP Scheduler that new data is
-available by appending to a notification table in the IPP Pixel Data Database.
-\tbd{In addition a notification maybe sent directly to the IPP Scheduler.}
+\item The node reports the finished task to the IPSS.
+\item The IPSS updates the IPSD to the new storage location.
+\item The IPSS notifies the IPP Scheduler that new
+data is available.
 \end{enumerate}
 
@@ -502 +502 @@
 \end{figure}
 
-\subparagraph{Processing}
-
-\begin{enumerate}
-
-\item The IPP Scheduler gives the IPP Controller a Phase 2 image processing task.
-
-\begin{enumerate}
-\item The IPP Controller passes the task to a \tbd{IPP Node Agent}.
-\item The \tbd{IPP Node Agent} {\em spawns} a \tbd{IPP Image Processing Agent} and
-passes it the task.
-\item The \tbd{IPP Image Processing Agent} retrieves the exposure data to be processed
-with the IPP Pixel Data I/O library and loads it into local memory.
-\item The \tbd{IPP Image Processing Agent} retrieves the calibration data with
-the IPP Pixel Data I/O library and loads it into local memory.
-\item The \tbd{IPP Image Processing Agent} processes the Phase 2 task.
-\item The \tbd{IPP Image Processing Agent} stores the processed data with the
-IPP Pixel Data I/O library.
-\end{enumerate}
-
-\item The IPP Scheduler gives the IPP Controller a Stage 4 image processing task.
-
-\begin{enumerate}
-\item The IPP Controller passes the task to a \tbd{IPP Node Agent}.
-\item The \tbd{IPP Node Agent} {\em spawns} a \tbd{IPP Image Processing Agent} and
-passes it the task.
-\item The \tbd{IPP Image Processing Agent} retrieves the reduced image data
-with the IPP Pixel Data I/O library and loads it into local memory.
-\item The \tbd{IPP Image Processing Agent} retrieves the best stacked image
-data with the IPP Pixel Data I/O library and loads it into local memory.
-\item The \tbd{IPP Image Processing Agent} retrieves the current working
-stacked image data with the IPP Pixel Data I/O library and loads it into
-local memory.
-\item The \tbd{IPP Image Processing Agent} processes the Phase 4 task.
-\item The \tbd{IPP Image Processing Agent} stores the difference image with
-the IPP Pixel Data I/O library.
-\item The \tbd{IPP Image Processing Agent} stores the new working stacked
-image with the IPP Pixel Data I/O library.
-\end{enumerate}
+\subparagraph{Processing Reads}
+
+\begin{enumerate}
+\item A processing stage needs pixel data, e.g.\ the appropriate
+flat-field for an image being processed.
+\item The processing stage uses the IPSIOL to look up the location of
+the appropriate image.
+\item The processing stage retrieves the required pixel data using the
+IPSIOL and loads it into local memory.
+\end{enumerate}
+
+\subparagraph{Processing Writes}
+
+\begin{enumerate}
+\item A processing stage has produced pixel data which should be saved, e.g.\ the
+subtracted image.
+\item The processing stage uses the IPSIOL to look up the location the
+image should be written to.
+\item The processing stage uses the IPSIOL to write the image.
+\end{enumerate}
+
+\subparagraph{Processing Updates}
+
+\begin{enumerate}
+\item A processing stage needs to update pixel data, e.g.\ the
+static sky image.
+\item The processing stage uses the IPSIOL to look up the location of
+the appropriate image.
+\item The processing stage retrieves the required pixel data using the
+IPSIOL and loads it into local memory.
+\item The processing stage modifies the pixel data in local memory.
+\item The processing stage uses the IPSIOL to write the image to the
+previous location with an overwrite flag.
 \end{enumerate}