Index: trunk/doc/design/design.tex =================================================================== --- trunk/doc/design/design.tex (revision 421) +++ trunk/doc/design/design.tex (revision 422) @@ -1,3 +1,3 @@ -%%% $Id: design.tex,v 1.3 2004-04-13 04:05:15 price Exp $ +%%% $Id: design.tex,v 1.4 2004-04-13 05:28:11 price Exp $ %\documentclass[panstarrs,psreport]{panstarrs} \documentclass[panstarrs]{panstarrs} @@ -325,5 +325,5 @@ stages on separate machines and monitors their progress. An example of the controller functionality is ``Run the phase 2 processing on -exposure number 1234''. +exposure number 1234 using machines 1,3,5,7,9''. \subsubsection{Scheduler} @@ -333,5 +333,5 @@ images as they are received. An example of the scheduler functionality is ``I've just received exposure number 1234; run phase -1--4 controllers on these''. +1--4 controllers on exposure 1234''. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -348,15 +348,16 @@ The IPP Pixel Server (IPS) is a repository for all image pixel data required by the IPP. Images may reside in the IPS for different -periods depending on their use and type. Images stored by the IPS +periods depending on their use and type. Data stored by the IPS include the raw images, the calibration images, intermediate processing stage images as needed, final processed images, difference -images, and image subsections. The IPS must retain images as long as -they are needed, up to the lifetime of the project. In order to -achieve the I/O requirements, the IPS may maintain the pixel data -distributed across the processor nodes in an organized fashion, i.e.\ -associating specific machines with specific OTAs. The IPS interacts -with the IPP Metadata Database to allow other systems or subsystems to -identify the available images meeting specified criteria. IPS -specifications are described in the IPS subsystem specification. +images, and image subsections, \tbd{along with the associated +metadata}. The IPS must retain images as long as they are needed, up +to the lifetime of the project. In order to achieve the I/O +requirements, the IPS may maintain the pixel data distributed across +the processor nodes in an organized fashion, i.e.\ associating +specific machines with specific OTAs. The IPS interacts with the IPP +Metadata Database to allow other systems or subsystems to identify the +available images meeting specified criteria. IPS specifications are +described in the IPS subsystem specification. In addition to storing the pixel data, the IPS is responsible for @@ -369,42 +370,38 @@ \begin{enumerate} -\item IPP Pixel Data Scheduler -\item IPP Pixel Data Locality Optimizer -\item IPP Pixel Data Database -\item IPP Pixel Data Retrieval Agent -\item IPP Pixel Data Query Library -\item IPP Pixel Data I/O Library -\end{enumerate} - -\subparagraph{IPP Pixel Data Scheduler (IPP-PDS)} - -The IPP Pixel Data Scheduler coordinates the movement of image data -onto {\em local} storage for processing by the IPP System and executes -batch image data management tasks. By ``local storage'' is meant -storage accessible from a particular local machine (i.e.\ either on a -disk physically connected to the machine, or a disk mounted over the -network). - -The IPP Pixel Data Scheduler has four basic modes of operation: +\item IPP Pixel Server Scheduler (IPSS) +\item IPP Pixel Server Data Locality Optimizer (IPSDLO) +\item IPP Pixel Server Database (IPSD) +\item IPP Pixel Server Node Agent (IPSNA) +\item IPP Pixel Server I/O Library (IPSIOL) +\end{enumerate} + +\subparagraph{IPP Pixel Server Scheduler (IPSS)} + +The IPSS coordinates the movement of image data and executes batch +image data management tasks. The IPSS has four basic modes of +operation: \begin{itemize} -\item Copy external data: The IPP-PDS generates {\em retrieve data} - tasks which are executed on nodes specified by the IPP-DLO. This - mode will be used frequently to copy data from the Summit Pixel - Server to the IPP nodes for processing. -\item Delete data: The IPP-PDS looks up the location of the data in - the IPP Pixel Data Database and generates {\em delete data} tasks - which are executed on the appropriate nodes. This mode will be used - on a regular basis to clean old data that is no longer required. -\item Replicate data: The IPP-PDS generates {\em retrieve data} tasks - which are executed on nodes specified either by the ``replicate - data'' command, or by the IPP-DLO. This mode differs from the - ``copy external data'' mode in that it copies data already within - the IPP-PDS. This mode will be used to backup and rearrange data. -\item Move data: the IPP-PDS executes a replication followed by a +\item Retrieve external data: The IPSS generates {\em retrieve data} + tasks which are executed by the IPSNAs on nodes specified by the + IPSDLO. This mode will be used frequently to copy data from the + Summit Pixel Server to the IPP nodes for processing. +\item Delete data: The IPSS looks up the location of the data in the + IPP Pixel Data Database and generates {\em delete data} tasks which + are executed by the IPSNAs on the appropriate nodes. This mode will + be used on a regular basis to clean old data that is no longer + required. +\item Replicate data: The IPSS generates {\em copy data} tasks which + are executed by the IPSNAs on nodes specified either by the + ``replicate data'' command, or by the IPPDLO. This mode differs + from the ``copy external data'' mode in that it copies data already + within the IPSS. This mode will be used to backup and rearrange + data. +\item Move data: the IPSS executes a replication followed by a deletion. This mode will be used to reorganise the storage. \end{itemize} -It is not intended that the IPP-PDS will be used by the nodes in the +It is not intended that the IPSS will be used by the nodes in the course of processing --- it is only for bulk data management. ``Copy external data'' mode will be used frequently to retrieve data from the @@ -415,53 +412,59 @@ -\subparagraph{IPP Pixel Data Locality Optimizer (IPP-PDLO)} - -The IPP Pixel Data Locality Optimizer is a data task filter. Data -tasks generated by the IPP Pixel Data Scheduler are passed through the -IPP Pixel Data Locality Optimizer which may assign tasks to specific -nodes. This component is a merely a plug-in and may be bypassed -depending upon the operating mode of the IPP Pixel Data Scheduler. - -\subparagraph{IPP Pixel Data Database (IPP-PDD)} - -The IPP Pixel Data Database contains image data locations \tbd{and the -associated metadata}. - -The IPP-PDD will contain at least: - +\subparagraph{IPP Pixel Server Data Locality Optimizer (IPPDLO)} + +Data tasks generated by the IPSS are passed through the IPSDLO which +assigns write tasks to specific nodes. This allows the location of +the data to be optimized so that it resides on the node which will +process it. + +\subparagraph{IPP Pixel Server Database (IPSD)} + +The IPSD contains image data locations \tbd{and the associated +metadata}. The IPSD will contain at least: \begin{itemize} \item The location of image data and its associated metadata that is -available for retrieval from the Summit Pixel Server. + available for retrieval from the Summit Pixel Server. \item The location of image data and its associated metadata that is -yet to be processed by the IPP System. + yet to be processed by the IPP System. \item The location of calibration data and its associated metadata for -processing within the IPP System. + processing within the IPP System. \item The location of reduced image data and its associated metadata as -generated by the IPP System. + generated by the IPP System. \item The location of difference image data and its associated metadata as -generated by the IPP System. + generated by the IPP System. \item The location of stacked image data and its associated metadata as -generated by the IPP System. + generated by the IPP System. \item A history of data management commands and actions. \end{itemize} -\subparagraph{IPP Pixel Data Retrieval Agent (IPP-PDRA)} - -The IPP Pixel Data Retrieval Agent acquires image data from a -specified location, possibly the Summit Pixel Server(s), and stores it -at a specified location. The IPP-PDRA is independent of the -underlying storage medium by using the IPP Pixel Data I/O Library. - - -\subparagraph{IPP Pixel Data I/O Library (IPP-PDIOL)} - -The PDIOL is the workhorse of the Pixel Server system. It is a -library for retrieving files from and storing files to Uniform -Resource Identifiers (URIs), which can be used on the nodes to access -the pixel data. It will be able to: +\subparagraph{IPP Pixel Server Node Agent (IPSNA)} + +The IPSNA runs on a node to perform the operations required by the IPSS +and IPSIOL. This includes: \begin{itemize} -\item Locate new and reduced data for an exposure. -\item Locate the appropriate calibration data for an exposure. -\item Add the storage location and metadata of new data. +\item Retrieve data from an external source (e.g.\ the Summit Pixel + Server) to a local disk as requested by the IPSS. +\item Copy data from one of the other nodes to a local disk as +requested by the IPPS. +\item Delete data from a local disk as requested by the IPSS or + through the IPSIOL. +\item Respond to requests for data made by nodes through the IPSIOL. +\end{itemize} + +\tbd{The Agent does not wear a suit, nor does it know kung fu.} + +\subparagraph{IPP Pixel Server I/O Library (IPSIOL)} + +The IPSIOL is the workhorse of the IPP Pixel Server system. It is a +library for reading and writing pixel data to the IPP Pixel Server. +It will generally be used on the nodes, although the IPSS will also +make use of it. The IPSIOL will be able to: +\begin{itemize} +\item Lookup the location of new and reduced data for an exposure. +\item Lookup the location of the appropriate calibration data for an + exposure. +\item Open a file at the location returned by a lookup. +\item Write new data and metadata to a specified location. \item Update the storage location and/or metadata of any data. \item Remove the storage location of data and metadata that has been @@ -470,27 +473,24 @@ - \paragraph{Pixel Data Flow} -\subparagraph{Acquisition} +Below we sketch out the intended sequence of events for common +operations. + +\subparagraph{Acquisition of data from the Summit Pixel Server} \begin{enumerate} \item The Summit Pixel Server sends a ``new data notification'' to the -IPP Pixel Data Data Scheduler. -\item The IPP Pixel Data Data Scheduler generates a {\em retrieve -data} task which is filtered through the IPP Pixel Data Locality -Optimizer, which possibly assigns the task to a specific node or group -of nodes. -\item The IPP Pixel Data Scheduler farms out the various copy tasks to -the nodes, which spawn IPP Pixel Data Retrieval Agents. -\item The IPP Pixel Data Retrieval Agents downloads the image data +IPSS. +\item The IPSS generates the {\em retrieve data} tasks which are to be +executed on specific nodes (i.e.\ those which will reduce the raw +data). +\item Each specified node spawn IPSDRAs which downloads the image data from the Summit Pixel Server to the disk physically mounted on the node. -\item The node reports the finished task to the IPP Pixel Data Scheduler. -\item The IPP Pixel Data Scheduler updates the IPP Pixel Data Database to -the new storage location. -\item The IPP Pixel Data Scheduler notifies the IPP Scheduler that new data is -available by appending to a notification table in the IPP Pixel Data Database. -\tbd{In addition a notification maybe sent directly to the IPP Scheduler.} +\item The node reports the finished task to the IPSS. +\item The IPSS updates the IPSD to the new storage location. +\item The IPSS notifies the IPP Scheduler that new +data is available. \end{enumerate} @@ -502,42 +502,37 @@ \end{figure} -\subparagraph{Processing} - -\begin{enumerate} - -\item The IPP Scheduler gives the IPP Controller a Phase 2 image processing task. - -\begin{enumerate} -\item The IPP Controller passes the task to a \tbd{IPP Node Agent}. -\item The \tbd{IPP Node Agent} {\em spawns} a \tbd{IPP Image Processing Agent} and -passes it the task. -\item The \tbd{IPP Image Processing Agent} retrieves the exposure data to be processed -with the IPP Pixel Data I/O library and loads it into local memory. -\item The \tbd{IPP Image Processing Agent} retrieves the calibration data with -the IPP Pixel Data I/O library and loads it into local memory. -\item The \tbd{IPP Image Processing Agent} processes the Phase 2 task. -\item The \tbd{IPP Image Processing Agent} stores the processed data with the -IPP Pixel Data I/O library. -\end{enumerate} - -\item The IPP Scheduler gives the IPP Controller a Stage 4 image processing task. - -\begin{enumerate} -\item The IPP Controller passes the task to a \tbd{IPP Node Agent}. -\item The \tbd{IPP Node Agent} {\em spawns} a \tbd{IPP Image Processing Agent} and -passes it the task. -\item The \tbd{IPP Image Processing Agent} retrieves the reduced image data -with the IPP Pixel Data I/O library and loads it into local memory. -\item The \tbd{IPP Image Processing Agent} retrieves the best stacked image -data with the IPP Pixel Data I/O library and loads it into local memory. -\item The \tbd{IPP Image Processing Agent} retrieves the current working -stacked image data with the IPP Pixel Data I/O library and loads it into -local memory. -\item The \tbd{IPP Image Processing Agent} processes the Phase 4 task. -\item The \tbd{IPP Image Processing Agent} stores the difference image with -the IPP Pixel Data I/O library. -\item The \tbd{IPP Image Processing Agent} stores the new working stacked -image with the IPP Pixel Data I/O library. -\end{enumerate} +\subparagraph{Processing Reads} + +\begin{enumerate} +\item A processing stage needs pixel data, e.g.\ the appropriate +flat-field for an image being processed. +\item The processing stage uses the IPSIOL to look up the location of +the appropriate image. +\item The processing stage retrieves the required pixel data using the +IPSIOL and loads it into local memory. +\end{enumerate} + +\subparagraph{Processing Writes} + +\begin{enumerate} +\item A processing stage has produced pixel data which should be saved, e.g.\ the +subtracted image. +\item The processing stage uses the IPSIOL to look up the location the +image should be written to. +\item The processing stage uses the IPSIOL to write the image. +\end{enumerate} + +\subparagraph{Processing Updates} + +\begin{enumerate} +\item A processing stage needs to update pixel data, e.g.\ the +static sky image. +\item The processing stage uses the IPSIOL to look up the location of +the appropriate image. +\item The processing stage retrieves the required pixel data using the +IPSIOL and loads it into local memory. +\item The processing stage modifies the pixel data in local memory. +\item The processing stage uses the IPSIOL to write the image to the +previous location with an overwrite flag. \end{enumerate}