Context Navigation

← Previous Change
Next Change →

ippSDRS.tex

Timestamp:

Oct 13, 2004, 7:06:32 PM (22 years ago)

Author:

eugene

Message:

design work towards PDR

File:

: 1 edited

trunk/doc/design/ippSDRS.tex (modified) (11 diffs)

Legend:

: Unmodified
: Added
: Removed

trunk/doc/design/ippSDRS.tex

-              r1399
+              r2114
 %%% $Id: ippSDRS.tex,v 1.4 2004-08-06 19:06:01 eugene Exp $
+%%% $Id: ippSDRS.tex,v 1.5 2004-10-14 05:06:31 eugene Exp $
 \documentclass[panstarrs]{panstarrs}
 …
 \pagenumbering{arabic}
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \section{Scope}
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \subsection{Identification}
 …
 Pan-STARRS 1 (PS-1), the initial demonstration telescope to be
 constructed on Haleakala by Jan 2006.
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \subsection{System Overview}
 …
 roughly 2 years.
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \subsection{Document Overview}
 …
 Open Issues and TBDs in this document are marked \tbd{in bold, red
 type with surrounding square brackets}.
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \DocumentsInternalSection
 …
 \DocumentsEnd
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\section{System Design Decisions}
+\section{Subsystem Overview}
+The Pan-STARRS Image Processing Pipeline (IPP) performs the image
+processing and data analysis tasks needed to enable the scientific use
+of the images obtained by the Pan-STARRS telescopes.  The primary
+goals of the IPP are to process the science images from the Pan-STARRS
+telescopes and make the results available to other systems within
+Pan-STARRS.  It also is responsible for combining all of the science
+images in a given filter into a single representation of the
+non-variable component of the night sky called the ``Static Sky''.  To
+achieve these goals, the IPP also performs other analysis functions to
+generate the calibrations needed in the science image processing and
+to occasionally use the derived data to generate improved astrometric
+and photometric reference catalogs.  It also provides the
+infrastructure needed to store the incoming data and the resulting
+data products.
+The IPP inherits lessons learned, and in some cases code and prototype
+code, from several other astronomy image analysis systems, including
+Imcat (Kaiser), the Sloan Digital Sky Survey (REF), the Elixir system
+(Magnier \& Cuillandre), and Vista (Tonry).  Imcat and Vista have a
+large number of robust image processing functions.  SDSS has
+demonstrated a working analysis pipeline and large-scale database
+system for a dedicated project.  The Elixir system has demonstrated an
+automatic image processing system and an object database system for
+operational usage.
+The users of the IPP output are all systems internal to the Pan-STARRS
+project.  They consist of the Transient Science Client, which will
+receive the detections of transient objects on short time-scales; the
+Moving Object Processing System (MOPS), which will receive the
+detections of non-stationary transient objects on day-to-week
+timescales; and the Published Science Products Subsystem (PSPS), which
+will receive all data products of interest to the outside world, and
+will act as the long-term archive and publishing clearinghouse.
+An important operational choice for the IPP is the decision not to
+attempt to save all raw data.  Once the IPP is running in a standard
+operational mode, data will be processed by the pipeline and deleted
+when it is no longer needed.  Raw images will only be saved for a
+short period to allow tests and quality assurance, and potentially to
+allow systems which study transient phenomena to return to recent data
+for closer inspection.  In general, during normal operations, raw
+science images will be deleted after $\sim$1 month.
+The primary IPP hardware system on which the software operates will
+not be located at the summit.  Instead, because of thermal, power, and
+space constraints, the hardware will likely be located in a facility
+off the mountain.  A subset of processing tasks may eventually be
+assigned to machines at the summit if justified by the savings in data
+transfer time and cost.
+\subsection{Analysis Tasks and Stages}
+Specific programs are required to perform the processing steps listed
+above.  These can be divided into well-defined analysis stages, each
+of which operates on a particular unit of data, such as a single OTA
+image or a collection of astronomical objects.  Analysis tasks
+representing the different analysis stages are performed on the IPP
+computer cluster.  Note the distinction between the generic analysis
+{\em stage} and a specific analysis {\em task}.  An analysis stage
+represents a type of analysis which is performed, such as the basic
+image calibration and object detection analysis.  An analysis task is
+a particular realization of an analysis stage, e.g., the analysis of
+OTA number 61 from exposure 654321 to produce a specific set of output
+data products.  The analysis stages are discussed in detail in
+Section~\ref{IPP:AnalysisStages}.
+Depending on the particular stage, it may process individual images,
+collections of images, or on derived data products.  Because of the
+nature of the image data, many of the analysis stages can be run in
+parallel because, for example, the analysis of a chip in one image
+does not depend on the results from another chip.
+\subsection{Architectural Components}
+In order to achieve the required functionality, the IPP provides an
+infrastructure within which the Analysis Stages above are exectuted.
+We have divided the IPP software infrastructure into a number of
+clearly-defined architectural software units, listed as follows:
+\begin{itemize}
+\item {\bf Image Server:} This component is a large data store for all
+  images used by the IPP, including the raw images from the telescope,
+  the master calibration images, the reference static-sky images, and
+  any temporary image data products produced by the IPP.  The Image
+  Server accepts the incoming data and stores it until it is no longer
+  needed by other portions of the IPP.  The Image Server is not
+  restricted to imaging data: it is capable of storing any large data
+  files which are not well-suited for inclusion in a more structured
+  relational database and for which access needs to be widely
+  available beyond the individual process which created the file.
+\item {\bf Astrometry \& Photometry Database (AP DB):} This component
+  stores and manipulates astronomical objects detected in various
+  images, as identified above, including individual measurements of
+  objects on the images, the summary information about those objects,
+  and reference object data.  It also provides mechanisms for users to
+  query and manipulate the objects and detections.
+\item {\bf Metadata Database:} This component stores the data which is
+  not directly related to images or astronomical objects, but which is
+  needed to perform the IPP analyses.  The metadata may include the
+  summary weather information for each night, or details about the
+  filters, camera, telescopes, etc.
+\item {\bf IPP Controller:} In order to perform the analysis stages
+  required by the IPP, it is necessary to use distributed computing
+  processes on a large number of computers.  The IPP Controller
+  manages the collection of analysis tasks performed on these
+  machines.
+\item {\bf IPP Scheduler:} This component is a decision-making
+  mechanism which guides the operation of the IPP.  It evaluates the
+  currently available collection of data, identifies the necessary
+  analysis, and assigns the analysis tasks to the IPP Controller.
+\end{itemize}
+The relationship between these software units is shown in
+Figure~\ref{overview}.  This figure also shows the interactions
+between the IPP and other Pan-STARRS systems.  The architectural
+components are discussed in detail in
+Section~\ref{IPP:ArchComponents}.
+\begin{figure}
+\begin{center}
+\resizebox{6in}{!}{\includegraphics{pics/IPPoverview}}
+\caption{ \label{overview} IPP System Overview}
+\end{center}
+\end{figure}
+\subsection{IPP Hardware Organization}
+\begin{figure}
+\begin{center}
+\resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}}
+\caption{ \label{hardware} IPP Hardware Organization}
+\end{center}
+\end{figure}
+The IPP needs substantial computer resources, both in terms of
+computational power and in terms of data storage and network
+bandwidth.  The IPP requires relatively large amounts of data storage
+space, primarily for the image data.  Image data is organized in two
+categories.  First, there is the per-OTA data -- data associated with
+specific OTAs, including the raw images, the calibration images, and
+temporary processed images at various stages.  Second, there is the
+data associated with the static sky imagery, which is in turn
+organized into smaller sky-cell units.  In addition to image data,
+there are the somewhat smaller data entities of the Metadata Database
+and AP Database.
+The computer hardware is organized into nodes which provide both data
+storage and computational resources.  The data storage nodes are
+divided into three classes: those which deal with the per-OTA image
+data, those that provide the storage for the static sky images, and
+those that provide the storage for the other data systems, the
+Metadata Database and the AP Database.  In addition, the computational
+tasks related to Phase 2 take place on the per-OTA storage nodes and
+the Phase 4 computation takes place on the static sky storage nodes.
+Figure~\ref{hardware} shows our basic concept for the hardware
+organization for the IPP.  This diagram shows the two types of compute
+nodes: OTA-level processing and storage nodes (dominated by Phase 2)
+and static sky processing and storage nodes (mostly Phase 4).  Also
+shown are two switches which divide the network into OTA and
+Static-Sky portions.  In such an organization, the interswitch
+communication must meet the throughput needs between these network
+portions.  The additional data systems (Metadata Database and AP
+Database) are also shown.
+%%% needs some work / move around elsewhere
+\subsection{System Interfaces}
+\paragraph{MOPS and other Client Science Pipelines}
+The Client Science Programs (CSPs) and the Moving Object Processing
+System (MOPS) are not a part of the IPP, but are external systems.  We
+include them here to show the required interfaces.
+The CSPs and MOPS may query the Pixel DB, the Metadata DB and the
+Object DB.  In addition, they may write certain fields to the object
+DB (e.g., the external identifiers and class of object) as they
+process objects, and they may retrieve pixel data from the Nodes.
+Since ``CSPs'' is a vague term, we now give some examples which may
+help to illustrate the functionality.
+One example of a CSP is a web front-end to retrieve (published) images
+and objects from the Pixel DB and Object DB.
+Another example would be a program interested in searching for
+transiting extrasolar planets.  Such a program may periodically poll
+the Metadata DB for new processed observations in its region of
+interest (such as the Galactic Plane), retrieve the object photometry
+of all high signal-to-noise stars in the processed observations, and
+attempt to identify a planetary transit in progress.
+Yet another example would be a Stationary Transient Object Pipeline,
+which would periodically poll the Metadata DB for new processed
+observations, and query the Object DB for variable sources which were
+identified twice (so that they are not moving objects).
+\subsection{System Design Decisions}
 Since \PS{} is a survey project, all data from the telescopes will be
 …
 System (MOPS), and potentially other client science pipelines.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsection{System Overview}
+The \PS{} Image Processing Pipeline (IPP) consists of a collection of
+computer hardware and software organized to perform the tasks required
+to process images from the \PS{} telescopes.  The primary goal of the
+IPP is to process the science images from the \PS{} telescopes and
+make the results available to other systems within \PS{}.  To achieve
+this goal, the IPP must also perform other analysis functions to
+generate the calibrations needed in the science image processing and
+to occasionally use the derived data to generate improved astrometric
+and photometric reference catalogs.
+In order to meet these broad goals, the IPP must have the following
+capabilities:
+\section{System Design : Architectural Components}
+\subsection{IPP Image Server}
+\begin{figure}
+\psfig{file=pics/ImageServer,width=15cm,angle=0}
+\caption{The components of the IPP Image Server.}
+\label{fig:ImageServer}
+\end{figure}
+The IPP Image Server is a repository for all images and other large
+data files required by the IPP.  In addition, it provides tools for
+managing the distribution of these large data files and for accessing
+the files.  Data files stored by the IPP Image Server include the raw
+images, the calibration images, intermediate processing stage images
+as needed, final processed images, difference images, image
+subsections, and any large non-imaging datafiles needed by the IPP.
+The IPP Image Server must retain the files for as long as they are
+needed by the IPP.
+The IPP Image Server is a parallel storage system.  It stores data
+across a collection of computer nodes, each with their own data
+storage resources.  Any single file is stored on only a single
+computer and storage system.  In order to achieve the data throughput
+requirements, the IPP Image Server may distribute the images across
+the processor nodes in an organized fashion, i.e.\ associating
+specific machines with specific detectors.  It is not the
+responsibility of the IPP Image Server to determine which computer
+should be associated with a specific data concept (Chip / region of
+sky), but it must enable the association of a particular file with a
+particular machine.
+There are three data concepts relevant to the IPP Image Server:
 \begin{itemize}
+\item Store a large amount of image data, and other derived data
+products (metadata and extracted objects);
+\item Provide access mechanisms to these data products (both to the
+subsystems of the IPP and in some cases to external users);
+\item Continuously accept new image data and metadata from the
+telescope system;
+\item Execute various analysis processes using these data products;
+and
+\item Provide the decision-making logic needed to guide the data
+processing, and to automatically launch the data processing tasks on
+an appropriate timescale.
+\item {\bf storage object} This represents a single, unique data
+  entity the Image Server.
+\item {\bf instance} A single copy of the storage object in the Image
+  Server.  In general, given storage object may have several instances
+  in the Image Server, normally on different computer nodes.
+\item {\bf file ID} This is the identifier of a particular storage
+  object in the Image Server.  The file ID is simply a unique string,
+  equivalent to the filename in a UNIX file system.
 \end{itemize}
+The IPP therefore includes subsystems which provide the data storage
+framework, the data analysis framework, and the scheduling of the
+analysis processes.  The data storage subsystems also provide
+interface mechanisms to the external \PS{} systems.
+The IPP architecture can be viewed in several possible ways.  We first
+consider the software architecture components needed by the IPP.
+These subsystems provide the infrastructure for the data storage and
+the data processing.  Next, we consider the analysis pipelines which
+make up the major processing tasks that must be performed by the IPP.
+Finally, we consider the hardware organization required to efficiently
+and cost-effectively achieve the necessary computing and storage
+requirements.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsection{System Architecture}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsubsection{Architectural Components}
+In Figure~\ref{fig:functionalities} we show the functionality of the
+IPP.
+The Observatory and Telescope System (\textbf{OATS}) system at the
+summit periodically produces metadata (e.g.\ weather measurements,
+observations completed) and pixel data (the image pixels from the
+cameras).  The \textbf{Pollster} regularly (e.g., twice per minute)
+polls OATS for the existence of new data.  If new data exists, the
+Pollster writes it to the \textbf{Metadata DB}, which maintains a
+table of observations that have been obtained and whether these
+observations are reduced, not reduced, or being reduced.  The
+\textbf{Scheduler} regularly (e.g., twice per minute) polls the
+Metadata DB for observations that match predefined criteria that are
+required to run reduction processes.  For example, the Phase 1
+processing requires that Phase 0 has been run on a focal plane
+metadata, and also requires that the observations are available and
+have not yet been processed.  If the criteria are met, the appropriate
+stage is passed to the \textbf{Localiser} which, checks the
+\textbf{Pixel DB} to determine if the stage should be performed on a
+particular node.  The Localiser passes the reduction stage to be
+processed, along with the preferred (or mandatory) node that should
+execute the reduction stage, to the \textbf{Controller}.  It is the
+Controller's responsibility to maintain the list of reduction stages
+to be processed and execute these stages on the \textbf{Nodes}.  The
+Nodes may retrieve the pixel data from OATS, they write to the Pixel
+DB the location of the products of the reduction and report their
+completion to the Controller.
+External systems, such as the Moving Object Processing System
+(\textbf{MOPS}) and other Client Science Pipelines (\textbf{CSPs})
+read the Metadata DB and the Object DB.  They may also write to the
+Object DB the classification of particular objects (e.g., identify an
+object as an asteroid).  Also, the MOPS and CSPs may also query the
+Pixel DB for the location of pixel data and copies data from the
+Nodes.
+\begin{figure}
+\psfig{file=pics/IPPfunctionalities,width=15cm,angle=0}
+\caption{The functionalities of the architectural design.  See the text
+for further explanation.}
+\label{fig:functionalities}
+\end{figure}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{OATS}
+The Observatory And Telescope System (OATS) is not a part of the IPP,
+but interfaces are required with it in order to allow the Pollster to
+get the list of observations not in the Metadata DB, and the nodes to
+retrieve pixel data.  Also, the Scheduler may report the need for new
+calibration data.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{Pollster}
+The Pollster is a program that polls OATS at regular intervals for the
+existence of observations not contained in the Metadata DB.  New
+weather and image metadata are written to the Metadata DB.
+There is no reason why this architectural component cannot be
+contained within another (such as the Scheduler), but it is shown here
+as separate for simplicity.
+A polling model is adopted so that OATS' interface may be kept as
+simple as possible --- OATS should not be concerned with whether the
+IPP has received notifications.  Under this polling model, it is
+specifically the responsibility of the IPP to retrieve from OATS the
+metadata that is not not already in the Metadata DB.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{Metadata DB}
+The Metadata DB stores and maintains the metadata\footnote{Note that
+metadata is any data which is not pixel data or object data.},
+including the list of images taken by the telescope system and whether
+these images have been processed.  The Metadata DB is regularly polled
+by the Scheduler to determine what images are ready to be processed.
+Both the Scheduler and the Pollster update the status of the Metadata
+DB --- the Pollster as new images become available at the Summit, and
+the Scheduler as images are processed.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{Scheduler}
+The Scheduler is responsible for determining the processing stages
+that are required to be run on any data.  Examples of these processing
+stages are ``Copy the pixels from the summit'' and ``Run Phase 2
+processing on chip 12 of exposure 123''.
+Processing stages to be executed are passed to the Localiser, which
+returns to the Scheduler the list of processing stages with node
+assignments to each of the stages.  This list of processing stages
+with node assignments is passed to the Controller for execution.
+Processing stages which have executed are reported by the Controller,
+which updates the Metadata DB appropriately.
+The Scheduler may also interact with OATS to inform it of the need
+for new calibration data.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{Localiser}
+\label{sec:localiser}
+It is the duty of the Localiser to assign processing stages to
+particular nodes.  This may be in order to optimise performance by
+distributing the stages across the nodes, or in the simplest possible
+case, it may make no recommendation upon the node which performs a
+particular stage.
+The Localiser may query the Pixel DB in order to identify the location
+of calibration data that may be needed for the processing stage to run
+(and in all likelihood, assign the processing stage to the same node as
+that which holds the calibration data).
+The Localiser may either demand or request that a stage is performed on
+a particular node, or make no recommendation, and passes the processing
+stage back to the Scheduler.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{Controller}
+The Controller's job is to control the execution of the processing
+stages on the nodes.  It is passed stages by the Localiser, and
+executes them on the appropriate nodes.  It must detect whether a node
+executing a processing stage has died, and re-execute the stage on an
+alternate node.
+The completed stages are reported back to the Scheduler.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{Pixel DB}
+\label{sec:pixeldb}
+The Pixel DB is responsible for storing and maintaining the location
+of pixel data in the IPP, including the raw images from the telescope,
+the master calibration images, the reference static-sky images, and
+any temporary image data products produced by the IPP.  It provides
+this information upon request to the Localiser.
+Note that this design assumes that the pixel data will be stored on
+the same nodes that will be doing the processing.
+The Pixel DB will be periodically ``published'' as the quality of the
+data is assured.  The external world will only have access to the
+published version of the Pixel DB.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{Nodes}
+The Nodes perform the grunt work of executing the processing stages as
+directed by the Controller.  When the processing stage has completed,
+they report back to the Controller.
+They may retrieve pixel data from OATS (the Summit) and write it to
+local disk when directed to do so by the Controller.  They also may
+access the Metadata DB to read configurations, weather information
+etc, and to write summary statistics etc.  They may also access the
+Object DB to read objects of interest, and to write objects from the
+processing stage.
+As they write products, the Nodes register with the Pixel DB that they
+have written the requested output (so that the Pixel DB is aware that
+the data has been written and is not merely scheduled to be written).
+The Nodes do not need to read from the Pixel DB, since everything
+(where to read input pixels from, where to write output pixels to) is
+specified by the Localiser.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{Object DB}
+The Object DB is a facility to store all of the information about
+astronomical objects, including individual measurements of objects on
+the images, the summary information about those objects, and reference
+object data\footnote{Note that this is (possibly) a separate entity
+from the object database being developed by SAIC.}.
+The Nodes, CSPs and MOPS may read objects from the Object DB, and the
+Nodes may write objects (either new objects or updates), and the CSPs
+and MOPS may write certain fields of objects (e.g., the external
+identifiers and class of object).
+The Object DB will be periodically ``published'' as the quality of the
+data is assured.  The external world will only have access to the
+published version of the Object DB.  The published version of the
+Object DB will likely be the DB being developed by SAIC.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{CSPs and MOPS}
+The Client Science Programs (CSPs) and the Moving Object Processing
+System (MOPS) are not a part of the IPP, but are external systems.  We
+include them here to show the required interfaces.
+The CSPs and MOPS may query the Pixel DB, the Metadata DB and the
+Object DB.  In addition, they may write certain fields to the object
+DB (e.g., the external identifiers and class of object) as they
+process objects, and they may retrieve pixel data from the Nodes.
+Since ``CSPs'' is a vague term, we now give some examples which may
+help to illustrate the functionality.
+One example of a CSP is a web front-end to retrieve (published) images
+and objects from the Pixel DB and Object DB.
+Another example would be a program interested in searching for
+transiting extrasolar planets.  Such a program may periodically poll
+the Metadata DB for new processed observations in its region of
+interest (such as the Galactic Plane), retrieve the object photometry
+of all high signal-to-noise stars in the processed observations, and
+attempt to identify a planetary transit in progress.
+Yet another example would be a Stationary Transient Object Pipeline,
+which would periodically poll the Metadata DB for new processed
+observations, and query the Object DB for variable sources which were
+identified twice (so that they are not moving objects).
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{Related/Connected components}
+The Pollster may be contained within the Scheduler (i.e., the
+Scheduler may initiate and/or schedule as a processing stage the
+Pollster), but this is not assumed to be so in this document; this
+decision is left to the implementation.
+The Localiser is strongly coupled to the Pixel DB, and throughout this
+document, these are both referred to as components of the ``IPP Pixel
+Server''.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{Responsibility}
+The IPP team will develop and have responsibility for maintaining
+these systems.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsubsection{Processing Stages}
+\label{sec:processingStages}
+We now consider the collection of IPP processing stages which are
+executed by the Controller on the Nodes.  We define a ``stage'' to be
+the largest complete task which may be performed in serial without
+interation between parallel threads.
+Depending on the particular stage, it may process individual images,
+collections of images, or on derived data products.  Because of the
+nature of the image data, many of the analysis stages can be run in
+parallel because, for example, the analysis of a chip in one image
+does not depend on the results from another chip.
+The data analysis stages are divided into several categories as follows:
+\begin{enumerate}
+\item Retrieval Stage --- pixel data are retrieved from OATS (the
+  Summit).
+\item Science Image Processing Stages
+  \begin{enumerate}
+  \item Phase 1: image processing preparation --- estimates
+    first-order astrometric and photometric solutions required to
+    process each major frame.
+  \item Phase 2: image reduction --- produces calibrated chips from
+    raw chips.
+  \item Phase 3: exposure analysis --- processes an FPA to produce
+    unified and consistent backgrounds, photometry and astrometry for
+    the component chips.
+  \item Phase 4: image combination --- processes sky cells overlapped
+    by a major frame.
+  \end{enumerate}
+\item Calibration Image Processing Stages
+  \begin{enumerate}
+  \item Cal 1: Basic master-detrend creation --- combination of simple
+    detrend images (e.g., bias, dome flat etc).
+  \item Cal 2: Sky-model/fringe-mode generation --- combination of
+    more-complicated detrend images (e.g., fringe, scattered light
+    etc).
+  \item Cal 3: Flat-field correction image creation --- analysis of
+    photometry from multiple dithered FPAs.
+  \end{enumerate}
+\item Calibration Test Processing Stage
+  \begin{enumerate}
+    \item CalTest 1: Detrend frame testing --- tests whether new
+      calibration frames are required.
+    \item CalTest 2: Photometric float correction testing --- tests
+      whether a new photometric flat correction is required.
+  \end{enumerate}
+\item Reference Catalog Processing Stages
+  \begin{enumerate}
+  \item Astrometry reference catalog generation --- processing of the
+    astrometric data to determine and apply a consistent global
+    solution.
+  \item Photometry reference catalog generation --- processing of the
+    photometric data to determine and apply a consistent global
+    solution.
+  \end{enumerate}
+\end{enumerate}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsubsection{Hardware Systems}
+The basic IPP hardware organization is shown in Figure~\ref{hardware}.
+The overall hardware organization, with a Detector subcluster and a
+Static Sky subcluster, is largely chosen to reduce the I/O load during
+the pre-reduction analysis of the raw science images.  In addition, we
+have specified distinct machines to maintain the object and metadata
+databases.  \tbd{This last aspect is largely theoretical until we have
+defined the details of these databases; it may be more appropriate
+depending on the eventual solutions to distribute these database
+elements across the Detector and Static Sky subclusters.}
+\begin{figure}
+The Image Server provides file pointers (in C), handles (in Perl), or
+file names corresponding to the instances of the storage objects.
+Image Server requires a file system which provides files in the local
+file system.  This may be done over many machines with a network file
+system such as NFS or GFS.  \tbd{select file system for IPP / test NFS
+vs GFS vs Mogile, etc}.
+The IPP Image Server provides the storage and access mechanisms, but
+it does not include any logic or information about the data.  The
+Image Server does not, e.g., monitor the age of images and delete them
+on some schedule.
+The IPP Image Server consists of the following components:
+\begin{itemize}
+\item Image Server storage hardware
+\item Image Server database
+\item Image Server daemon
+\item Image Server client APIs
+\end{itemize}
+\paragraph{IPP Image Server Client APIs}
+Clients interact with the IPP Image Server with a small number of C
+APIs (Bindings are also provided for Perl \tbr{and Python}).  The
+client commands are:
+\begin{itemize}
+\item {\tt new object}: create a new storage object in the Image
+  Server.  This function takes as input the file ID and returns a
+  C-style file pointer or a Perl file handle to the instance of the
+  storage object.  The arguments to the function include an optional
+  node name on which the new storage object must be located.  If this
+  target is not given, the Image Server places the new storage object
+  on an appropriate machine from the pool (least filled?  most data?
+  randomized?  the details need to be decided).
+\item {\tt open object}: open an instance of an existing storage
+  object, as identified by the file ID.  This function may also
+  specify the node on which the object should be opened (if an
+  instance of the object is not stored on that node, the function
+  returns an error).  On success, the function returns a file pointer.
+\item {\tt find object}: return a list of filenames in the UNIX name
+  space associated with the storage object identified by the given
+  file ID.  Since there are in general multiple instances for a given
+  storage object, this function returns the collection of all
+  available instances.  These may be freely opened by the client
+  server using the standard \code{fopen} functions.
+\item {\tt stat object}: returns status information about the
+  specified storage object, including the number of instances of the object.
+\item {\tt increment object count}: adds a new instance of the given
+  storage object.  The target node may be optionally specified,
+  otherwise an appropriate node is selected.
+\item {\tt decrement object count}: removes one of the instances of
+  the storage object.  The input parameters may optionally specify the
+  target machine to delete.
+\item {\tt delete object}: deletes all instances of the storage object
+  and sets the storage object status to {\tt deleted}.
+\end{itemize}
+\subsubsection{IPP Image Server Daemon}
+The Image Server client requests are mediated via the Image Server
+daemon.  Communication between the clients and the server is via
+\tbr{SOAP (or flat text commands)} implementing the commands above.
+\subsubsection{IPP Image Server Database}
+The IPP Image Server daemon uses a database to store the information
+about the data storage objects, their instances, and the available
+hardware resources.  A \tt{mysql} database engine is used to manage
+the database.  The database tables defined for the Image Server are
+listed in Table~\ref{ImageServerTables}, and their current contents
+are listed in Appendix A.  This database engine need not the same one
+as used for the IPP Metadata Database.
+%
+\begin{table}
 \begin{center}
+\resizebox{8cm}{!}{\includegraphics{pics/hardware}}
+\caption{ \label{hardware} IPP Hardware Organization}
+\caption{Image Server Database Tables\label{ImageServerTables}}
+\begin{tabular}{ll}
+\hline
+\hline
+{\bf Table Name} & {\bf Description} \\
+\hline
+\code{storage_object}  & all storage objects known to Image Server \\
+\code{instance}        & all instances of all storage objects \\
+\code{volume}          & data storage devices known to Image Server \\
+\hline
+\end{tabular}
 \end{center}
+\end{figure}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsection{Software Hierarchy}
+In order to facilitate testing and development, and to encourage
+flexibility, the IPP will be built in a layered fashion.  The lowest
+level functions will be written in C and collected together into a
+\PS{} library.  These library functions will be used to write more
+complex modules.  The modules will be written in C but will make use
+of the SWIG tool to make their functionality available within other
+frameworks.  In particular, the modules can be tied together with a
+simple framework (an `engine') or with detailed flow-control through
+the use of a high-level language such as Perl, Python, or Tcl
+employing the SWIG interfaces.  For the high-level functions in the
+operational system, the IPP will make use of \tbd{Python} as the
+scripting language to provide the required flow-control to tie the
+modules together.
+This approach satisfies the requirement that complicated low-level
+analysis steps run fast, while preserving flexibility for coding the
+high-level wrappers for which the speed requirements are not so
+stringent.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsubsection{External Libraries}
+\PS{} will employ several external libraries to save duplicating
+functionality that is already available.  These external libraries
+will be wrapped by the \PS{} Library, insulating the project from the
+implementation details of the external libraries.  Examples of the
+external libraries are FFTW and SLALib.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsubsection{\PS{} Library}
+The \PS{} Library will consist of C structures describing the basic
+data types needed by the IPP and C functions which perform the basic
+data manipulation operations.  Note that a subset of the library
+functions will be provided with SWIG interfaces as well to allow for
+their use in the creation of the processing stages.  Examples of the
+\PS{} Library are fourier transforms and transforming between pixel
+and celestial coordinates.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsubsection{Modules}
+The IPP analysis stages are broken down into modules which represent
+specific functional operations.  The modules will be written in C
+using the \PS{} Library functions and will be grouped into a \PS{}
+Module Library.  The modules will be provided with SWIG interfaces to
+all public APIs for their use in processing stages.  Examples of
+modules are overscan subtraction and image combination.  Some modules
+(e.g.\ find objects on an image) will be used by multiple stages.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsubsection{Stages}
+The major IPP processing tasks are organized into stages, which
+consist of multiple modules.  Each stage represents a collection of
+complex operations performed on a single data entity.  Each stage
+therefore represents the maximum amount of effort which can be
+performed in serial without interaction between parallel threads.  The
+stages will be written in \tbd{Python}, linking the modules together.
+Examples of stages are Phase 2 (detrend images) and Phase 4 (combine
+images from multiple telescopes and search for transients).
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsubsection{Orchestration}
+High-level components such as the Scheduler, the Controller and the
+Localiser are for process control.  As such, they shall be written in
+\tbd{Python} in order to maintain flexibility.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsection{System Interfaces}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\section{System Architectural Design}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsection{Architectural Components}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsubsection{Pollster}
+The Pollster simply polls OATS on a regular basis for metadata
+(including telescope exposures) which is not known by the IPP (i.e.,
+already written in the Metadata DB).  On the discovery of such metadata,
+it is written to the Metadata DB.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsubsection{Pixel Server}
+The IPP Pixel Server (IPS) is a repository for all image pixel data
+required by the IPP, and fulfills the roles of the Pixel DB
+(\S\ref{sec:pixeldb}) and the Localiser (\S\ref{sec:localiser}).  In
+addition, it also provides components for managing the distribution of
+data, and accessing the data.
+Images may reside in the IPS for different periods depending on their
+use and type.  Data stored by the IPS include the raw images, the
+calibration images, intermediate processing stage images as needed,
+final processed images, difference images, and image subsections,
+\tbd{along with the associated metadata}.  The IPS must retain images
+as long as they are needed, up to the lifetime of the project.  In
+order to achieve the I/O requirements, the IPS may maintain the pixel
+data distributed across the processor nodes in an organized fashion,
+i.e.\ associating specific machines with specific detectors.  The IPS
+interacts with the IPP Metadata Database to allow other systems or
+subsystems to identify the available images meeting specified
+criteria.  IPS specifications are described in the IPS subsystem
+specification.
+In addition to storing the pixel data, the IPS is responsible for
+acquiring new image data and metadata from the Summit Pixel Server and
+making it available for processing by the IPP System.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{IPP Pixel Server Components}
+The IPP Pixel Server (IPS) fulfills the roles of the Pixel DB
+(\S\ref{sec:pixeldb}) and the Localiser (\S\ref{sec:localiser}), and
+consists of the following components:
+\begin{enumerate}
+\item IPP Pixel Server Data Locality Optimizer (IPSDLO)
+\item IPP Pixel Server Database (IPSD)
+\item IPP Pixel Server Maintainance (IPSM)
+\item IPP Pixel Server I/O Library (IPSIOL)
+\end{enumerate}
+This assumes that the pixel data will be stored on the nodes.  Each
+image shall have a unique Universal Resource Identifier (URI) which
+specifies the location of the pixel data.  As an example, consider a
+cluster with cross-mounted disks --- in this case, the filename
+incorporating the full path would serve as the URI.
+The components of the IPS and their relation to other components (both
+within the IPS and without) are showin in Figure~\ref{fig:ips}.
+\begin{figure}
+\psfig{file=pics/IPS,width=15cm,angle=0}
+\caption{The components of the IPS.  In addition to the IPSDLO, IPSD
+and IPSM, the IPSIOL is also a component of the IPS; use of the IPSIOL
+is shown as dotted arrows in the interactions.  Note that the nodes use
+the IPSIOL to pass pixel data between each other.}
+\label{fig:ips}
+\end{figure}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subparagraph{IPP Pixel Server Data Locality Optimizer (IPPDLO)}
+Processing stages generated by the Scheduler are passed through the
+IPSDLO which does the following:
+\begin{enumerate}
+\item assigns tasks to specific nodes;
+\item identifies the URI of the required input data; and
+\item identifies the URI the output data should be written to.
+\end{enumerate}
+This allows the choice of processing node to be optimized so that it
+resides on the node which will process it, as well as allowing the
+output to be written to the node which requires it for the next
+processing stage.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subparagraph{IPP Pixel Server Database (IPSD)}
+\label{sec:ipsd}
+The IPSD maintains a database of URIs for the pixel data on the nodes.
+It should be able to return the URI of the pixel data given one of:
+\begin{enumerate}
+\item an exposure identifier and a chip identifier (raw and processed
+  pixel data from the telescope);
+\item a calibration identifier (detrend pixel data); and
+\item a sky cell identifier (summed static sky, reduced and difference
+  pixel data).
+\end{enumerate}
+The IPSD will also contain a history of data management commands and
+actions.
+\tbd{Is there a reason why this is not a part of the Metadata DB?}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subparagraph{IPP Pixel Server Maintenance (IPSM)}
+The IPSM initiates the execution of bulk data management processing
+stages.  It may have an automated component which, e.g., monitors the
+disk space on each of the nodes and redistributes them if they become
+unbalanced.  However, the main intent is that it is used by a human
+operator to reorgainise the data, e.g., after a new data optimisation
+plan has been formulated, or to delete old data.
+The IPSM passes processing stages to the Controller which executes
+them on the specified nodes.
+The IPSM allows four types of operation:
+\begin{itemize}
+\item Retrieve external data --- to manually trigger the copying of
+  external data (routine copying of the pixel data from OATS is
+  handled by the Scheduler).  The IPSM generates {\em retrieve data}
+  stages which are passed to the Controller for execution.
+\item Delete data --- to delete old data.  The IPSM looks up the
+  location of the data in the IPSD and generates {\em delete data}
+  stages which are passed to the Controller for execution.
+\item Replicate data --- to backup and rearrange data.  The IPSM
+  generates {\em copy data} stages which are passed to the Controller
+  for execution.  Note that this mode differs from the ``copy external
+  data'' mode in that it copies data already within the IPS.
+\item Move data --- to reorganise storage.  The IPSM executes a
+  replication followed by a deletion.
+\end{itemize}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subparagraph{IPP Pixel Server I/O Library (IPSIOL)}
+The IPSIOL provides a mechanism for reading and writing pixel data to
+the IPS.  The existence of the IPSIOL insulates the processing stages
+from the details of how the pixel data are stored (i.e., the
+processing stages need not worry whether the data is stored locally or
+remotely).  It will generally be used on the nodes and the IPSDLO,
+although other components will also make use of it.
+The IPSIOL will be able to:
+\begin{itemize}
+\item Open a file specified by a URI --- it may simply open the file
+  if it exists on the particular node, or it may retrieve the file
+  over the network.
+\item Write a file specified by a URI --- it may simply write the file
+  if it exists on the particular node, or it may copy the file over
+  the network.  It should also register with the IPSD that a file
+  specified by a URI has been written.
+\item Delete a file specified by a URI --- it may simply delete the
+  file if it exists on the particular node, or it may delete the file
+  over the network.
+\item Interface with the IPSD to return a URI given one of the
+  identifiers in \S\ref{sec:ipsd}.
+\end{itemize}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{Pixel Data Flow Examples}
+For examples of the operation of the IPS, below we sketch out the
+intended sequence of events for common operations.
+Reads during processing:
+\begin{enumerate}
+\item A processing stage has been passed (from the Scheduler) the URI
+  for an image that it needs to load into memory.
+\item The processing stage uses the IPSIOL to open the image.
+\item The processing stage reads the image into local memory in the
+  usual manner.
+\item The processing stage closes the image using the IPSIOL.
+\end{enumerate}
+Writes during processing:
+\begin{enumerate}
+\item A processing stage has been passed (from the Scheduler) the URI
+  for an image that needs to be saved, e.g., a subtracted image.
+\item The processing stage uses the IPSIOL to open the image.
+\item The processing stage writes the image in the usual manner.
+\item The processing stage closes the image using the IPSIOL.
+\end{enumerate}
+Note how the IPSIOL has insulated the processing stage from the details
+of the reading and writing.
+Maintenance:
+\begin{enumerate}
+\item A human operator decides that all the pixel data for chip 12
+  should be stored on node 3.
+\item Operator plugs this into the IPSM.
+\item The IPSM queries the IPSD using the IPSIOL.
+\item The IPSD returns the URIs for all the pixel data for chip 12.
+\item The IPSM generates processing tasks to be executed on the nodes
+  that will copy the data from the old URIs to a new URI which
+  specifies node 3.
+\item The IPSM generates processing tasks to be executed on the nodes
+  that deletes the data pointed to by the old URIs.
+\item The IPSM reports success to the operator.
+\end{enumerate}
+Client Science Pipelines:
+\begin{enumerate}
+\item A CSP wants some pixel data.
+\item The CSP queries the IPSD using the IPSIOL (e.g., asking for a
+  particular exposure or sky cell).
+\item The IPSD returns the URI for the pixel data.
+\item The CSP opens the image using the IPSIOL and the URI.
+\item The CSP reads the pixel data into memory in the usual manner.
+\item The CSP closes the image using the IPSIOL.
+\end{enumerate}
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\end{table}
+\subsubsection{IPP Image Server Storage Hardware}
+The IPP Image Server manages data across a collection of computers and
+possibly on multiple storage devices on those computer nodes.  The
+Image Server maintains a table of the available data volumes.  The
+Image Server tracks information about each volume such as the total
+capacity, the current capacity, the association between computer and
+data volume.
+\paragraph{IPP Image Server Maintenance Tools}
+The IPP Image Server provides a collection of administration tools
+which allow for maintainence.  These are operations which may be
+automatically scheduled for the IPP or which may be initiated by a
+human via a command-shell interface.  The maintainence functions
+include migrating data between nodes to rebalance the available space
+(this would only occur for instances which have not been placed on a
+specific node by the client API).  Other functions include checking
+for file corruption, which involves sweeping all files on a data
+volume and comparing the calculated file checksum to the currently
+recorded value.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 …
 manner given the capabilities of the science pipelines.
+\paragraph{Pollster}
+The Pollster is a program that polls OATS at regular intervals for the
+existence of observations not contained in the Metadata DB.  New
+weather and image metadata are written to the Metadata DB.
+There is no reason why this architectural component cannot be
+contained within another (such as the Scheduler), but it is shown here
+as separate for simplicity.
+A polling model is adopted so that OATS' interface may be kept as
+simple as possible --- OATS should not be concerned with whether the
+IPP has received notifications.  Under this polling model, it is
+specifically the responsibility of the IPP to retrieve from OATS the
+metadata that is not not already in the Metadata DB.
+\subsubsection{Pollster}
+The Pollster simply polls OATS on a regular basis for metadata
+(including telescope exposures) which is not known by the IPP (i.e.,
+already written in the Metadata DB).  On the discovery of such metadata,
+it is written to the Metadata DB.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 …
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \subsection{Processing Stages}
+\subsection{Analysis Tasks and Stages}
 In this section, we review the processing stages which are executed on
 …
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\subsection{Software Hierarchy}
+In order to facilitate testing and development, and to encourage
+flexibility, the IPP will be built in a layered fashion.  The lowest
+level functions will be written in C and collected together into a
+\PS{} library.  These library functions will be used to write more
+complex modules.  The modules will be written in C but will make use
+of the SWIG tool to make their functionality available within other
+frameworks.  In particular, the modules can be tied together with a
+simple framework (an `engine') or with detailed flow-control through
+the use of a high-level language such as Perl, Python, or Tcl
+employing the SWIG interfaces.  For the high-level functions in the
+operational system, the IPP will make use of \tbd{Python} as the
+scripting language to provide the required flow-control to tie the
+modules together.
+This approach satisfies the requirement that complicated low-level
+analysis steps run fast, while preserving flexibility for coding the
+high-level wrappers for which the speed requirements are not so
+stringent.
+\subsubsection{External Libraries}
+\PS{} will employ several external libraries to save duplicating
+functionality that is already available.  These external libraries
+will be wrapped by the \PS{} Library, insulating the project from the
+implementation details of the external libraries.  Examples of the
+external libraries are FFTW and SLALib.
+\subsubsection{\PS{} Library}
+The \PS{} Library will consist of C structures describing the basic
+data types needed by the IPP and C functions which perform the basic
+data manipulation operations.  Note that a subset of the library
+functions will be provided with SWIG interfaces as well to allow for
+their use in the creation of the processing stages.  Examples of the
+\PS{} Library are fourier transforms and transforming between pixel
+and celestial coordinates.
+\subsubsection{Modules}
+The IPP analysis stages are broken down into modules which represent
+specific functional operations.  The modules will be written in C
+using the \PS{} Library functions and will be grouped into a \PS{}
+Module Library.  The modules will be provided with SWIG interfaces to
+all public APIs for their use in processing stages.  Examples of
+modules are overscan subtraction and image combination.  Some modules
+(e.g.\ find objects on an image) will be used by multiple stages.
+\subsubsection{Stages}
+The major IPP processing tasks are organized into stages, which
+consist of multiple modules.  Each stage represents a collection of
+complex operations performed on a single data entity.  Each stage
+therefore represents the maximum amount of effort which can be
+performed in serial without interaction between parallel threads.  The
+stages will be written in \tbd{Python}, linking the modules together.
+Examples of stages are Phase 2 (detrend images) and Phase 4 (combine
+images from multiple telescopes and search for transients).
 \subsection{Modules}
 …
 \section{Appendices}
+\subsection{Image Server Database Tables}
+\begin{table}
+\begin{center}
+\caption{Storage Object Table Contents\label{ImageServerTables:SO}}
+\begin{tabular}{ll}
+\hline
+\hline
+{\bf Column Name} & {\bf Datatype} & {\bf Description} \\
+\hline
+\code{so_id}      & integer        & internal storage object identifier \\
+\code{ext_id}     & string         & external storage object identifier (file ID) \\
+\code{comment}    & string         & user description of object \\
+\code{epoch}      & time/date      & last date of access \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+\begin{table}
+\begin{center}
+\caption{Instance Table Contents\label{ImageServerTables:INT}}
+\begin{tabular}{ll}
+\hline
+\hline
+{\bf Column Name} & {\bf Datatype} & {\bf Description} \\
+\hline
+\code{ins_id}     & integer        & internal instance identifier \\
+\code{so_id}      & integer        & key to storage object table \\
+\code{uri}        & string         & location in hardware collection \\
+\code{sha1sum}    & string         & checksum information \\
+\code{assigned_location} & boolean & is location user-specified? \\
+\code{epoch}      & time/date      & last date of access \\
+\code{atime}      & time/date      & last date of access \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+\begin{table}
+\begin{center}
+\caption{Volume Table Contents\label{ImageServerTables:VOL}}
+\begin{tabular}{ll}
+\hline
+\hline
+{\bf Column Name} & {\bf Datatype} & {\bf Description} \\
+\hline
+\code{vol_id}     & integer        & internal volume identifier \\
+\code{uri}        & string         & node name? \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
 \bibliographystyle{plain}

Note: See TracChangeset for help on using the changeset viewer.

Context Navigation

Changeset 2114 for trunk/doc/design/ippSDRS.tex

Legend:

trunk/doc/design/ippSDRS.tex

Download in other formats: