Changeset 413

Timestamp:

Apr 9, 2004, 3:14:49 PM (22 years ago)

Author:

eugene

Message:

major rewrite of pixel server, added sections for metadata.db,
object.db, controller.db. the scheduler still needs some work.

File:

• trunk/doc/design/specs.tex (modified) (3 diffs)

trunk/doc/design/specs.tex

@@ -1 @@
-%%% $Id: specs.tex,v 1.1 2004-04-09 02:25:41 eugene Exp $
+%%% $Id: specs.tex,v 1.2 2004-04-10 01:14:49 eugene Exp $
 \documentclass[panstarrs]{panstarrs}
 
@@ -53 +53 @@
 Open Issues and TBDs in this document are marked \tbd{in bold, red
 with surrounding square brackets}.
+
+All timing measurements are to execution time as measured on a
+\tbd{Reference Pan-Starrs Computation Node} and assumed to be not
+limited by network bandwidth.
+
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -279 +284 @@
 \paragraph{Pixel Server}
 
-\begin{itemize}
-\item $T_{\rm min}$ is the minimum time between exposures.  $T_{\rm min}$ is
-assumed to always be $\ge 30s$.
-\item All timing measurements are to execution time as measured on a
-\tbd{Reference Pan-Starrs Computation Node} and assumed to be not limited
-by network bandwidth.
-\end{itemize}
-
-\begin{enumerate}
-\item IPP Pixel Data Scheduler
-\item IPP Pixel Data Locality Optimizer
-\item IPP Pixel Data Database
-\item IPP Pixel Data Retrieval Agent
-\item IPP Pixel Data Query Library
-\item IPP Pixel Data I/O Library
-\end{enumerate}
-
-\subparagraph{IPP Pixel Data Scheduler (IPP-PDS)}
-
-{\it Inputs}
-
-\begin{itemize}
-\item Accepts an XML document containing the type of operation and image
-meta-data if applicable. 
-\end{itemize}
-
-The input document is one of the follow classes of message.
-
-\begin{itemize}
-\item {\em new data notification}
-\item {\em move data request}
-\item {\em copy data request}
-\item {\em delete data request}
-\end{itemize}
-
-\tbd{The format of this XML doc is TBD.}
-\tbd{The application layer transport protocol is TBD.}
-
-{\it Outputs}
-
-Outputs an XML document containing one or more {\em data managements tasks}.
-This document is to be passed through all registered filters that match it's
-task type.  The document is then sent to the IPP Controller.
-
-The IPP-PDS can fulfill all of it's modes of operations by generating just two
-types of {\em data management tasks}.
-
-\begin{itemize}
-\item {\em retrieve data task}
-\item {\em delete data task}
-\end{itemize}
-
-Seemingly there should be a third task type {\em move data} but this can be
-broken down into a {\em retrieve data} task and a {\em delete data}.  This
-strategy has the added benefit of adding atomicity to the operation.
-
-\tbd{The format of this XML doc is TBD.}
-\tbd{The application layer transport protocol is TBD.}
-
-{\it Configuration}
-
-A configuration file defining at least the address of IPP Controller and the
-IPP Pixel Data Database connection string.
-
-\tbd{The format of this file is TDB.}
-
-{\it Performance}
-
-The IPP-PDS must be able to concurrently:
-
-\begin{itemize}
-\item receive and process 260 {\em new data notifications} in less time
-then ${T_{\rm min}}$.
-\item generate, filter, and transmit 260 {\em data management tasks} in less
-time then ${T_{\rm min}}$.
-\end{itemize}
-
-\subparagraph{IPP Pixel Data Locality Optimizer (IPP-PDLO)}
-
-{\it Inputs}
-
-\begin{itemize}
-\item Accepts an XML document containing one or {\em data management tasks}.
-\end{itemize}
-
-\tbd{The format of this XML doc is TBD.}
-\tbd{The I/O protocol is TBD (possibly stdin/stdout).}
-
-{\it Outputs}
-
-\begin{itemize}
-\item Outputs an XML document containing one or {\em data management tasks}.
-\end{itemize}
-
-\tbd{The format of this XML doc is TBD.}
-\tbd{The I/O protocol is TBD (possibly stdin/stdout).}
-
-\subparagraph{Configuration}
-
-A configuration file defining what sort of optimization should be done and
-the IPP Pixel Data Database connection string.
-
-\tbd{The format of this file is TDB.}
-
-\subparagraph{Performance}
-
-\begin{itemize}
-\item The time spent in this filter should be added to the execution timing of
-the IPP Pixel Data Scheduler and can not cause it to exceed $T_{\rm min}$. 
-\end{itemize}
-
-\subparagraph{IPP Pixel Data Database (IPP-PDD)}
-
-The IPP Pixel Data Database will maintain a record of {\em new data notifications}
-received from the Summit Pixel Server, the storage location of downloaded but
-unreduced image data, the storage location of reduced image data, the storage
-location of stacked image data, and the storage location of calibration data.
-\tbd{In addition to the storage location(s) of image data some or all of it's
-associated meta-data will contained in the IPP-PDD}
-
-{\it Interfaces}
-
-\begin{itemize}
-\item Native database bindings
-\item ODBC
-\end{itemize}
-
-{\it Configuration}
-
-\tbd{Database scheme is TBD.}
-
-{\it Functionality}
-
-\begin{itemize}
-\item Linux $\ge$ 2.4.x and Glibc $\ge$ 2.3
-\item SQL Syntax $\ge$ SQL-99
-\item native bindings for C and Perl
-\item databases size $\ge 1TB$ 
-\item ODBC $\ge 3.5$
-\item basic stored procedure support
-\item hot backups
-\item replication
-\end{itemize}
-
-{\it Performance}
-
-\begin{itemize}
-\item Process $> 30 \times 260$ select, insert, update, or delete queries in less time then
-${T_{\rm min}}$.
-\end{itemize}
-
-\subparagraph{IPP Pixel Data Retrieval Agent (IPP-PDRA)}
-
-One instance of the IPP Pixel Data retrieval Agent is spawned per 
-{\em data management task} that needs to be serviced.  The IPP Pixel Data I/O
-Library will be used to retrieve a URI into memory and to write data from
-memory to a specified URI.  \tbd{File lock management may or may not be
-necessary within this component.}
-
-{\it Inputs}
-
-\begin{itemize}
-\item Accepts an XML document containing one or {\em data management tasks}.
-\end{itemize}
-
-\tbd{The format of this XML doc is TBD.}
-\tbd{The I/O protocol is TBD (possibly stdin/stdout).}
-
-{\it Outputs}
-
-\begin{itemize}
-\item Returns an XML document containing one or completed {\em data management tasks}.
-\end{itemize}
-
-\tbd{The format of this XML doc is TBD.}
-\tbd{The I/O protocol is TBD (possibly stdin/stdout).}
-
-{\it Configuration}
-
-A configuration file defining the address of the IPP Controller.
-
-\tbd{The format of this file is TDB.}
-
-{\it Performance}
-
-\begin{itemize}
-\item Must capable of fully saturating a $1Gb/s$ network connection via the
-IPP-PDIOL.
-\end{itemize}
-
-\subparagraph{IPP Pixel Data Query Library (IPP-PDQL)}
-
-The IPP Pixel Data Query Library must hide all SQL details from the caller.
-
-{\it Interfaces}
-
-\begin{itemize}
-\item C API
-\item Perl (XSub of the C API) API
-\end{itemize}
-
-{\it Configuration}
-
-A configuration file with the database connection string.
-
-\tbd{The format of this file is TDB.}
-
-{\it Query Types}
-
-The IPP-PDQL only supports simple database queries.
-
-\tbd{The specific queries supported is TDB.}
-
-{\it Performance}
-
-\begin{itemize}
-\item Process $> 30$ select, insert, update, or delete queries in less time then
-${T_{\rm min}}$.
-\end{itemize}
-
-\subparagraph{IPP Pixel Data I/O Library (IPP-PDIOL)}
-
-The IPP Pixel Data I/O Library retrieves data from or writes data to \cite{uri}s.
-Must be able to download multiple segments of a file simultaneously if the 
-transport protocol supports it.  Similar to the \cite{proz} download accelerator.
-Must be able to handle file locking issues if the transport protocol supports it.
-The HTTP/WEBDAV protocol should be implement with the \cite{neon} library. 
-
-{\it Interfaces}
-
-\begin{itemize}
-\item C API
-\item Perl (XSub of the C API) API
-\end{itemize}
-
-{\it Configuration}
-
-A configuration file defining the optional behaviors for the protocols that
-have optional features.
-
-\tbd{The format of this file is TDB.}
-
-{\it Protocols}
-
-Must support at least the following protocols:
-
-\begin{itemize}
-\item \cite{http}
-\item \cite{http} w/\cite{webdav}
-\item \cite{ftp}
-\item \cite{rsync}
-\item file
-\end{itemize}
-
-{\it Performance}
-
-\begin{itemize}
-\item Must capable of fully saturating a $1Gb/s$ network connection.
-\end{itemize}
-
-\subparagraph{Pixel Data Flow}
-
-\subparagraph{Bandwidth}
-
-\begin{enumerate}
-\item Summit Pixel Server(s)
-\begin{itemize}
-\item $n \times$ \tbd{TBD}
-\begin{itemize}
-\item $\frac{2 \times 2.4Gb/s}{n}$ 
-\end{itemize}
-\end{itemize}
-\item Summit Core Switch
-\begin{itemize}
-\item Cisco 65xx
-\begin{itemize}
-\item $2 \times 2.4Gb/s$ 
-\end{itemize}
-\end{itemize}
-\item Summit Border Router
-\begin{itemize}
-\item Cisco 76xx
-\begin{itemize}
-\item $1 \times 2.4Gb/s$
-\end{itemize}
-\end{itemize}
-\item Summit $\Longleftrightarrow$ Data Center connection (WAN link)
-\begin{itemize}
-\item $2 \times 1Gb/s$ Ethernet (over ATM/Sonet) or OC-48 Sonet
-\begin{itemize}
-\item $1 \times 2.4Gb/s$
-\end{itemize}
-\end{itemize}
-\item Data Center Border Router
-\begin{itemize}
-\item Cisco 76xx
-\begin{itemize}
-\item $1 \times 2.4Gb/s$
-\end{itemize}
-\end{itemize}
-\item IPP Cluster Core Switch
-\begin{itemize}
-\item Cisco 65xx
-\begin{itemize}
-\item $48Gb/s$
-\end{itemize}
-\end{itemize}
-\item IPP Cluster Nodes
-\begin{itemize}
-\item $240 \times$ \tbd{Reference Pan-Starrs Computation Node}
-\begin{itemize}
-\item $\frac{48Gb/s}{240}$
-\end{itemize}
-\end{itemize}
-\item Extended Network
-\begin{itemize}
-\item Cisco 65xx
-\begin{itemize}
-\item \tbd{TBD}
-\end{itemize}
-\end{itemize}
-\item Static Sky DB, other components, etc.
-\begin{itemize}
-\item \tbd{TBD}
-\begin{itemize}
-\item \tbd{TBD}
-\end{itemize}
-\end{itemize}
-\end{enumerate}
-
-\begin{figure}
-\begin{center}
-% \resizebox{!}{20cm}{\includegraphics{pixel_wan.epsi}}
-\caption{ \label{acquisition} Pixel Data Flow: Bandwidth}
-\end{center}
-\end{figure}
-\pagebreak
-
-\subparagraph{Bandwidth Estimates}
-
-{\it Assumptions}
-
-\begin{itemize}
-\item $T_{\rm min} = 30s$
-\end{itemize}
-
-{\it Exposure with overclocks in integer}
-
-{\it Storage Size}
-$$2bytes \times (4096^2pixels \times 1.125overclocks) \times 240otas = 72477573120b$$
-
-{\it Bandwidth Requirement}
-$$\frac{72477573120b}{T_{\rm min}} = 2415919104b/s$$
-
-{\it Exposure in float}
-
-{\it Storage Size}
-$$4bytes \times 4096^2pixels \times 240otas = 128849018880b$$
-
-{\it Bandwidth Requirement}
-$$\frac{128849018880b}{T_{\rm min}} = 4294967296b/s$$
-
-{\it Stacked exposure in float}
-
-{\it Storage Size}
-$$4bytes \times 4096^2pixels \times 60otas = 32212254720b$$
-
-{\it Bandwidth Requirement}
-$$\frac{32212254720b}{T_{\rm min}} = 1073741824b/s$$
-
-{\it Full calibration set in float}
-
-$$(1 \times debias, 1 \times dark, 1 \times flat, 2 \times fringe, 2 \times sky)$$
-
-{\it Storage Size}
-$$7 \times (exposure\ in\ float) = 901943132160b$$
-
-{\it Bandwidth Requirement}
-$$\frac{901943132160b}{T_{\rm min}} = 30064771072b/s$$
-
-{\it Aggregate Bandwidth Requirement}
-
-\begin{center}
-% \begin{tabular}{>{$}l<{$}>{$}r<{$}l}
-\begin{tabular}{lrl}
-[phase 2]&&\\
- & 2415919104b/s & summit $\rightarrow$ disk (exposure)\\
-+& 2415919104b/s & non-local disk $\rightarrow$ memory (exposure)\\
-+& 30064771072b/s& non-local disk $\rightarrow$ memory (calibration)\\
-+& 4294967296b/s & memory $\rightarrow$ non-local disk (reduced)\\
-\cline{1-2}
- & 39,191,576,576b/s &\\
-
-[phase 4]&&\\
- & 4294967296b/s & non-local disk $\rightarrow$ memory (reduced)\\
-+& 1073741824b/s & non-local disk $\rightarrow$ memory (best)\\
-+& 1073741824b/s & non-local disk $\rightarrow$ memory (working)\\
-+& 1073741824b/s & memory $\rightarrow$ non-local disk (diff)\\
-+& 1073741824b/s & memory $\rightarrow$ non-local disk (working)\\
-\cline{1-2}
- & 8,589,934,592b/s &\\
-
-[total]&&\\
- & 39191576576b/s & [phase 2] total\\
-+& 8589934592b/s & [phase 4] total\\
-\cline{1-2}
- & 47,781,511,168b/s & $\sim48Gb/s$
-\end{tabular}
-\end{center}
-
-\subparagraph{IPP Pixel Data Database Query Estimates}
-
-{\it Assumptions}
-
-\begin{itemize}
-\item There is no caching of query results.
-\end{itemize}
-
-{\it Acquisition}
-
-\begin{itemize}
-\item select new data notification from IPP Data Scheduler
-\item insert new data notification from IPP Data Scheduler
-\item select from IPP Pixel Data Locality Optimizer
-\item select new data notification from IPP Pixel Data Scheduler
-\item update new data notification from IPP Pixel Data Scheduler
-\item select data available from IPP Pixel Data Scheduler
-\item insert data available from IPP Pixel Data Scheduler
-\end{itemize}
-
-{\it Phase 2}
-
-\begin{itemize}
-\item select data available from IPP Scheduler
-\item select $\times 7$ calibration data from IPP Image Agent
-\item select data available from IPP Scheduler
-\item update data available from IPP Scheduler
-\item select reduced data available from IPP Scheduler
-\item insert reduced data available from IPP Scheduler
-\end{itemize}
-
-{\it yPhase 4}
-
-\begin{itemize}
-\item select reduced data available from IPP Scheduler
-\item select $\times 2$ stacked data from IPP Image Agent
-\item select reduced data available from IPP Scheduler
-\item update reduced data available from IPP Scheduler
-\item select stacked data available from IPP Scheduler
-\item insert stacked data available from IPP Scheduler
-\item select difference data available from IPP Scheduler
-\item insert difference data available from IPP Scheduler
-\end{itemize}
-
-\begin{verbatim}
-\bibitem[Link aggregation]{aggregation}
-http://cisco.com/en/US/products/hw/switches/ps708/products\_configuration\_guide\_chapter09186a008019f011.html
-\bibitem[ProZilla]{proz}
-http://prozilla.genesys.ro/
-\bibitem[neon]{neon}
-http://www.webdav.org/neon/
-\bibitem[Uniform Resource Identifiers (URI)]{uri}
-ftp://ftp.rfc-editor.org/in-notes/rfc2396.txt
-\bibitem[HTTP]{http}
-ftp://ftp.rfc-editor.org/in-notes/rfc2616.txt
-\bibitem[WEBDAV]{webdav}
-ftp://ftp.rfc-editor.org/in-notes/rfc2518.txt\\
-ftp://ftp.rfc-editor.org/in-notes/rfc3253.txt\\
-ftp://ftp.rfc-editor.org/in-notes/rfc3648.txt
-\bibitem[FTP]{ftp}
-ftp://ftp.rfc-editor.org/in-notes/rfc454.txt
-\bibitem[rsync]{rsync}
-http://rsync.samba.org/
-\end{verbatim}
+The IPP Pixel Server \tbd{rename as Image Server?} is a large data
+store for all images used by the IPP.  The Pixel Server is required to
+store all of the images needed by the IPP for the length of time they
+are required; total data volume is specified in detail in the hardware
+summary, but is in the vicinity of \tbd{700 GB}.
+
+The IPP Pixel Server must maintain a record of all images currently
+available in the repository \tbd{and all no longer available}.  This
+record shall include the image name, location (which machine), the
+state of the image (available, deleted), the image size, the image
+type, and the existence and location of secondary copies of the
+image.  This information need not include other metadata such as the
+image summary statistics or the state of the image processing for the
+image.
+
+The IPP Pixel Server shall store images as FITS files on disk.  Raw
+images from the telescope shall be stored as individual OTA images for
+each file, with multiple Cell images per file as well as video
+sequences from the guide stars.  Images of the Static Sky shall be
+stored in the form of \tbd{triangular segments} to minimize the total
+data volume and pixel overlap. 
+
+The IPP Pixel Server shall distribute images across a cluster of
+machines.  The IPP Pixel Server shall be capable of honoring requests
+to store an image on a specific machine.  If such a request cannot be
+honored, the IPP Pixel Server shall select an appropriate machine and
+notify the requesting agent of the new locations.  The IPP Pixel
+Server shall provide a mechanism to maintain multiple (at least two)
+copies of a single known image.
+
+The IPP Pixel Server must interface with other subsystems of the IPP.
+It must provide an interface to other IPP subsystems to identify the
+image location (the computer on which it resides).  It must provide a
+mechanism to serve a specified image to another IPP or Pan-STARRS
+subsystem.  It must provide a mechanism for deletion of images in the
+Pixel Server.  It must have a mechanism to accept or retrieve an image
+from another Pan-STARRS subsystem, in particular OATS.  Communication
+of messages between the IPP Pixel Server and other subsystem shall be
+via \tbd{XML messages} passed via \tbd{some transport}.
+
+The IPP Pixel Server must accept images at the telescope maximum rate
+of 1 full-camera image every 30 seconds.  The IPP Pixel Server must
+accept notifications and process retrievals at a rate of 128 raw OTAs
+per 60 seconds.
+
+\tbd{O/S, language, SQL, ODBC requirements?}
+
+\tbd{hardware requirements?}
+
+\tbd{communication protocols?} 
+
+\paragraph{P\&A Database}
+
+The IPP requires a mechanism to store data related to astronomical
+objects derived from various sources with a variety of associations.
+The PnA (Photometry and Astrometry) Database serves this function.
+The PnA Database deals with two related concepts: {\em objects} and
+{\em detections}.  The objects are descriptions of astronomical
+objects while the detections are the specific measurements of those
+objects on an image.  A collection of {\em detections} may be used to
+derive average quantities which describe a particular {\em object}.
+
+The PnA Database must store the collections of detections which were
+derived from specific images from any of the analysis stages.  It must
+be possible to determine and locate (perhaps via interactions with the
+pixel server) the image from which a specific detection was derved.
+It must also be possible to extract all detections derived from a
+specific images.  These associations must include descriptive
+information including the coordinates of the detection on the image.
+
+The PnA Database must provide a mechanism to associate together
+multiple detections of a specific object.  Several major classes of
+objects will be present, each of which must be handled correctly.
+
+First, the distant stars will have nearly fixed locations relative to
+other nearby stars, with only small deviations for individual
+measurements.  The association between multiple detections of such
+objects must be made on the basis of their coincident positions.  The
+PnA Database must be able to determine the average position of the
+object and the deviations of the individual detections from that
+average.  
+
+Second, solar system objects do not have a fixed location and
+detections of such objects must associated on the basis of their
+coincidence with the orbit of the objects.  The PnA Database must be
+able to associate detections with the orbits of known objects.  The
+determination of this association is the responsibility of the MOPS
+and must be communicated to the IPP PnA Database on \tbd{some
+  timescale}.  The PnD Database must be able to retrieve the
+detections associated with the object and to provide the object
+associated with the specific detections.  This association must
+include descriptive information such as the offset of the detection
+from the predicted location of the detection based on the orbit.
+
+Third, stars in the general vicinity of the solar system fall in
+between these first two classes of objects.  Their proper motion and
+parallax response is significant enough ($>1\asec$ in 10 years) that
+they are not well-described by an average location and a collection of
+offsets.  These objects must be described by a distance and a proper
+motion vector.  The PnA Database must be able to find and associate
+detections of objects for which either of the parallax or the proper
+motion are substantial.  
+
+Fourth, many detections, especially in their initial states, will not
+be associated with a specific astronomical object of any of the above
+classes and should be treated as orphans.  Some of these will be
+suprious (not represent real objects), some will be from solar system
+objects for which orbits are not yet determined, some will be from
+faint stars near the detection limits, some will be from short-term
+transients which have only been detected once.  The PnA Database must
+be able to carry these detections until they have been associated with
+one of the objects above.  It must be possible to migrate individual
+detections associated with an astronomical object back to the orphan
+state.  
+
+For every object, and all orphaned detections, it must be possible to
+determine the images for which the coordinates were included but for
+which no detection was made.  The minimum set of information which
+must be carried for these non-detections is the image and the
+associated object or orphan.
+
+The PnA Database must store the relationships between various
+photometric systems and, in some cases, the evolution of that
+relationship.  It must be possible, given a determined set of
+calibrations, to convert between the measured instrumental magnitude
+of a detection with a specific filter, detector, and telescope, and at
+particular time and the implied magnitude in the average Pan-STARRS
+magnitude systems.  It must also be possible, given the magnitudes of
+an object in one system to convert those to the magnitudes in another
+system; an example of such a conversion is between the average
+Pan-STARRS filter systems and the various reference systems
+appropriate for those filters.
+
+The PnA Database must provide interfaces to extract lists of objects
+and detections based on various query parameters.  It must be possible
+to extract all detections associated with a specific object, all
+non-detections of that object (or orphan) and summary statistics from
+these collections.  It must be possible to extract all objects or
+detections within specified spatial regions including regions bounded
+by great circles (RA,DEC; GLAT,GLON; ELAT,ELON) and regions described
+by a location and a search radius.  It must be possible to extract the
+image parameters associated with a specific detection including image
+coordinates of the detection, exposure time, time and date of the
+detection, etc.
+
+\tbd{volume requirements}
+
+\tbd{speed / access requirements}
+
+\paragraph{Metadata Database}
+
+The IPP requires a Metadata Database to store and provide access to
+metadata of various types and from various sources.  Metadata in the
+context of the IPP represents all data which is not included in the
+two data stores discussed above (Images and Detection/Objects).
+Metadata is generated at the telescope and during the various analysis
+stages
+
+The Metadata Database must store and provide metadata for all raw
+images, for processed images, for the calibration images (both raw and
+master), for the extracted object lists.  Metadata describing the
+environmental conditions at the telescope must also be stored and
+provided as needed.  
+
+If analysis results are exchanged via the metadata database, it must
+provide access to the queried data on timescales of $<2 sec$ to avoid
+slowing down the analysis systems. 
+
+\tbd{volume requirements}
+
+\tbd{does the description of images belong in the Metadata database or
+  in the Pixel / Image Server?}
+
+\tbd{queries}
+
+\paragraph{Configuration Database -- a subset of the metadata database?}
+
+The IPP requires a Configuration Database to store and provide access to
+information about the IPP itself.  Examples of data in the
+configuration database include the default parameters for the various
+analysis programs, the description of the computing environment, the
+process status information, etc.  
+
+\paragraph{Controller}
+
+The IPP uses a collection of computers to store and process images and
+to manipulate collections of detections.  These computers perform any
+of a large number of analysis stages or other processing tasks without
+significant interprocess communication.  It is necessary to have a
+mechanism which initiates computing tasks on the different computers,
+which monitors the tasks as they are executed, which handles the
+output and the errors from these tasks, and which reacts to the
+failure of any of the computing nodes.  The system responsible for the
+tasks in the IPP is the Controller.
+
+The Controller must interact with the collection of computers under
+its management and with other subsystems in the IPP.  The controller
+must accept a variety of inputs from other subsystems, described
+below, and respond accordingly.  The controller must also provide
+information to other subsystems on demand.
+
+Computers managed by the controller are allowed to be in one of
+several states, and the controller must interact with it in an
+appropriate way for each of those states.  A computer may be {\tt
+alive}, {\tt dead} or {\tt off}.  If the computer is {\tt alive}, it
+responds to commands from the controller and may be used for tasks
+subject to other constraints.  If it is {\tt dead}, the computer is
+not responsive and should not be used for executing tasks.  The
+controller must identify computers which have died and occasionally
+test them to see if they are {\tt alive} again.  Computers which are
+{\tt off} are not available for tests and should not be tested.
+Computers may be set to the {\tt off} or {\tt dead} states by external
+subsystems; it is the responsibility of the Controller to move a
+computer to the {\tt alive} state if possible.  
+
+Computers which are in the {\tt alive} state may be in one of two
+modes: {\tt busy} and {\tt free}.  A computer which is {\tt busy}
+currently has a task assigned to it.  The controller may only assign
+one task to one computer at a time\footnote{a physical piece of
+hardware may be defined to the Controller as multiple computers to
+allow multi-processor nodes to execute more than one simultaneous
+task.}.  Computers which are in the {\tt free} state may have tasks
+assigned to it.  The controller must also manage an additional set of
+constraint tables for each machine: the allowed tasks.  Each computer
+may have a list of allowed tasks which may include {\tt all} tasks,
+{\tt none} of the tasks, or specified task names.  The controller must
+only execute the allowed tasks on a machine.  
+
+The Controller must accept tasks from other IPP subsystems.  The task
+requests must include the specific command to be executed.  The
+commands must be in the form of a UNIX command which could be
+performed on any of the computing nodes.  Any input or output data
+structures in the commands must be a valid resource regardless of the
+node on which the task is executed.  Input and output data resources
+must be unique where necessary to avoid conflicts.  Tasks must be
+given an identified, which must be returned to the requesting agent,
+to be used to control the specific task.
+
+Task requests may specify a desired node for the task execution.  The
+Controller must attempt to honor the request if the node is {\tt
+alive}, but must execute on another node if the requested one is {\tt
+dead} or {\tt off}.  Even if a node is {\tt alive} the controller must
+choose another node if the specified tasks is not allowed on the
+requested node.  In all other cases, the controller must wait until
+executing processes, and processes with higher priority, are completed
+before executing the specified task on the requested node.
+
+Task requests may specify an urgency level.  The controller determines
+the priority of the task by sorting first by priority and next by the
+sequence of the request.  An executing task must be completed before
+any new task is started, regardless of priority.  Tasks may be
+assigned a priority of 0 in which case they are maintained in the
+queue and never executed.  
+
+The controller must monitor the output streams from the executing
+tasks and the exit status of the tasks.  \tbd{where do we send the
+output logs?}.  The status, including the exit status, of each task
+must be maintained for other subsystems to query as needed.  \tbd{how
+long?  on disk / database?}
+
+The controller must accept commands from other IPP subsystems.  These
+commands include those which govern the processing of specified tasks,
+those which govern the behavior of specific computing nodes, and those
+which request information from the controller.  The controller must be
+able to halt the execution of a specified task, delete an unexecuted
+task from the task list, change the priority of tasks, change the
+requested nodes for tasks.  The controller must also be able to stop
+the current execution of a task and push it to the end of the queue
+and also change its priority.
+
+The controller must honor requests to change the mode of any computing
+node on demand between {\tt off} and {\tt dead}.  It must also be able
+to change the list of allowed tasks as requested by external
+commands.  
+
+The controller must respond to informational requests regarding the
+collection of machines and their states as well as the collection of
+tasks and their states.  The controller must monitor the execution
+times of the different tasks and provide summary statistics.  Finally,
+the controller must respond to three top-level commands: {\tt finish},
+{\tt stop} and {\tt abort}.  When {\tt finish} is requested, no more
+new tasks are accepted, and when all tasks have completed, the
+controller must exti.  When {\tt stop} is requested, the currently
+executing tasks must be completed at which point the controller must
+exti.  When {\tt abort} is issued, the controller immediately kills
+all executing tasks and exits.
+
+\paragraph{Scheduler}
 
 \subsubsection{Analysis Stages}