Index: /trunk/doc/design/specs.tex =================================================================== --- /trunk/doc/design/specs.tex (revision 412) +++ /trunk/doc/design/specs.tex (revision 413) @@ -1,3 +1,3 @@ -%%% $Id: specs.tex,v 1.1 2004-04-09 02:25:41 eugene Exp $ +%%% $Id: specs.tex,v 1.2 2004-04-10 01:14:49 eugene Exp $ \documentclass[panstarrs]{panstarrs} @@ -53,4 +53,9 @@ Open Issues and TBDs in this document are marked \tbd{in bold, red with surrounding square brackets}. + +All timing measurements are to execution time as measured on a +\tbd{Reference Pan-Starrs Computation Node} and assumed to be not +limited by network bandwidth. + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -279,478 +284,292 @@ \paragraph{Pixel Server} -\begin{itemize} -\item $T_{\rm min}$ is the minimum time between exposures. $T_{\rm min}$ is -assumed to always be $\ge 30s$. -\item All timing measurements are to execution time as measured on a -\tbd{Reference Pan-Starrs Computation Node} and assumed to be not limited -by network bandwidth. -\end{itemize} - -\begin{enumerate} -\item IPP Pixel Data Scheduler -\item IPP Pixel Data Locality Optimizer -\item IPP Pixel Data Database -\item IPP Pixel Data Retrieval Agent -\item IPP Pixel Data Query Library -\item IPP Pixel Data I/O Library -\end{enumerate} - -\subparagraph{IPP Pixel Data Scheduler (IPP-PDS)} - -{\it Inputs} - -\begin{itemize} -\item Accepts an XML document containing the type of operation and image -meta-data if applicable. -\end{itemize} - -The input document is one of the follow classes of message. - -\begin{itemize} -\item {\em new data notification} -\item {\em move data request} -\item {\em copy data request} -\item {\em delete data request} -\end{itemize} - -\tbd{The format of this XML doc is TBD.} -\tbd{The application layer transport protocol is TBD.} - -{\it Outputs} - -Outputs an XML document containing one or more {\em data managements tasks}. -This document is to be passed through all registered filters that match it's -task type. The document is then sent to the IPP Controller. - -The IPP-PDS can fulfill all of it's modes of operations by generating just two -types of {\em data management tasks}. - -\begin{itemize} -\item {\em retrieve data task} -\item {\em delete data task} -\end{itemize} - -Seemingly there should be a third task type {\em move data} but this can be -broken down into a {\em retrieve data} task and a {\em delete data}. This -strategy has the added benefit of adding atomicity to the operation. - -\tbd{The format of this XML doc is TBD.} -\tbd{The application layer transport protocol is TBD.} - -{\it Configuration} - -A configuration file defining at least the address of IPP Controller and the -IPP Pixel Data Database connection string. - -\tbd{The format of this file is TDB.} - -{\it Performance} - -The IPP-PDS must be able to concurrently: - -\begin{itemize} -\item receive and process 260 {\em new data notifications} in less time -then ${T_{\rm min}}$. -\item generate, filter, and transmit 260 {\em data management tasks} in less -time then ${T_{\rm min}}$. -\end{itemize} - -\subparagraph{IPP Pixel Data Locality Optimizer (IPP-PDLO)} - -{\it Inputs} - -\begin{itemize} -\item Accepts an XML document containing one or {\em data management tasks}. -\end{itemize} - -\tbd{The format of this XML doc is TBD.} -\tbd{The I/O protocol is TBD (possibly stdin/stdout).} - -{\it Outputs} - -\begin{itemize} -\item Outputs an XML document containing one or {\em data management tasks}. -\end{itemize} - -\tbd{The format of this XML doc is TBD.} -\tbd{The I/O protocol is TBD (possibly stdin/stdout).} - -\subparagraph{Configuration} - -A configuration file defining what sort of optimization should be done and -the IPP Pixel Data Database connection string. - -\tbd{The format of this file is TDB.} - -\subparagraph{Performance} - -\begin{itemize} -\item The time spent in this filter should be added to the execution timing of -the IPP Pixel Data Scheduler and can not cause it to exceed $T_{\rm min}$. -\end{itemize} - -\subparagraph{IPP Pixel Data Database (IPP-PDD)} - -The IPP Pixel Data Database will maintain a record of {\em new data notifications} -received from the Summit Pixel Server, the storage location of downloaded but -unreduced image data, the storage location of reduced image data, the storage -location of stacked image data, and the storage location of calibration data. -\tbd{In addition to the storage location(s) of image data some or all of it's -associated meta-data will contained in the IPP-PDD} - -{\it Interfaces} - -\begin{itemize} -\item Native database bindings -\item ODBC -\end{itemize} - -{\it Configuration} - -\tbd{Database scheme is TBD.} - -{\it Functionality} - -\begin{itemize} -\item Linux $\ge$ 2.4.x and Glibc $\ge$ 2.3 -\item SQL Syntax $\ge$ SQL-99 -\item native bindings for C and Perl -\item databases size $\ge 1TB$ -\item ODBC $\ge 3.5$ -\item basic stored procedure support -\item hot backups -\item replication -\end{itemize} - -{\it Performance} - -\begin{itemize} -\item Process $> 30 \times 260$ select, insert, update, or delete queries in less time then -${T_{\rm min}}$. -\end{itemize} - -\subparagraph{IPP Pixel Data Retrieval Agent (IPP-PDRA)} - -One instance of the IPP Pixel Data retrieval Agent is spawned per -{\em data management task} that needs to be serviced. The IPP Pixel Data I/O -Library will be used to retrieve a URI into memory and to write data from -memory to a specified URI. \tbd{File lock management may or may not be -necessary within this component.} - -{\it Inputs} - -\begin{itemize} -\item Accepts an XML document containing one or {\em data management tasks}. -\end{itemize} - -\tbd{The format of this XML doc is TBD.} -\tbd{The I/O protocol is TBD (possibly stdin/stdout).} - -{\it Outputs} - -\begin{itemize} -\item Returns an XML document containing one or completed {\em data management tasks}. -\end{itemize} - -\tbd{The format of this XML doc is TBD.} -\tbd{The I/O protocol is TBD (possibly stdin/stdout).} - -{\it Configuration} - -A configuration file defining the address of the IPP Controller. - -\tbd{The format of this file is TDB.} - -{\it Performance} - -\begin{itemize} -\item Must capable of fully saturating a $1Gb/s$ network connection via the -IPP-PDIOL. -\end{itemize} - -\subparagraph{IPP Pixel Data Query Library (IPP-PDQL)} - -The IPP Pixel Data Query Library must hide all SQL details from the caller. - -{\it Interfaces} - -\begin{itemize} -\item C API -\item Perl (XSub of the C API) API -\end{itemize} - -{\it Configuration} - -A configuration file with the database connection string. - -\tbd{The format of this file is TDB.} - -{\it Query Types} - -The IPP-PDQL only supports simple database queries. - -\tbd{The specific queries supported is TDB.} - -{\it Performance} - -\begin{itemize} -\item Process $> 30$ select, insert, update, or delete queries in less time then -${T_{\rm min}}$. -\end{itemize} - -\subparagraph{IPP Pixel Data I/O Library (IPP-PDIOL)} - -The IPP Pixel Data I/O Library retrieves data from or writes data to \cite{uri}s. -Must be able to download multiple segments of a file simultaneously if the -transport protocol supports it. Similar to the \cite{proz} download accelerator. -Must be able to handle file locking issues if the transport protocol supports it. -The HTTP/WEBDAV protocol should be implement with the \cite{neon} library. - -{\it Interfaces} - -\begin{itemize} -\item C API -\item Perl (XSub of the C API) API -\end{itemize} - -{\it Configuration} - -A configuration file defining the optional behaviors for the protocols that -have optional features. - -\tbd{The format of this file is TDB.} - -{\it Protocols} - -Must support at least the following protocols: - -\begin{itemize} -\item \cite{http} -\item \cite{http} w/\cite{webdav} -\item \cite{ftp} -\item \cite{rsync} -\item file -\end{itemize} - -{\it Performance} - -\begin{itemize} -\item Must capable of fully saturating a $1Gb/s$ network connection. -\end{itemize} - -\subparagraph{Pixel Data Flow} - -\subparagraph{Bandwidth} - -\begin{enumerate} -\item Summit Pixel Server(s) -\begin{itemize} -\item $n \times$ \tbd{TBD} -\begin{itemize} -\item $\frac{2 \times 2.4Gb/s}{n}$ -\end{itemize} -\end{itemize} -\item Summit Core Switch -\begin{itemize} -\item Cisco 65xx -\begin{itemize} -\item $2 \times 2.4Gb/s$ -\end{itemize} -\end{itemize} -\item Summit Border Router -\begin{itemize} -\item Cisco 76xx -\begin{itemize} -\item $1 \times 2.4Gb/s$ -\end{itemize} -\end{itemize} -\item Summit $\Longleftrightarrow$ Data Center connection (WAN link) -\begin{itemize} -\item $2 \times 1Gb/s$ Ethernet (over ATM/Sonet) or OC-48 Sonet -\begin{itemize} -\item $1 \times 2.4Gb/s$ -\end{itemize} -\end{itemize} -\item Data Center Border Router -\begin{itemize} -\item Cisco 76xx -\begin{itemize} -\item $1 \times 2.4Gb/s$ -\end{itemize} -\end{itemize} -\item IPP Cluster Core Switch -\begin{itemize} -\item Cisco 65xx -\begin{itemize} -\item $48Gb/s$ -\end{itemize} -\end{itemize} -\item IPP Cluster Nodes -\begin{itemize} -\item $240 \times$ \tbd{Reference Pan-Starrs Computation Node} -\begin{itemize} -\item $\frac{48Gb/s}{240}$ -\end{itemize} -\end{itemize} -\item Extended Network -\begin{itemize} -\item Cisco 65xx -\begin{itemize} -\item \tbd{TBD} -\end{itemize} -\end{itemize} -\item Static Sky DB, other components, etc. -\begin{itemize} -\item \tbd{TBD} -\begin{itemize} -\item \tbd{TBD} -\end{itemize} -\end{itemize} -\end{enumerate} - -\begin{figure} -\begin{center} -% \resizebox{!}{20cm}{\includegraphics{pixel_wan.epsi}} -\caption{ \label{acquisition} Pixel Data Flow: Bandwidth} -\end{center} -\end{figure} -\pagebreak - -\subparagraph{Bandwidth Estimates} - -{\it Assumptions} - -\begin{itemize} -\item $T_{\rm min} = 30s$ -\end{itemize} - -{\it Exposure with overclocks in integer} - -{\it Storage Size} -$$2bytes \times (4096^2pixels \times 1.125overclocks) \times 240otas = 72477573120b$$ - -{\it Bandwidth Requirement} -$$\frac{72477573120b}{T_{\rm min}} = 2415919104b/s$$ - -{\it Exposure in float} - -{\it Storage Size} -$$4bytes \times 4096^2pixels \times 240otas = 128849018880b$$ - -{\it Bandwidth Requirement} -$$\frac{128849018880b}{T_{\rm min}} = 4294967296b/s$$ - -{\it Stacked exposure in float} - -{\it Storage Size} -$$4bytes \times 4096^2pixels \times 60otas = 32212254720b$$ - -{\it Bandwidth Requirement} -$$\frac{32212254720b}{T_{\rm min}} = 1073741824b/s$$ - -{\it Full calibration set in float} - -$$(1 \times debias, 1 \times dark, 1 \times flat, 2 \times fringe, 2 \times sky)$$ - -{\it Storage Size} -$$7 \times (exposure\ in\ float) = 901943132160b$$ - -{\it Bandwidth Requirement} -$$\frac{901943132160b}{T_{\rm min}} = 30064771072b/s$$ - -{\it Aggregate Bandwidth Requirement} - -\begin{center} -% \begin{tabular}{>{$}l<{$}>{$}r<{$}l} -\begin{tabular}{lrl} -[phase 2]&&\\ - & 2415919104b/s & summit $\rightarrow$ disk (exposure)\\ -+& 2415919104b/s & non-local disk $\rightarrow$ memory (exposure)\\ -+& 30064771072b/s& non-local disk $\rightarrow$ memory (calibration)\\ -+& 4294967296b/s & memory $\rightarrow$ non-local disk (reduced)\\ -\cline{1-2} - & 39,191,576,576b/s &\\ - -[phase 4]&&\\ - & 4294967296b/s & non-local disk $\rightarrow$ memory (reduced)\\ -+& 1073741824b/s & non-local disk $\rightarrow$ memory (best)\\ -+& 1073741824b/s & non-local disk $\rightarrow$ memory (working)\\ -+& 1073741824b/s & memory $\rightarrow$ non-local disk (diff)\\ -+& 1073741824b/s & memory $\rightarrow$ non-local disk (working)\\ -\cline{1-2} - & 8,589,934,592b/s &\\ - -[total]&&\\ - & 39191576576b/s & [phase 2] total\\ -+& 8589934592b/s & [phase 4] total\\ -\cline{1-2} - & 47,781,511,168b/s & $\sim48Gb/s$ -\end{tabular} -\end{center} - -\subparagraph{IPP Pixel Data Database Query Estimates} - -{\it Assumptions} - -\begin{itemize} -\item There is no caching of query results. -\end{itemize} - -{\it Acquisition} - -\begin{itemize} -\item select new data notification from IPP Data Scheduler -\item insert new data notification from IPP Data Scheduler -\item select from IPP Pixel Data Locality Optimizer -\item select new data notification from IPP Pixel Data Scheduler -\item update new data notification from IPP Pixel Data Scheduler -\item select data available from IPP Pixel Data Scheduler -\item insert data available from IPP Pixel Data Scheduler -\end{itemize} - -{\it Phase 2} - -\begin{itemize} -\item select data available from IPP Scheduler -\item select $\times 7$ calibration data from IPP Image Agent -\item select data available from IPP Scheduler -\item update data available from IPP Scheduler -\item select reduced data available from IPP Scheduler -\item insert reduced data available from IPP Scheduler -\end{itemize} - -{\it yPhase 4} - -\begin{itemize} -\item select reduced data available from IPP Scheduler -\item select $\times 2$ stacked data from IPP Image Agent -\item select reduced data available from IPP Scheduler -\item update reduced data available from IPP Scheduler -\item select stacked data available from IPP Scheduler -\item insert stacked data available from IPP Scheduler -\item select difference data available from IPP Scheduler -\item insert difference data available from IPP Scheduler -\end{itemize} - -\begin{verbatim} -\bibitem[Link aggregation]{aggregation} -http://cisco.com/en/US/products/hw/switches/ps708/products\_configuration\_guide\_chapter09186a008019f011.html -\bibitem[ProZilla]{proz} -http://prozilla.genesys.ro/ -\bibitem[neon]{neon} -http://www.webdav.org/neon/ -\bibitem[Uniform Resource Identifiers (URI)]{uri} -ftp://ftp.rfc-editor.org/in-notes/rfc2396.txt -\bibitem[HTTP]{http} -ftp://ftp.rfc-editor.org/in-notes/rfc2616.txt -\bibitem[WEBDAV]{webdav} -ftp://ftp.rfc-editor.org/in-notes/rfc2518.txt\\ -ftp://ftp.rfc-editor.org/in-notes/rfc3253.txt\\ -ftp://ftp.rfc-editor.org/in-notes/rfc3648.txt -\bibitem[FTP]{ftp} -ftp://ftp.rfc-editor.org/in-notes/rfc454.txt -\bibitem[rsync]{rsync} -http://rsync.samba.org/ -\end{verbatim} +The IPP Pixel Server \tbd{rename as Image Server?} is a large data +store for all images used by the IPP. The Pixel Server is required to +store all of the images needed by the IPP for the length of time they +are required; total data volume is specified in detail in the hardware +summary, but is in the vicinity of \tbd{700 GB}. + +The IPP Pixel Server must maintain a record of all images currently +available in the repository \tbd{and all no longer available}. This +record shall include the image name, location (which machine), the +state of the image (available, deleted), the image size, the image +type, and the existence and location of secondary copies of the +image. This information need not include other metadata such as the +image summary statistics or the state of the image processing for the +image. + +The IPP Pixel Server shall store images as FITS files on disk. Raw +images from the telescope shall be stored as individual OTA images for +each file, with multiple Cell images per file as well as video +sequences from the guide stars. Images of the Static Sky shall be +stored in the form of \tbd{triangular segments} to minimize the total +data volume and pixel overlap. + +The IPP Pixel Server shall distribute images across a cluster of +machines. The IPP Pixel Server shall be capable of honoring requests +to store an image on a specific machine. If such a request cannot be +honored, the IPP Pixel Server shall select an appropriate machine and +notify the requesting agent of the new locations. The IPP Pixel +Server shall provide a mechanism to maintain multiple (at least two) +copies of a single known image. + +The IPP Pixel Server must interface with other subsystems of the IPP. +It must provide an interface to other IPP subsystems to identify the +image location (the computer on which it resides). It must provide a +mechanism to serve a specified image to another IPP or Pan-STARRS +subsystem. It must provide a mechanism for deletion of images in the +Pixel Server. It must have a mechanism to accept or retrieve an image +from another Pan-STARRS subsystem, in particular OATS. Communication +of messages between the IPP Pixel Server and other subsystem shall be +via \tbd{XML messages} passed via \tbd{some transport}. + +The IPP Pixel Server must accept images at the telescope maximum rate +of 1 full-camera image every 30 seconds. The IPP Pixel Server must +accept notifications and process retrievals at a rate of 128 raw OTAs +per 60 seconds. + +\tbd{O/S, language, SQL, ODBC requirements?} + +\tbd{hardware requirements?} + +\tbd{communication protocols?} + +\paragraph{P\&A Database} + +The IPP requires a mechanism to store data related to astronomical +objects derived from various sources with a variety of associations. +The PnA (Photometry and Astrometry) Database serves this function. +The PnA Database deals with two related concepts: {\em objects} and +{\em detections}. The objects are descriptions of astronomical +objects while the detections are the specific measurements of those +objects on an image. A collection of {\em detections} may be used to +derive average quantities which describe a particular {\em object}. + +The PnA Database must store the collections of detections which were +derived from specific images from any of the analysis stages. It must +be possible to determine and locate (perhaps via interactions with the +pixel server) the image from which a specific detection was derved. +It must also be possible to extract all detections derived from a +specific images. These associations must include descriptive +information including the coordinates of the detection on the image. + +The PnA Database must provide a mechanism to associate together +multiple detections of a specific object. Several major classes of +objects will be present, each of which must be handled correctly. + +First, the distant stars will have nearly fixed locations relative to +other nearby stars, with only small deviations for individual +measurements. The association between multiple detections of such +objects must be made on the basis of their coincident positions. The +PnA Database must be able to determine the average position of the +object and the deviations of the individual detections from that +average. + +Second, solar system objects do not have a fixed location and +detections of such objects must associated on the basis of their +coincidence with the orbit of the objects. The PnA Database must be +able to associate detections with the orbits of known objects. The +determination of this association is the responsibility of the MOPS +and must be communicated to the IPP PnA Database on \tbd{some + timescale}. The PnD Database must be able to retrieve the +detections associated with the object and to provide the object +associated with the specific detections. This association must +include descriptive information such as the offset of the detection +from the predicted location of the detection based on the orbit. + +Third, stars in the general vicinity of the solar system fall in +between these first two classes of objects. Their proper motion and +parallax response is significant enough ($>1\asec$ in 10 years) that +they are not well-described by an average location and a collection of +offsets. These objects must be described by a distance and a proper +motion vector. The PnA Database must be able to find and associate +detections of objects for which either of the parallax or the proper +motion are substantial. + +Fourth, many detections, especially in their initial states, will not +be associated with a specific astronomical object of any of the above +classes and should be treated as orphans. Some of these will be +suprious (not represent real objects), some will be from solar system +objects for which orbits are not yet determined, some will be from +faint stars near the detection limits, some will be from short-term +transients which have only been detected once. The PnA Database must +be able to carry these detections until they have been associated with +one of the objects above. It must be possible to migrate individual +detections associated with an astronomical object back to the orphan +state. + +For every object, and all orphaned detections, it must be possible to +determine the images for which the coordinates were included but for +which no detection was made. The minimum set of information which +must be carried for these non-detections is the image and the +associated object or orphan. + +The PnA Database must store the relationships between various +photometric systems and, in some cases, the evolution of that +relationship. It must be possible, given a determined set of +calibrations, to convert between the measured instrumental magnitude +of a detection with a specific filter, detector, and telescope, and at +particular time and the implied magnitude in the average Pan-STARRS +magnitude systems. It must also be possible, given the magnitudes of +an object in one system to convert those to the magnitudes in another +system; an example of such a conversion is between the average +Pan-STARRS filter systems and the various reference systems +appropriate for those filters. + +The PnA Database must provide interfaces to extract lists of objects +and detections based on various query parameters. It must be possible +to extract all detections associated with a specific object, all +non-detections of that object (or orphan) and summary statistics from +these collections. It must be possible to extract all objects or +detections within specified spatial regions including regions bounded +by great circles (RA,DEC; GLAT,GLON; ELAT,ELON) and regions described +by a location and a search radius. It must be possible to extract the +image parameters associated with a specific detection including image +coordinates of the detection, exposure time, time and date of the +detection, etc. + +\tbd{volume requirements} + +\tbd{speed / access requirements} + +\paragraph{Metadata Database} + +The IPP requires a Metadata Database to store and provide access to +metadata of various types and from various sources. Metadata in the +context of the IPP represents all data which is not included in the +two data stores discussed above (Images and Detection/Objects). +Metadata is generated at the telescope and during the various analysis +stages + +The Metadata Database must store and provide metadata for all raw +images, for processed images, for the calibration images (both raw and +master), for the extracted object lists. Metadata describing the +environmental conditions at the telescope must also be stored and +provided as needed. + +If analysis results are exchanged via the metadata database, it must +provide access to the queried data on timescales of $<2 sec$ to avoid +slowing down the analysis systems. + +\tbd{volume requirements} + +\tbd{does the description of images belong in the Metadata database or + in the Pixel / Image Server?} + +\tbd{queries} + +\paragraph{Configuration Database -- a subset of the metadata database?} + +The IPP requires a Configuration Database to store and provide access to +information about the IPP itself. Examples of data in the +configuration database include the default parameters for the various +analysis programs, the description of the computing environment, the +process status information, etc. + +\paragraph{Controller} + +The IPP uses a collection of computers to store and process images and +to manipulate collections of detections. These computers perform any +of a large number of analysis stages or other processing tasks without +significant interprocess communication. It is necessary to have a +mechanism which initiates computing tasks on the different computers, +which monitors the tasks as they are executed, which handles the +output and the errors from these tasks, and which reacts to the +failure of any of the computing nodes. The system responsible for the +tasks in the IPP is the Controller. + +The Controller must interact with the collection of computers under +its management and with other subsystems in the IPP. The controller +must accept a variety of inputs from other subsystems, described +below, and respond accordingly. The controller must also provide +information to other subsystems on demand. + +Computers managed by the controller are allowed to be in one of +several states, and the controller must interact with it in an +appropriate way for each of those states. A computer may be {\tt +alive}, {\tt dead} or {\tt off}. If the computer is {\tt alive}, it +responds to commands from the controller and may be used for tasks +subject to other constraints. If it is {\tt dead}, the computer is +not responsive and should not be used for executing tasks. The +controller must identify computers which have died and occasionally +test them to see if they are {\tt alive} again. Computers which are +{\tt off} are not available for tests and should not be tested. +Computers may be set to the {\tt off} or {\tt dead} states by external +subsystems; it is the responsibility of the Controller to move a +computer to the {\tt alive} state if possible. + +Computers which are in the {\tt alive} state may be in one of two +modes: {\tt busy} and {\tt free}. A computer which is {\tt busy} +currently has a task assigned to it. The controller may only assign +one task to one computer at a time\footnote{a physical piece of +hardware may be defined to the Controller as multiple computers to +allow multi-processor nodes to execute more than one simultaneous +task.}. Computers which are in the {\tt free} state may have tasks +assigned to it. The controller must also manage an additional set of +constraint tables for each machine: the allowed tasks. Each computer +may have a list of allowed tasks which may include {\tt all} tasks, +{\tt none} of the tasks, or specified task names. The controller must +only execute the allowed tasks on a machine. + +The Controller must accept tasks from other IPP subsystems. The task +requests must include the specific command to be executed. The +commands must be in the form of a UNIX command which could be +performed on any of the computing nodes. Any input or output data +structures in the commands must be a valid resource regardless of the +node on which the task is executed. Input and output data resources +must be unique where necessary to avoid conflicts. Tasks must be +given an identified, which must be returned to the requesting agent, +to be used to control the specific task. + +Task requests may specify a desired node for the task execution. The +Controller must attempt to honor the request if the node is {\tt +alive}, but must execute on another node if the requested one is {\tt +dead} or {\tt off}. Even if a node is {\tt alive} the controller must +choose another node if the specified tasks is not allowed on the +requested node. In all other cases, the controller must wait until +executing processes, and processes with higher priority, are completed +before executing the specified task on the requested node. + +Task requests may specify an urgency level. The controller determines +the priority of the task by sorting first by priority and next by the +sequence of the request. An executing task must be completed before +any new task is started, regardless of priority. Tasks may be +assigned a priority of 0 in which case they are maintained in the +queue and never executed. + +The controller must monitor the output streams from the executing +tasks and the exit status of the tasks. \tbd{where do we send the +output logs?}. The status, including the exit status, of each task +must be maintained for other subsystems to query as needed. \tbd{how +long? on disk / database?} + +The controller must accept commands from other IPP subsystems. These +commands include those which govern the processing of specified tasks, +those which govern the behavior of specific computing nodes, and those +which request information from the controller. The controller must be +able to halt the execution of a specified task, delete an unexecuted +task from the task list, change the priority of tasks, change the +requested nodes for tasks. The controller must also be able to stop +the current execution of a task and push it to the end of the queue +and also change its priority. + +The controller must honor requests to change the mode of any computing +node on demand between {\tt off} and {\tt dead}. It must also be able +to change the list of allowed tasks as requested by external +commands. + +The controller must respond to informational requests regarding the +collection of machines and their states as well as the collection of +tasks and their states. The controller must monitor the execution +times of the different tasks and provide summary statistics. Finally, +the controller must respond to three top-level commands: {\tt finish}, +{\tt stop} and {\tt abort}. When {\tt finish} is requested, no more +new tasks are accepted, and when all tasks have completed, the +controller must exti. When {\tt stop} is requested, the currently +executing tasks must be completed at which point the controller must +exti. When {\tt abort} is issued, the controller immediately kills +all executing tasks and exits. + +\paragraph{Scheduler} \subsubsection{Analysis Stages}