Index: trunk/doc/design/ippSDRS.tex =================================================================== --- trunk/doc/design/ippSDRS.tex (revision 1399) +++ trunk/doc/design/ippSDRS.tex (revision 2114) @@ -1,3 +1,3 @@ -%%% $Id: ippSDRS.tex,v 1.4 2004-08-06 19:06:01 eugene Exp $ +%%% $Id: ippSDRS.tex,v 1.5 2004-10-14 05:06:31 eugene Exp $ \documentclass[panstarrs]{panstarrs} @@ -36,14 +36,5 @@ \pagenumbering{arabic} -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - \section{Scope} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Identification} @@ -54,8 +45,4 @@ Pan-STARRS 1 (PS-1), the initial demonstration telescope to be constructed on Haleakala by Jan 2006. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{System Overview} @@ -72,8 +59,4 @@ roughly 2 years. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - \subsection{Document Overview} @@ -85,9 +68,4 @@ Open Issues and TBDs in this document are marked \tbd{in bold, red type with surrounding square brackets}. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \DocumentsInternalSection @@ -100,10 +78,210 @@ \DocumentsEnd -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{System Design Decisions} +\section{Subsystem Overview} + +The Pan-STARRS Image Processing Pipeline (IPP) performs the image +processing and data analysis tasks needed to enable the scientific use +of the images obtained by the Pan-STARRS telescopes. The primary +goals of the IPP are to process the science images from the Pan-STARRS +telescopes and make the results available to other systems within +Pan-STARRS. It also is responsible for combining all of the science +images in a given filter into a single representation of the +non-variable component of the night sky called the ``Static Sky''. To +achieve these goals, the IPP also performs other analysis functions to +generate the calibrations needed in the science image processing and +to occasionally use the derived data to generate improved astrometric +and photometric reference catalogs. It also provides the +infrastructure needed to store the incoming data and the resulting +data products. + +The IPP inherits lessons learned, and in some cases code and prototype +code, from several other astronomy image analysis systems, including +Imcat (Kaiser), the Sloan Digital Sky Survey (REF), the Elixir system +(Magnier \& Cuillandre), and Vista (Tonry). Imcat and Vista have a +large number of robust image processing functions. SDSS has +demonstrated a working analysis pipeline and large-scale database +system for a dedicated project. The Elixir system has demonstrated an +automatic image processing system and an object database system for +operational usage. + +The users of the IPP output are all systems internal to the Pan-STARRS +project. They consist of the Transient Science Client, which will +receive the detections of transient objects on short time-scales; the +Moving Object Processing System (MOPS), which will receive the +detections of non-stationary transient objects on day-to-week +timescales; and the Published Science Products Subsystem (PSPS), which +will receive all data products of interest to the outside world, and +will act as the long-term archive and publishing clearinghouse. + +An important operational choice for the IPP is the decision not to +attempt to save all raw data. Once the IPP is running in a standard +operational mode, data will be processed by the pipeline and deleted +when it is no longer needed. Raw images will only be saved for a +short period to allow tests and quality assurance, and potentially to +allow systems which study transient phenomena to return to recent data +for closer inspection. In general, during normal operations, raw +science images will be deleted after $\sim$1 month. + +The primary IPP hardware system on which the software operates will +not be located at the summit. Instead, because of thermal, power, and +space constraints, the hardware will likely be located in a facility +off the mountain. A subset of processing tasks may eventually be +assigned to machines at the summit if justified by the savings in data +transfer time and cost. + +\subsection{Analysis Tasks and Stages} + +Specific programs are required to perform the processing steps listed +above. These can be divided into well-defined analysis stages, each +of which operates on a particular unit of data, such as a single OTA +image or a collection of astronomical objects. Analysis tasks +representing the different analysis stages are performed on the IPP +computer cluster. Note the distinction between the generic analysis +{\em stage} and a specific analysis {\em task}. An analysis stage +represents a type of analysis which is performed, such as the basic +image calibration and object detection analysis. An analysis task is +a particular realization of an analysis stage, e.g., the analysis of +OTA number 61 from exposure 654321 to produce a specific set of output +data products. The analysis stages are discussed in detail in +Section~\ref{IPP:AnalysisStages}. + +Depending on the particular stage, it may process individual images, +collections of images, or on derived data products. Because of the +nature of the image data, many of the analysis stages can be run in +parallel because, for example, the analysis of a chip in one image +does not depend on the results from another chip. + +\subsection{Architectural Components} + +In order to achieve the required functionality, the IPP provides an +infrastructure within which the Analysis Stages above are exectuted. +We have divided the IPP software infrastructure into a number of +clearly-defined architectural software units, listed as follows: + +\begin{itemize} + +\item {\bf Image Server:} This component is a large data store for all + images used by the IPP, including the raw images from the telescope, + the master calibration images, the reference static-sky images, and + any temporary image data products produced by the IPP. The Image + Server accepts the incoming data and stores it until it is no longer + needed by other portions of the IPP. The Image Server is not + restricted to imaging data: it is capable of storing any large data + files which are not well-suited for inclusion in a more structured + relational database and for which access needs to be widely + available beyond the individual process which created the file. + +\item {\bf Astrometry \& Photometry Database (AP DB):} This component + stores and manipulates astronomical objects detected in various + images, as identified above, including individual measurements of + objects on the images, the summary information about those objects, + and reference object data. It also provides mechanisms for users to + query and manipulate the objects and detections. + +\item {\bf Metadata Database:} This component stores the data which is + not directly related to images or astronomical objects, but which is + needed to perform the IPP analyses. The metadata may include the + summary weather information for each night, or details about the + filters, camera, telescopes, etc. + +\item {\bf IPP Controller:} In order to perform the analysis stages + required by the IPP, it is necessary to use distributed computing + processes on a large number of computers. The IPP Controller + manages the collection of analysis tasks performed on these + machines. + +\item {\bf IPP Scheduler:} This component is a decision-making + mechanism which guides the operation of the IPP. It evaluates the + currently available collection of data, identifies the necessary + analysis, and assigns the analysis tasks to the IPP Controller. + +\end{itemize} + +The relationship between these software units is shown in +Figure~\ref{overview}. This figure also shows the interactions +between the IPP and other Pan-STARRS systems. The architectural +components are discussed in detail in +Section~\ref{IPP:ArchComponents}. + +\begin{figure} +\begin{center} +\resizebox{6in}{!}{\includegraphics{pics/IPPoverview}} +\caption{ \label{overview} IPP System Overview} +\end{center} +\end{figure} + +\subsection{IPP Hardware Organization} + +\begin{figure} +\begin{center} +\resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}} +\caption{ \label{hardware} IPP Hardware Organization} +\end{center} +\end{figure} + +The IPP needs substantial computer resources, both in terms of +computational power and in terms of data storage and network +bandwidth. The IPP requires relatively large amounts of data storage +space, primarily for the image data. Image data is organized in two +categories. First, there is the per-OTA data -- data associated with +specific OTAs, including the raw images, the calibration images, and +temporary processed images at various stages. Second, there is the +data associated with the static sky imagery, which is in turn +organized into smaller sky-cell units. In addition to image data, +there are the somewhat smaller data entities of the Metadata Database +and AP Database. + +The computer hardware is organized into nodes which provide both data +storage and computational resources. The data storage nodes are +divided into three classes: those which deal with the per-OTA image +data, those that provide the storage for the static sky images, and +those that provide the storage for the other data systems, the +Metadata Database and the AP Database. In addition, the computational +tasks related to Phase 2 take place on the per-OTA storage nodes and +the Phase 4 computation takes place on the static sky storage nodes. + +Figure~\ref{hardware} shows our basic concept for the hardware +organization for the IPP. This diagram shows the two types of compute +nodes: OTA-level processing and storage nodes (dominated by Phase 2) +and static sky processing and storage nodes (mostly Phase 4). Also +shown are two switches which divide the network into OTA and +Static-Sky portions. In such an organization, the interswitch +communication must meet the throughput needs between these network +portions. The additional data systems (Metadata Database and AP +Database) are also shown. + +%%% needs some work / move around elsewhere +\subsection{System Interfaces} + +\paragraph{MOPS and other Client Science Pipelines} + +The Client Science Programs (CSPs) and the Moving Object Processing +System (MOPS) are not a part of the IPP, but are external systems. We +include them here to show the required interfaces. + +The CSPs and MOPS may query the Pixel DB, the Metadata DB and the +Object DB. In addition, they may write certain fields to the object +DB (e.g., the external identifiers and class of object) as they +process objects, and they may retrieve pixel data from the Nodes. + +Since ``CSPs'' is a vague term, we now give some examples which may +help to illustrate the functionality. + +One example of a CSP is a web front-end to retrieve (published) images +and objects from the Pixel DB and Object DB. + +Another example would be a program interested in searching for +transiting extrasolar planets. Such a program may periodically poll +the Metadata DB for new processed observations in its region of +interest (such as the Galactic Plane), retrieve the object photometry +of all high signal-to-noise stars in the processed observations, and +attempt to identify a planetary transit in progress. + +Yet another example would be a Stationary Transient Object Pipeline, +which would periodically poll the Metadata DB for new processed +observations, and query the Object DB for variable sources which were +identified twice (so that they are not moving objects). + +\subsection{System Design Decisions} Since \PS{} is a survey project, all data from the telescopes will be @@ -128,721 +306,169 @@ System (MOPS), and potentially other client science pipelines. -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{System Overview} - -The \PS{} Image Processing Pipeline (IPP) consists of a collection of -computer hardware and software organized to perform the tasks required -to process images from the \PS{} telescopes. The primary goal of the -IPP is to process the science images from the \PS{} telescopes and -make the results available to other systems within \PS{}. To achieve -this goal, the IPP must also perform other analysis functions to -generate the calibrations needed in the science image processing and -to occasionally use the derived data to generate improved astrometric -and photometric reference catalogs. - -In order to meet these broad goals, the IPP must have the following -capabilities: +\section{System Design : Architectural Components} + +\subsection{IPP Image Server} + +\begin{figure} +\psfig{file=pics/ImageServer,width=15cm,angle=0} +\caption{The components of the IPP Image Server.} +\label{fig:ImageServer} +\end{figure} + +The IPP Image Server is a repository for all images and other large +data files required by the IPP. In addition, it provides tools for +managing the distribution of these large data files and for accessing +the files. Data files stored by the IPP Image Server include the raw +images, the calibration images, intermediate processing stage images +as needed, final processed images, difference images, image +subsections, and any large non-imaging datafiles needed by the IPP. +The IPP Image Server must retain the files for as long as they are +needed by the IPP. + +The IPP Image Server is a parallel storage system. It stores data +across a collection of computer nodes, each with their own data +storage resources. Any single file is stored on only a single +computer and storage system. In order to achieve the data throughput +requirements, the IPP Image Server may distribute the images across +the processor nodes in an organized fashion, i.e.\ associating +specific machines with specific detectors. It is not the +responsibility of the IPP Image Server to determine which computer +should be associated with a specific data concept (Chip / region of +sky), but it must enable the association of a particular file with a +particular machine. + +There are three data concepts relevant to the IPP Image Server: \begin{itemize} -\item Store a large amount of image data, and other derived data -products (metadata and extracted objects); -\item Provide access mechanisms to these data products (both to the -subsystems of the IPP and in some cases to external users); -\item Continuously accept new image data and metadata from the -telescope system; -\item Execute various analysis processes using these data products; -and -\item Provide the decision-making logic needed to guide the data -processing, and to automatically launch the data processing tasks on -an appropriate timescale. +\item {\bf storage object} This represents a single, unique data + entity the Image Server. + +\item {\bf instance} A single copy of the storage object in the Image + Server. In general, given storage object may have several instances + in the Image Server, normally on different computer nodes. + +\item {\bf file ID} This is the identifier of a particular storage + object in the Image Server. The file ID is simply a unique string, + equivalent to the filename in a UNIX file system. \end{itemize} -The IPP therefore includes subsystems which provide the data storage -framework, the data analysis framework, and the scheduling of the -analysis processes. The data storage subsystems also provide -interface mechanisms to the external \PS{} systems. - -The IPP architecture can be viewed in several possible ways. We first -consider the software architecture components needed by the IPP. -These subsystems provide the infrastructure for the data storage and -the data processing. Next, we consider the analysis pipelines which -make up the major processing tasks that must be performed by the IPP. -Finally, we consider the hardware organization required to efficiently -and cost-effectively achieve the necessary computing and storage -requirements. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{System Architecture} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Architectural Components} - -In Figure~\ref{fig:functionalities} we show the functionality of the -IPP. - -The Observatory and Telescope System (\textbf{OATS}) system at the -summit periodically produces metadata (e.g.\ weather measurements, -observations completed) and pixel data (the image pixels from the -cameras). The \textbf{Pollster} regularly (e.g., twice per minute) -polls OATS for the existence of new data. If new data exists, the -Pollster writes it to the \textbf{Metadata DB}, which maintains a -table of observations that have been obtained and whether these -observations are reduced, not reduced, or being reduced. The -\textbf{Scheduler} regularly (e.g., twice per minute) polls the -Metadata DB for observations that match predefined criteria that are -required to run reduction processes. For example, the Phase 1 -processing requires that Phase 0 has been run on a focal plane -metadata, and also requires that the observations are available and -have not yet been processed. If the criteria are met, the appropriate -stage is passed to the \textbf{Localiser} which, checks the -\textbf{Pixel DB} to determine if the stage should be performed on a -particular node. The Localiser passes the reduction stage to be -processed, along with the preferred (or mandatory) node that should -execute the reduction stage, to the \textbf{Controller}. It is the -Controller's responsibility to maintain the list of reduction stages -to be processed and execute these stages on the \textbf{Nodes}. The -Nodes may retrieve the pixel data from OATS, they write to the Pixel -DB the location of the products of the reduction and report their -completion to the Controller. - -External systems, such as the Moving Object Processing System -(\textbf{MOPS}) and other Client Science Pipelines (\textbf{CSPs}) -read the Metadata DB and the Object DB. They may also write to the -Object DB the classification of particular objects (e.g., identify an -object as an asteroid). Also, the MOPS and CSPs may also query the -Pixel DB for the location of pixel data and copies data from the -Nodes. - -\begin{figure} -\psfig{file=pics/IPPfunctionalities,width=15cm,angle=0} -\caption{The functionalities of the architectural design. See the text -for further explanation.} -\label{fig:functionalities} -\end{figure} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{OATS} - -The Observatory And Telescope System (OATS) is not a part of the IPP, -but interfaces are required with it in order to allow the Pollster to -get the list of observations not in the Metadata DB, and the nodes to -retrieve pixel data. Also, the Scheduler may report the need for new -calibration data. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Pollster} - -The Pollster is a program that polls OATS at regular intervals for the -existence of observations not contained in the Metadata DB. New -weather and image metadata are written to the Metadata DB. - -There is no reason why this architectural component cannot be -contained within another (such as the Scheduler), but it is shown here -as separate for simplicity. - -A polling model is adopted so that OATS' interface may be kept as -simple as possible --- OATS should not be concerned with whether the -IPP has received notifications. Under this polling model, it is -specifically the responsibility of the IPP to retrieve from OATS the -metadata that is not not already in the Metadata DB. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Metadata DB} - -The Metadata DB stores and maintains the metadata\footnote{Note that -metadata is any data which is not pixel data or object data.}, -including the list of images taken by the telescope system and whether -these images have been processed. The Metadata DB is regularly polled -by the Scheduler to determine what images are ready to be processed. - -Both the Scheduler and the Pollster update the status of the Metadata -DB --- the Pollster as new images become available at the Summit, and -the Scheduler as images are processed. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Scheduler} - -The Scheduler is responsible for determining the processing stages -that are required to be run on any data. Examples of these processing -stages are ``Copy the pixels from the summit'' and ``Run Phase 2 -processing on chip 12 of exposure 123''. - -Processing stages to be executed are passed to the Localiser, which -returns to the Scheduler the list of processing stages with node -assignments to each of the stages. This list of processing stages -with node assignments is passed to the Controller for execution. - -Processing stages which have executed are reported by the Controller, -which updates the Metadata DB appropriately. - -The Scheduler may also interact with OATS to inform it of the need -for new calibration data. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Localiser} -\label{sec:localiser} - -It is the duty of the Localiser to assign processing stages to -particular nodes. This may be in order to optimise performance by -distributing the stages across the nodes, or in the simplest possible -case, it may make no recommendation upon the node which performs a -particular stage. - -The Localiser may query the Pixel DB in order to identify the location -of calibration data that may be needed for the processing stage to run -(and in all likelihood, assign the processing stage to the same node as -that which holds the calibration data). - -The Localiser may either demand or request that a stage is performed on -a particular node, or make no recommendation, and passes the processing -stage back to the Scheduler. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Controller} - -The Controller's job is to control the execution of the processing -stages on the nodes. It is passed stages by the Localiser, and -executes them on the appropriate nodes. It must detect whether a node -executing a processing stage has died, and re-execute the stage on an -alternate node. - -The completed stages are reported back to the Scheduler. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Pixel DB} -\label{sec:pixeldb} - -The Pixel DB is responsible for storing and maintaining the location -of pixel data in the IPP, including the raw images from the telescope, -the master calibration images, the reference static-sky images, and -any temporary image data products produced by the IPP. It provides -this information upon request to the Localiser. - -Note that this design assumes that the pixel data will be stored on -the same nodes that will be doing the processing. - -The Pixel DB will be periodically ``published'' as the quality of the -data is assured. The external world will only have access to the -published version of the Pixel DB. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Nodes} - -The Nodes perform the grunt work of executing the processing stages as -directed by the Controller. When the processing stage has completed, -they report back to the Controller. - -They may retrieve pixel data from OATS (the Summit) and write it to -local disk when directed to do so by the Controller. They also may -access the Metadata DB to read configurations, weather information -etc, and to write summary statistics etc. They may also access the -Object DB to read objects of interest, and to write objects from the -processing stage. - -As they write products, the Nodes register with the Pixel DB that they -have written the requested output (so that the Pixel DB is aware that -the data has been written and is not merely scheduled to be written). -The Nodes do not need to read from the Pixel DB, since everything -(where to read input pixels from, where to write output pixels to) is -specified by the Localiser. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Object DB} - -The Object DB is a facility to store all of the information about -astronomical objects, including individual measurements of objects on -the images, the summary information about those objects, and reference -object data\footnote{Note that this is (possibly) a separate entity -from the object database being developed by SAIC.}. - -The Nodes, CSPs and MOPS may read objects from the Object DB, and the -Nodes may write objects (either new objects or updates), and the CSPs -and MOPS may write certain fields of objects (e.g., the external -identifiers and class of object). - -The Object DB will be periodically ``published'' as the quality of the -data is assured. The external world will only have access to the -published version of the Object DB. The published version of the -Object DB will likely be the DB being developed by SAIC. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{CSPs and MOPS} - -The Client Science Programs (CSPs) and the Moving Object Processing -System (MOPS) are not a part of the IPP, but are external systems. We -include them here to show the required interfaces. - -The CSPs and MOPS may query the Pixel DB, the Metadata DB and the -Object DB. In addition, they may write certain fields to the object -DB (e.g., the external identifiers and class of object) as they -process objects, and they may retrieve pixel data from the Nodes. - -Since ``CSPs'' is a vague term, we now give some examples which may -help to illustrate the functionality. - -One example of a CSP is a web front-end to retrieve (published) images -and objects from the Pixel DB and Object DB. - -Another example would be a program interested in searching for -transiting extrasolar planets. Such a program may periodically poll -the Metadata DB for new processed observations in its region of -interest (such as the Galactic Plane), retrieve the object photometry -of all high signal-to-noise stars in the processed observations, and -attempt to identify a planetary transit in progress. - -Yet another example would be a Stationary Transient Object Pipeline, -which would periodically poll the Metadata DB for new processed -observations, and query the Object DB for variable sources which were -identified twice (so that they are not moving objects). - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Related/Connected components} - -The Pollster may be contained within the Scheduler (i.e., the -Scheduler may initiate and/or schedule as a processing stage the -Pollster), but this is not assumed to be so in this document; this -decision is left to the implementation. - -The Localiser is strongly coupled to the Pixel DB, and throughout this -document, these are both referred to as components of the ``IPP Pixel -Server''. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Responsibility} - -The IPP team will develop and have responsibility for maintaining -these systems. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Processing Stages} -\label{sec:processingStages} - -We now consider the collection of IPP processing stages which are -executed by the Controller on the Nodes. We define a ``stage'' to be -the largest complete task which may be performed in serial without -interation between parallel threads. - -Depending on the particular stage, it may process individual images, -collections of images, or on derived data products. Because of the -nature of the image data, many of the analysis stages can be run in -parallel because, for example, the analysis of a chip in one image -does not depend on the results from another chip. - -The data analysis stages are divided into several categories as follows: - -\begin{enumerate} -\item Retrieval Stage --- pixel data are retrieved from OATS (the - Summit). -\item Science Image Processing Stages - \begin{enumerate} - \item Phase 1: image processing preparation --- estimates - first-order astrometric and photometric solutions required to - process each major frame. - \item Phase 2: image reduction --- produces calibrated chips from - raw chips. - \item Phase 3: exposure analysis --- processes an FPA to produce - unified and consistent backgrounds, photometry and astrometry for - the component chips. - \item Phase 4: image combination --- processes sky cells overlapped - by a major frame. - \end{enumerate} -\item Calibration Image Processing Stages - \begin{enumerate} - \item Cal 1: Basic master-detrend creation --- combination of simple - detrend images (e.g., bias, dome flat etc). - \item Cal 2: Sky-model/fringe-mode generation --- combination of - more-complicated detrend images (e.g., fringe, scattered light - etc). - \item Cal 3: Flat-field correction image creation --- analysis of - photometry from multiple dithered FPAs. - \end{enumerate} -\item Calibration Test Processing Stage - \begin{enumerate} - \item CalTest 1: Detrend frame testing --- tests whether new - calibration frames are required. - \item CalTest 2: Photometric float correction testing --- tests - whether a new photometric flat correction is required. - \end{enumerate} -\item Reference Catalog Processing Stages - \begin{enumerate} - \item Astrometry reference catalog generation --- processing of the - astrometric data to determine and apply a consistent global - solution. - \item Photometry reference catalog generation --- processing of the - photometric data to determine and apply a consistent global - solution. - \end{enumerate} -\end{enumerate} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Hardware Systems} - -The basic IPP hardware organization is shown in Figure~\ref{hardware}. -The overall hardware organization, with a Detector subcluster and a -Static Sky subcluster, is largely chosen to reduce the I/O load during -the pre-reduction analysis of the raw science images. In addition, we -have specified distinct machines to maintain the object and metadata -databases. \tbd{This last aspect is largely theoretical until we have -defined the details of these databases; it may be more appropriate -depending on the eventual solutions to distribute these database -elements across the Detector and Static Sky subclusters.} - -\begin{figure} + +The Image Server provides file pointers (in C), handles (in Perl), or +file names corresponding to the instances of the storage objects. +Image Server requires a file system which provides files in the local +file system. This may be done over many machines with a network file +system such as NFS or GFS. \tbd{select file system for IPP / test NFS +vs GFS vs Mogile, etc}. + +The IPP Image Server provides the storage and access mechanisms, but +it does not include any logic or information about the data. The +Image Server does not, e.g., monitor the age of images and delete them +on some schedule. + +The IPP Image Server consists of the following components: + +\begin{itemize} +\item Image Server storage hardware +\item Image Server database +\item Image Server daemon +\item Image Server client APIs +\end{itemize} + +\paragraph{IPP Image Server Client APIs} + +Clients interact with the IPP Image Server with a small number of C +APIs (Bindings are also provided for Perl \tbr{and Python}). The +client commands are: + +\begin{itemize} +\item {\tt new object}: create a new storage object in the Image + Server. This function takes as input the file ID and returns a + C-style file pointer or a Perl file handle to the instance of the + storage object. The arguments to the function include an optional + node name on which the new storage object must be located. If this + target is not given, the Image Server places the new storage object + on an appropriate machine from the pool (least filled? most data? + randomized? the details need to be decided). + +\item {\tt open object}: open an instance of an existing storage + object, as identified by the file ID. This function may also + specify the node on which the object should be opened (if an + instance of the object is not stored on that node, the function + returns an error). On success, the function returns a file pointer. + +\item {\tt find object}: return a list of filenames in the UNIX name + space associated with the storage object identified by the given + file ID. Since there are in general multiple instances for a given + storage object, this function returns the collection of all + available instances. These may be freely opened by the client + server using the standard \code{fopen} functions. + +\item {\tt stat object}: returns status information about the + specified storage object, including the number of instances of the object. + +\item {\tt increment object count}: adds a new instance of the given + storage object. The target node may be optionally specified, + otherwise an appropriate node is selected. + +\item {\tt decrement object count}: removes one of the instances of + the storage object. The input parameters may optionally specify the + target machine to delete. + +\item {\tt delete object}: deletes all instances of the storage object + and sets the storage object status to {\tt deleted}. +\end{itemize} + +\subsubsection{IPP Image Server Daemon} + +The Image Server client requests are mediated via the Image Server +daemon. Communication between the clients and the server is via +\tbr{SOAP (or flat text commands)} implementing the commands above. + +\subsubsection{IPP Image Server Database} + +The IPP Image Server daemon uses a database to store the information +about the data storage objects, their instances, and the available +hardware resources. A \tt{mysql} database engine is used to manage +the database. The database tables defined for the Image Server are +listed in Table~\ref{ImageServerTables}, and their current contents +are listed in Appendix A. This database engine need not the same one +as used for the IPP Metadata Database. +% +\begin{table} \begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/hardware}} -\caption{ \label{hardware} IPP Hardware Organization} +\caption{Image Server Database Tables\label{ImageServerTables}} +\begin{tabular}{ll} +\hline +\hline +{\bf Table Name} & {\bf Description} \\ +\hline +\code{storage_object} & all storage objects known to Image Server \\ +\code{instance} & all instances of all storage objects \\ +\code{volume} & data storage devices known to Image Server \\ +\hline +\end{tabular} \end{center} -\end{figure} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Software Hierarchy} - -In order to facilitate testing and development, and to encourage -flexibility, the IPP will be built in a layered fashion. The lowest -level functions will be written in C and collected together into a -\PS{} library. These library functions will be used to write more -complex modules. The modules will be written in C but will make use -of the SWIG tool to make their functionality available within other -frameworks. In particular, the modules can be tied together with a -simple framework (an `engine') or with detailed flow-control through -the use of a high-level language such as Perl, Python, or Tcl -employing the SWIG interfaces. For the high-level functions in the -operational system, the IPP will make use of \tbd{Python} as the -scripting language to provide the required flow-control to tie the -modules together. - -This approach satisfies the requirement that complicated low-level -analysis steps run fast, while preserving flexibility for coding the -high-level wrappers for which the speed requirements are not so -stringent. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{External Libraries} - -\PS{} will employ several external libraries to save duplicating -functionality that is already available. These external libraries -will be wrapped by the \PS{} Library, insulating the project from the -implementation details of the external libraries. Examples of the -external libraries are FFTW and SLALib. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{\PS{} Library} - -The \PS{} Library will consist of C structures describing the basic -data types needed by the IPP and C functions which perform the basic -data manipulation operations. Note that a subset of the library -functions will be provided with SWIG interfaces as well to allow for -their use in the creation of the processing stages. Examples of the -\PS{} Library are fourier transforms and transforming between pixel -and celestial coordinates. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Modules} - -The IPP analysis stages are broken down into modules which represent -specific functional operations. The modules will be written in C -using the \PS{} Library functions and will be grouped into a \PS{} -Module Library. The modules will be provided with SWIG interfaces to -all public APIs for their use in processing stages. Examples of -modules are overscan subtraction and image combination. Some modules -(e.g.\ find objects on an image) will be used by multiple stages. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Stages} - -The major IPP processing tasks are organized into stages, which -consist of multiple modules. Each stage represents a collection of -complex operations performed on a single data entity. Each stage -therefore represents the maximum amount of effort which can be -performed in serial without interaction between parallel threads. The -stages will be written in \tbd{Python}, linking the modules together. -Examples of stages are Phase 2 (detrend images) and Phase 4 (combine -images from multiple telescopes and search for transients). - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Orchestration} - -High-level components such as the Scheduler, the Controller and the -Localiser are for process control. As such, they shall be written in -\tbd{Python} in order to maintain flexibility. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{System Interfaces} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{System Architectural Design} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Architectural Components} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Pollster} - -The Pollster simply polls OATS on a regular basis for metadata -(including telescope exposures) which is not known by the IPP (i.e., -already written in the Metadata DB). On the discovery of such metadata, -it is written to the Metadata DB. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Pixel Server} - -The IPP Pixel Server (IPS) is a repository for all image pixel data -required by the IPP, and fulfills the roles of the Pixel DB -(\S\ref{sec:pixeldb}) and the Localiser (\S\ref{sec:localiser}). In -addition, it also provides components for managing the distribution of -data, and accessing the data. - -Images may reside in the IPS for different periods depending on their -use and type. Data stored by the IPS include the raw images, the -calibration images, intermediate processing stage images as needed, -final processed images, difference images, and image subsections, -\tbd{along with the associated metadata}. The IPS must retain images -as long as they are needed, up to the lifetime of the project. In -order to achieve the I/O requirements, the IPS may maintain the pixel -data distributed across the processor nodes in an organized fashion, -i.e.\ associating specific machines with specific detectors. The IPS -interacts with the IPP Metadata Database to allow other systems or -subsystems to identify the available images meeting specified -criteria. IPS specifications are described in the IPS subsystem -specification. - -In addition to storing the pixel data, the IPS is responsible for -acquiring new image data and metadata from the Summit Pixel Server and -making it available for processing by the IPP System. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{IPP Pixel Server Components} - -The IPP Pixel Server (IPS) fulfills the roles of the Pixel DB -(\S\ref{sec:pixeldb}) and the Localiser (\S\ref{sec:localiser}), and -consists of the following components: - -\begin{enumerate} -\item IPP Pixel Server Data Locality Optimizer (IPSDLO) -\item IPP Pixel Server Database (IPSD) -\item IPP Pixel Server Maintainance (IPSM) -\item IPP Pixel Server I/O Library (IPSIOL) -\end{enumerate} - -This assumes that the pixel data will be stored on the nodes. Each -image shall have a unique Universal Resource Identifier (URI) which -specifies the location of the pixel data. As an example, consider a -cluster with cross-mounted disks --- in this case, the filename -incorporating the full path would serve as the URI. - -The components of the IPS and their relation to other components (both -within the IPS and without) are showin in Figure~\ref{fig:ips}. - -\begin{figure} -\psfig{file=pics/IPS,width=15cm,angle=0} -\caption{The components of the IPS. In addition to the IPSDLO, IPSD -and IPSM, the IPSIOL is also a component of the IPS; use of the IPSIOL -is shown as dotted arrows in the interactions. Note that the nodes use -the IPSIOL to pass pixel data between each other.} -\label{fig:ips} -\end{figure} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{IPP Pixel Server Data Locality Optimizer (IPPDLO)} - -Processing stages generated by the Scheduler are passed through the -IPSDLO which does the following: -\begin{enumerate} -\item assigns tasks to specific nodes; -\item identifies the URI of the required input data; and -\item identifies the URI the output data should be written to. -\end{enumerate} - -This allows the choice of processing node to be optimized so that it -resides on the node which will process it, as well as allowing the -output to be written to the node which requires it for the next -processing stage. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{IPP Pixel Server Database (IPSD)} -\label{sec:ipsd} - -The IPSD maintains a database of URIs for the pixel data on the nodes. -It should be able to return the URI of the pixel data given one of: -\begin{enumerate} -\item an exposure identifier and a chip identifier (raw and processed - pixel data from the telescope); -\item a calibration identifier (detrend pixel data); and -\item a sky cell identifier (summed static sky, reduced and difference - pixel data). -\end{enumerate} - -The IPSD will also contain a history of data management commands and -actions. - -\tbd{Is there a reason why this is not a part of the Metadata DB?} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{IPP Pixel Server Maintenance (IPSM)} - -The IPSM initiates the execution of bulk data management processing -stages. It may have an automated component which, e.g., monitors the -disk space on each of the nodes and redistributes them if they become -unbalanced. However, the main intent is that it is used by a human -operator to reorgainise the data, e.g., after a new data optimisation -plan has been formulated, or to delete old data. - -The IPSM passes processing stages to the Controller which executes -them on the specified nodes. - -The IPSM allows four types of operation: -\begin{itemize} -\item Retrieve external data --- to manually trigger the copying of - external data (routine copying of the pixel data from OATS is - handled by the Scheduler). The IPSM generates {\em retrieve data} - stages which are passed to the Controller for execution. -\item Delete data --- to delete old data. The IPSM looks up the - location of the data in the IPSD and generates {\em delete data} - stages which are passed to the Controller for execution. -\item Replicate data --- to backup and rearrange data. The IPSM - generates {\em copy data} stages which are passed to the Controller - for execution. Note that this mode differs from the ``copy external - data'' mode in that it copies data already within the IPS. -\item Move data --- to reorganise storage. The IPSM executes a - replication followed by a deletion. -\end{itemize} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{IPP Pixel Server I/O Library (IPSIOL)} - -The IPSIOL provides a mechanism for reading and writing pixel data to -the IPS. The existence of the IPSIOL insulates the processing stages -from the details of how the pixel data are stored (i.e., the -processing stages need not worry whether the data is stored locally or -remotely). It will generally be used on the nodes and the IPSDLO, -although other components will also make use of it. - -The IPSIOL will be able to: -\begin{itemize} -\item Open a file specified by a URI --- it may simply open the file - if it exists on the particular node, or it may retrieve the file - over the network. -\item Write a file specified by a URI --- it may simply write the file - if it exists on the particular node, or it may copy the file over - the network. It should also register with the IPSD that a file - specified by a URI has been written. -\item Delete a file specified by a URI --- it may simply delete the - file if it exists on the particular node, or it may delete the file - over the network. -\item Interface with the IPSD to return a URI given one of the - identifiers in \S\ref{sec:ipsd}. -\end{itemize} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Pixel Data Flow Examples} - -For examples of the operation of the IPS, below we sketch out the -intended sequence of events for common operations. - -Reads during processing: -\begin{enumerate} -\item A processing stage has been passed (from the Scheduler) the URI - for an image that it needs to load into memory. -\item The processing stage uses the IPSIOL to open the image. -\item The processing stage reads the image into local memory in the - usual manner. -\item The processing stage closes the image using the IPSIOL. -\end{enumerate} - -Writes during processing: -\begin{enumerate} -\item A processing stage has been passed (from the Scheduler) the URI - for an image that needs to be saved, e.g., a subtracted image. -\item The processing stage uses the IPSIOL to open the image. -\item The processing stage writes the image in the usual manner. -\item The processing stage closes the image using the IPSIOL. -\end{enumerate} - -Note how the IPSIOL has insulated the processing stage from the details -of the reading and writing. - -Maintenance: -\begin{enumerate} -\item A human operator decides that all the pixel data for chip 12 - should be stored on node 3. -\item Operator plugs this into the IPSM. -\item The IPSM queries the IPSD using the IPSIOL. -\item The IPSD returns the URIs for all the pixel data for chip 12. -\item The IPSM generates processing tasks to be executed on the nodes - that will copy the data from the old URIs to a new URI which - specifies node 3. -\item The IPSM generates processing tasks to be executed on the nodes - that deletes the data pointed to by the old URIs. -\item The IPSM reports success to the operator. -\end{enumerate} - -Client Science Pipelines: -\begin{enumerate} -\item A CSP wants some pixel data. -\item The CSP queries the IPSD using the IPSIOL (e.g., asking for a - particular exposure or sky cell). -\item The IPSD returns the URI for the pixel data. -\item The CSP opens the image using the IPSIOL and the URI. -\item The CSP reads the pixel data into memory in the usual manner. -\item The CSP closes the image using the IPSIOL. -\end{enumerate} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\end{table} + +\subsubsection{IPP Image Server Storage Hardware} + +The IPP Image Server manages data across a collection of computers and +possibly on multiple storage devices on those computer nodes. The +Image Server maintains a table of the available data volumes. The +Image Server tracks information about each volume such as the total +capacity, the current capacity, the association between computer and +data volume. + +\paragraph{IPP Image Server Maintenance Tools} + +The IPP Image Server provides a collection of administration tools +which allow for maintainence. These are operations which may be +automatically scheduled for the IPP or which may be initiated by a +human via a command-shell interface. The maintainence functions +include migrating data between nodes to rebalance the available space +(this would only occur for instances which have not been placed on a +specific node by the client API). Other functions include checking +for file corruption, which involves sweeping all files on a data +volume and comparing the calculated file checksum to the currently +recorded value. + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1477,4 +1103,27 @@ manner given the capabilities of the science pipelines. +\paragraph{Pollster} + +The Pollster is a program that polls OATS at regular intervals for the +existence of observations not contained in the Metadata DB. New +weather and image metadata are written to the Metadata DB. + +There is no reason why this architectural component cannot be +contained within another (such as the Scheduler), but it is shown here +as separate for simplicity. + +A polling model is adopted so that OATS' interface may be kept as +simple as possible --- OATS should not be concerned with whether the +IPP has received notifications. Under this polling model, it is +specifically the responsibility of the IPP to retrieve from OATS the +metadata that is not not already in the Metadata DB. + +\subsubsection{Pollster} + +The Pollster simply polls OATS on a regular basis for metadata +(including telescope exposures) which is not known by the IPP (i.e., +already written in the Metadata DB). On the discovery of such metadata, +it is written to the Metadata DB. + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1525,5 +1174,5 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsection{Processing Stages} +\subsection{Analysis Tasks and Stages} In this section, we review the processing stages which are executed on @@ -2326,4 +1975,64 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Software Hierarchy} + +In order to facilitate testing and development, and to encourage +flexibility, the IPP will be built in a layered fashion. The lowest +level functions will be written in C and collected together into a +\PS{} library. These library functions will be used to write more +complex modules. The modules will be written in C but will make use +of the SWIG tool to make their functionality available within other +frameworks. In particular, the modules can be tied together with a +simple framework (an `engine') or with detailed flow-control through +the use of a high-level language such as Perl, Python, or Tcl +employing the SWIG interfaces. For the high-level functions in the +operational system, the IPP will make use of \tbd{Python} as the +scripting language to provide the required flow-control to tie the +modules together. + +This approach satisfies the requirement that complicated low-level +analysis steps run fast, while preserving flexibility for coding the +high-level wrappers for which the speed requirements are not so +stringent. + +\subsubsection{External Libraries} + +\PS{} will employ several external libraries to save duplicating +functionality that is already available. These external libraries +will be wrapped by the \PS{} Library, insulating the project from the +implementation details of the external libraries. Examples of the +external libraries are FFTW and SLALib. + +\subsubsection{\PS{} Library} + +The \PS{} Library will consist of C structures describing the basic +data types needed by the IPP and C functions which perform the basic +data manipulation operations. Note that a subset of the library +functions will be provided with SWIG interfaces as well to allow for +their use in the creation of the processing stages. Examples of the +\PS{} Library are fourier transforms and transforming between pixel +and celestial coordinates. + +\subsubsection{Modules} + +The IPP analysis stages are broken down into modules which represent +specific functional operations. The modules will be written in C +using the \PS{} Library functions and will be grouped into a \PS{} +Module Library. The modules will be provided with SWIG interfaces to +all public APIs for their use in processing stages. Examples of +modules are overscan subtraction and image combination. Some modules +(e.g.\ find objects on an image) will be used by multiple stages. + +\subsubsection{Stages} + +The major IPP processing tasks are organized into stages, which +consist of multiple modules. Each stage represents a collection of +complex operations performed on a single data entity. Each stage +therefore represents the maximum amount of effort which can be +performed in serial without interaction between parallel threads. The +stages will be written in \tbd{Python}, linking the modules together. +Examples of stages are Phase 2 (detrend images) and Phase 4 (combine +images from multiple telescopes and search for transients). \subsection{Modules} @@ -3265,4 +2974,57 @@ \section{Appendices} +\subsection{Image Server Database Tables} + +\begin{table} +\begin{center} +\caption{Storage Object Table Contents\label{ImageServerTables:SO}} +\begin{tabular}{ll} +\hline +\hline +{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ +\hline +\code{so_id} & integer & internal storage object identifier \\ +\code{ext_id} & string & external storage object identifier (file ID) \\ +\code{comment} & string & user description of object \\ +\code{epoch} & time/date & last date of access \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Instance Table Contents\label{ImageServerTables:INT}} +\begin{tabular}{ll} +\hline +\hline +{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ +\hline +\code{ins_id} & integer & internal instance identifier \\ +\code{so_id} & integer & key to storage object table \\ +\code{uri} & string & location in hardware collection \\ +\code{sha1sum} & string & checksum information \\ +\code{assigned_location} & boolean & is location user-specified? \\ +\code{epoch} & time/date & last date of access \\ +\code{atime} & time/date & last date of access \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Volume Table Contents\label{ImageServerTables:VOL}} +\begin{tabular}{ll} +\hline +\hline +{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ +\hline +\code{vol_id} & integer & internal volume identifier \\ +\code{uri} & string & node name? \\ +\hline +\end{tabular} +\end{center} +\end{table} \bibliographystyle{plain}