Index: unk/doc/design/design.tex =================================================================== --- /trunk/doc/design/design.tex (revision 770) +++ (revision ) @@ -1,3369 +1,0 @@ -%%% $Id: design.tex,v 1.10 2004-04-30 03:41:45 price Exp $ -\documentclass[panstarrs]{panstarrs} - -% basic document variables -\title{Pan-STARRS Image Processing Pipeline} -\subtitle{Supplementary Design Requirements Specification} -\shorttitle{IPP SDRS} -\author{Eugene Magnier, Paul Price, Josh Hoblitt} -\group{\PS{} Algorithm Group} -\project{\PS{} Image Processing Pipeline} -\organization{Institute for Astronomy} -\version{DR} -\docnumber{PSDC-430-008} - -% allow paragraphs to be listed in TOC for now -\setcounter{tocdepth}{4} - -\begin{document} -\maketitle - -% -- Revision History -- -\RevisionsStart -% version Date Description -DR.01 & 2004.01.01 & First draft \\ \hline -DR.02 & 2004.03.05 & Second draft \\ \hline -DR.03 & 2004.03.25 & Section reorganization \\ \hline -DR.04 & 2004.04.13 & Most sections fleshed out \\ \hline -DR.05 & 2004.04.29 & Reorganisation for consistency --- PAP. \\ \hline -\RevisionsEnd - -\listoffigures -\pagebreak - -\tableofcontents -\pagebreak -\pagenumbering{arabic} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Scope} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Identification} - -This document establishes additional design requirements, beyond those -specified in the Software Requirement Specification (PSDC-430-005), for -the Pan-STARRS Image Processing Pipeline (IPP) as applied to -Pan-STARRS 1 (PS-1), the initial demonstration telescope to be -constructed on Haleakala by Jan 2006. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{System Overview} - -\PS{} is a survey telescope system being developed by the University -of Hawaii Institute for Astronomy (IfA), the Maui High Performance -Computing Center (MHPCC), Science Applications International -Corporation (SAIC), and Massachusetts Institute of Technology (MIT) -Lincoln Laboratory. The baseline system will consist of four 1.8m -telescopes, each with a 1 gigapixel camera capable of sustained image -rates of 2 per minute. A single initial test telescope (PS-1) will -be constructed on Haleakala and will see first light at the beginning -of 2006. The full four-telescope system (PS-4) will follow PS-1 by -roughly 2 years. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Document Overview} - -The Pan-STARRS document naming scheme is PSDC-NNN-MMM-VV, where the VV -entry specifies the document version number. Where documents are -identified without the version number, the latest official version in -that series is implied. - -Open Issues and TBDs in this document are marked \tbd{in bold, red -type with surrounding square brackets}. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\DocumentsInternalSection -PSDC-130-001 & PS-1 Design Reference Mission \\ \hline -PSDC-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline -PSDC-430-006 & Pan-STARRS IPP ADD \\ \hline -PSDC-430-007 & Pan-STARRS IPP PSLib SDR \\ \hline -\DocumentsExternalSection -Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ -\DocumentsEnd - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{System Design Decisions} - -Since \PS{} is a survey project, all data from the telescopes will be -uniformly analysed by the \PS{} Image Processing Pipeline (IPP) and -the appropriate resulting data products made available to internal and -external science analysis systems as they become available. The -processing performed by the IPP on the science images will consist of -detrending and object detection for the individual images, combination -of multiple overlapping images and further object detection, -subtraction of a reference (static-sky) image and detection of -residual objects, update of the static sky images, and detailed object -analysis of the static sky images. In addition, the IPP will produce -improved astrometric and photometric reference catalogs on an -occasional basis as needed. The output data products from the IPP -consist of the calibration images, reduced images from the individual -telescopes, combined images, difference images, the static sky image, -object photometry, and reference astrometry and photometry. - -The IPP interacts closely with other \PS{} systems responsible for -other aspects of the \PS{} operation, including the summit systems -(OATS), the science object database, the Moving Object Processing -System (MOPS), and potentially other client science pipelines. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{System Overview} - -The \PS{} Image Processing Pipeline (IPP) consists of a collection of -computer hardware and software organized to perform the tasks required -to process images from the \PS{} telescopes. The primary goal of the -IPP is to process the science images from the \PS{} telescopes and -make the results available to other systems within \PS{}. To achieve -this goal, the IPP must also perform other analysis functions to -generate the calibrations needed in the science image processing and -to occasionally use the derived data to generate improved astrometric -and photometric reference catalogs. - -In order to meet these broad goals, the IPP must have the following -capabilities: -\begin{itemize} -\item Store a large amount of image data, and other derived data -products (metadata and extracted objects); -\item Provide access mechanisms to these data products (both to the -subsystems of the IPP and in some cases to external users); -\item Continuously accept new image data and metadata from the -telescope system; -\item Execute various analysis processes using these data products; -and -\item Provide the decision-making logic needed to guide the data -processing, and to automatically launch the data processing tasks on -an appropriate timescale. -\end{itemize} -The IPP therefore includes subsystems which provide the data storage -framework, the data analysis framework, and the scheduling of the -analysis processes. The data storage subsystems also provide -interface mechanisms to the external \PS{} systems. - -The IPP architecture can be viewed in several possible ways. We first -consider the software architecture components needed by the IPP. -These subsystems provide the infrastructure for the data storage and -the data processing. Next, we consider the analysis pipelines which -make up the major processing tasks that must be performed by the IPP. -Finally, we consider the hardware organization required to efficiently -and cost-effectively achieve the necessary computing and storage -requirements. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{System Architecture} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Architectural Components} - -In Figure~\ref{fig:functionalities} we show the functionality of the -IPP. - -The Observatory and Telescope System (\textbf{OATS}) system at the -summit periodically produces metadata (e.g.\ weather measurements, -observations completed) and pixel data (the image pixels from the -cameras). The \textbf{Pollster} regularly (e.g., twice per minute) -polls OATS for the existence of new data. If new data exists, the -Pollster writes it to the \textbf{Metadata DB}, which maintains a -table of observations that have been obtained and whether these -observations are reduced, not reduced, or being reduced. The -\textbf{Scheduler} regularly (e.g., twice per minute) polls the -Metadata DB for observations that match predefined criteria that are -required to run reduction processes. For example, the Phase 1 -processing requires that Phase 0 has been run on a focal plane -metadata, and also requires that the observations are available and -have not yet been processed. If the criteria are met, the appropriate -stage is passed to the \textbf{Localiser} which, checks the -\textbf{Pixel DB} to determine if the stage should be performed on a -particular node. The Localiser passes the reduction stage to be -processed, along with the preferred (or mandatory) node that should -execute the reduction stage, to the \textbf{Controller}. It is the -Controller's responsibility to maintain the list of reduction stages -to be processed and execute these stages on the \textbf{Nodes}. The -Nodes may retrieve the pixel data from OATS, they write to the Pixel -DB the location of the products of the reduction and report their -completion to the Controller. - -External systems, such as the Moving Object Processing System -(\textbf{MOPS}) and other Client Science Pipelines (\textbf{CSPs}) -read the Metadata DB and the Object DB. They may also write to the -Object DB the classification of particular objects (e.g., identify an -object as an asteroid). Also, the MOPS and CSPs may also query the -Pixel DB for the location of pixel data and copies data from the -Nodes. - -\begin{figure} -\psfig{file=pics/IPPfunctionalities,width=15cm,angle=0} -\caption{The functionalities of the architectural design. See the text -for further explanation.} -\label{fig:functionalities} -\end{figure} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{OATS} - -The Observatory And Telescope System (OATS) is not a part of the IPP, -but interfaces are required with it in order to allow the Pollster to -get the list of observations not in the Metadata DB, and the nodes to -retrieve pixel data. Also, the Scheduler may report the need for new -calibration data. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Pollster} - -The Pollster is a program that polls OATS at regular intervals for the -existence of observations not contained in the Metadata DB. New -weather and image metadata are written to the Metadata DB. - -There is no reason why this architectural component cannot be -contained within another (such as the Scheduler), but it is shown here -as separate for simplicity. - -A polling model is adopted so that OATS' interface may be kept as -simple as possible --- OATS should not be concerned with whether the -IPP has received notifications. Under this polling model, it is -specifically the responsibility of the IPP to retrieve from OATS the -metadata that is not not already in the Metadata DB. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Metadata DB} - -The Metadata DB stores and maintains the metadata\footnote{Note that -metadata is any data which is not pixel data or object data.}, -including the list of images taken by the telescope system and whether -these images have been processed. The Metadata DB is regularly polled -by the Scheduler to determine what images are ready to be processed. - -Both the Scheduler and the Pollster update the status of the Metadata -DB --- the Pollster as new images become available at the Summit, and -the Scheduler as images are processed. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Scheduler} - -The Scheduler is responsible for determining the processing stages -that are required to be run on any data. Examples of these processing -stages are ``Copy the pixels from the summit'' and ``Run Phase 2 -processing on chip 12 of exposure 123''. - -Processing stages to be executed are passed to the Localiser, which -returns to the Scheduler the list of processing stages with node -assignments to each of the stages. This list of processing stages -with node assignments is passed to the Controller for execution. - -Processing stages which have executed are reported by the Controller, -which updates the Metadata DB appropriately. - -The Scheduler may also interact with OATS to inform it of the need -for new calibration data. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Localiser} -\label{sec:localiser} - -It is the duty of the Localiser to assign processing stages to -particular nodes. This may be in order to optimise performance by -distributing the stages across the nodes, or in the simplest possible -case, it may make no recommendation upon the node which performs a -particular stage. - -The Localiser may query the Pixel DB in order to identify the location -of calibration data that may be needed for the processing stage to run -(and in all likelihood, assign the processing stage to the same node as -that which holds the calibration data). - -The Localiser may either demand or request that a stage is performed on -a particular node, or make no recommendation, and passes the processing -stage back to the Scheduler. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Controller} - -The Controller's job is to control the execution of the processing -stages on the nodes. It is passed stages by the Localiser, and -executes them on the appropriate nodes. It must detect whether a node -executing a processing stage has died, and re-execute the stage on an -alternate node. - -The completed stages are reported back to the Scheduler. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Pixel DB} -\label{sec:pixeldb} - -The Pixel DB is responsible for storing and maintaining the location -of pixel data in the IPP, including the raw images from the telescope, -the master calibration images, the reference static-sky images, and -any temporary image data products produced by the IPP. It provides -this information upon request to the Localiser. - -Note that this design assumes that the pixel data will be stored on -the same nodes that will be doing the processing. - -The Pixel DB will be periodically ``published'' as the quality of the -data is assured. The external world will only have access to the -published version of the Pixel DB. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Nodes} - -The Nodes perform the grunt work of executing the processing stages as -directed by the Controller. When the processing stage has completed, -they report back to the Controller. - -They may retrieve pixel data from OATS (the Summit) and write it to -local disk when directed to do so by the Controller. They also may -access the Metadata DB to read configurations, weather information -etc, and to write summary statistics etc. They may also access the -Object DB to read objects of interest, and to write objects from the -processing stage. - -As they write products, the Nodes register with the Pixel DB that they -have written the requested output (so that the Pixel DB is aware that -the data has been written and is not merely scheduled to be written). -The Nodes do not need to read from the Pixel DB, since everything -(where to read input pixels from, where to write output pixels to) is -specified by the Localiser. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Object DB} - -The Object DB is a facility to store all of the information about -astronomical objects, including individual measurements of objects on -the images, the summary information about those objects, and reference -object data\footnote{Note that this is (possibly) a separate entity -from the object database being developed by SAIC.}. - -The Nodes, CSPs and MOPS may read objects from the Object DB, and the -Nodes may write objects (either new objects or updates), and the CSPs -and MOPS may write certain fields of objects (e.g., the external -identifiers and class of object). - -The Object DB will be periodically ``published'' as the quality of the -data is assured. The external world will only have access to the -published version of the Object DB. The published version of the -Object DB will likely be the DB being developed by SAIC. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{CSPs and MOPS} - -The Client Science Programs (CSPs) and the Moving Object Processing -System (MOPS) are not a part of the IPP, but are external systems. We -include them here to show the required interfaces. - -The CSPs and MOPS may query the Pixel DB, the Metadata DB and the -Object DB. In addition, they may write certain fields to the object -DB (e.g., the external identifiers and class of object) as they -process objects, and they may retrieve pixel data from the Nodes. - -Since ``CSPs'' is a vague term, we now give some examples which may -help to illustrate the functionality. - -One example of a CSP is a web front-end to retrieve (published) images -and objects from the Pixel DB and Object DB. - -Another example would be a program interested in searching for -transiting extrasolar planets. Such a program may periodically poll -the Metadata DB for new processed observations in its region of -interest (such as the Galactic Plane), retrieve the object photometry -of all high signal-to-noise stars in the processed observations, and -attempt to identify a planetary transit in progress. - -Yet another example would be a Stationary Transient Object Pipeline, -which would periodically poll the Metadata DB for new processed -observations, and query the Object DB for variable sources which were -identified twice (so that they are not moving objects). - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Related/Connected components} - -The Pollster may be contained within the Scheduler (i.e., the -Scheduler may initiate and/or schedule as a processing stage the -Pollster), but this is not assumed to be so in this document; this -decision is left to the implementation. - -The Localiser is strongly coupled to the Pixel DB, and throughout this -document, these are both referred to as components of the ``IPP Pixel -Server''. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Responsibility} - -The IPP team will develop and have responsibility for maintaining -these systems. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Processing Stages} -\label{sec:processingStages} - -We now consider the collection of IPP processing stages which are -executed by the Controller on the Nodes. We define a ``stage'' to be -the largest complete task which may be performed in serial without -interation between parallel threads. - -Depending on the particular stage, it may process individual images, -collections of images, or on derived data products. Because of the -nature of the image data, many of the analysis stages can be run in -parallel because, for example, the analysis of a chip in one image -does not depend on the results from another chip. - -The data analysis stages are divided into several categories as follows: - -\begin{enumerate} -\item Retrieval Stage --- pixel data are retrieved from OATS (the - Summit). -\item Science Image Processing Stages - \begin{enumerate} - \item Phase 1: image processing preparation --- estimates - first-order astrometric and photometric solutions required to - process each major frame. - \item Phase 2: image reduction --- produces calibrated chips from - raw chips. - \item Phase 3: exposure analysis --- processes an FPA to produce - unified and consistent backgrounds, photometry and astrometry for - the component chips. - \item Phase 4: image combination --- processes sky cells overlapped - by a major frame. - \end{enumerate} -\item Calibration Image Processing Stages - \begin{enumerate} - \item Cal 1: Basic master-detrend creation --- combination of simple - detrend images (e.g., bias, dome flat etc). - \item Cal 2: Sky-model/fringe-mode generation --- combination of - more-complicated detrend images (e.g., fringe, scattered light - etc). - \item Cal 3: Flat-field correction image creation --- analysis of - photometry from multiple dithered FPAs. - \end{enumerate} -\item Calibration Test Processing Stage - \begin{enumerate} - \item CalTest 1: Detrend frame testing --- tests whether new - calibration frames are required. - \item CalTest 2: Photometric float correction testing --- tests - whether a new photometric flat correction is required. - \end{enumerate} -\item Reference Catalog Processing Stages - \begin{enumerate} - \item Astrometry reference catalog generation --- processing of the - astrometric data to determine and apply a consistent global - solution. - \item Photometry reference catalog generation --- processing of the - photometric data to determine and apply a consistent global - solution. - \end{enumerate} -\end{enumerate} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Hardware Systems} - -The basic IPP hardware organization is shown in Figure~\ref{hardware}. -The overall hardware organization, with a Detector subcluster and a -Static Sky subcluster, is largely chosen to reduce the I/O load during -the pre-reduction analysis of the raw science images. In addition, we -have specified distinct machines to maintain the object and metadata -databases. \tbd{This last aspect is largely theoretical until we have -defined the details of these databases; it may be more appropriate -depending on the eventual solutions to distribute these database -elements across the Detector and Static Sky subclusters.} - -\begin{figure} -\begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/hardware}} -\caption{ \label{hardware} IPP Hardware Organization} -\end{center} -\end{figure} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Software Hierarchy} - -In order to facilitate testing and development, and to encourage -flexibility, the IPP will be built in a layered fashion. The lowest -level functions will be written in C and collected together into a -\PS{} library. These library functions will be used to write more -complex modules. The modules will be written in C but will make use -of the SWIG tool to make their functionality available within other -frameworks. In particular, the modules can be tied together with a -simple framework (an `engine') or with detailed flow-control through -the use of a high-level language such as Perl, Python, or Tcl -employing the SWIG interfaces. For the high-level functions in the -operational system, the IPP will make use of \tbd{Python} as the -scripting language to provide the required flow-control to tie the -modules together. - -This approach satisfies the requirement that complicated low-level -analysis steps run fast, while preserving flexibility for coding the -high-level wrappers for which the speed requirements are not so -stringent. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{External Libraries} - -\PS{} will employ several external libraries to save duplicating -functionality that is already available. These external libraries -will be wrapped by the \PS{} Library, insulating the project from the -implementation details of the external libraries. Examples of the -external libraries are FFTW and SLALib. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{\PS{} Library} - -The \PS{} Library will consist of C structures describing the basic -data types needed by the IPP and C functions which perform the basic -data manipulation operations. Note that a subset of the library -functions will be provided with SWIG interfaces as well to allow for -their use in the creation of the processing stages. Examples of the -\PS{} Library are fourier transforms and transforming between pixel -and celestial coordinates. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Modules} - -The IPP analysis stages are broken down into modules which represent -specific functional operations. The modules will be written in C -using the \PS{} Library functions and will be grouped into a \PS{} -Module Library. The modules will be provided with SWIG interfaces to -all public APIs for their use in processing stages. Examples of -modules are overscan subtraction and image combination. Some modules -(e.g.\ find objects on an image) will be used by multiple stages. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Stages} - -The major IPP processing tasks are organized into stages, which -consist of multiple modules. Each stage represents a collection of -complex operations performed on a single data entity. Each stage -therefore represents the maximum amount of effort which can be -performed in serial without interaction between parallel threads. The -stages will be written in \tbd{Python}, linking the modules together. -Examples of stages are Phase 2 (detrend images) and Phase 4 (combine -images from multiple telescopes and search for transients). - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Orchestration} - -High-level components such as the Scheduler, the Controller and the -Localiser are for process control. As such, they shall be written in -\tbd{Python} in order to maintain flexibility. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{System Interfaces} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{System Architectural Design} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Architectural Components} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Pollster} - -The Pollster simply polls OATS on a regular basis for metadata -(including telescope exposures) which is not known by the IPP (i.e., -already written in the Metadata DB). On the discovery of such metadata, -it is written to the Metadata DB. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Pixel Server} - -The IPP Pixel Server (IPS) is a repository for all image pixel data -required by the IPP, and fulfills the roles of the Pixel DB -(\S\ref{sec:pixeldb}) and the Localiser (\S\ref{sec:localiser}). In -addition, it also provides components for managing the distribution of -data, and accessing the data. - -Images may reside in the IPS for different periods depending on their -use and type. Data stored by the IPS include the raw images, the -calibration images, intermediate processing stage images as needed, -final processed images, difference images, and image subsections, -\tbd{along with the associated metadata}. The IPS must retain images -as long as they are needed, up to the lifetime of the project. In -order to achieve the I/O requirements, the IPS may maintain the pixel -data distributed across the processor nodes in an organized fashion, -i.e.\ associating specific machines with specific detectors. The IPS -interacts with the IPP Metadata Database to allow other systems or -subsystems to identify the available images meeting specified -criteria. IPS specifications are described in the IPS subsystem -specification. - -In addition to storing the pixel data, the IPS is responsible for -acquiring new image data and metadata from the Summit Pixel Server and -making it available for processing by the IPP System. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{IPP Pixel Server Components} - -The IPP Pixel Server (IPS) fulfills the roles of the Pixel DB -(\S\ref{sec:pixeldb}) and the Localiser (\S\ref{sec:localiser}), and -consists of the following components: - -\begin{enumerate} -\item IPP Pixel Server Data Locality Optimizer (IPSDLO) -\item IPP Pixel Server Database (IPSD) -\item IPP Pixel Server Maintainance (IPSM) -\item IPP Pixel Server I/O Library (IPSIOL) -\end{enumerate} - -This assumes that the pixel data will be stored on the nodes. Each -image shall have a unique Universal Resource Identifier (URI) which -specifies the location of the pixel data. As an example, consider a -cluster with cross-mounted disks --- in this case, the filename -incorporating the full path would serve as the URI. - -The components of the IPS and their relation to other components (both -within the IPS and without) are showin in Figure~\ref{fig:ips}. - -\begin{figure} -\psfig{file=pics/IPS,width=15cm,angle=0} -\caption{The components of the IPS. In addition to the IPSDLO, IPSD -and IPSM, the IPSIOL is also a component of the IPS; use of the IPSIOL -is shown as dotted arrows in the interactions. Note that the nodes use -the IPSIOL to pass pixel data between each other.} -\label{fig:ips} -\end{figure} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{IPP Pixel Server Data Locality Optimizer (IPPDLO)} - -Processing stages generated by the Scheduler are passed through the -IPSDLO which does the following: -\begin{enumerate} -\item assigns tasks to specific nodes; -\item identifies the URI of the required input data; and -\item identifies the URI the output data should be written to. -\end{enumerate} - -This allows the choice of processing node to be optimized so that it -resides on the node which will process it, as well as allowing the -output to be written to the node which requires it for the next -processing stage. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{IPP Pixel Server Database (IPSD)} -\label{sec:ipsd} - -The IPSD maintains a database of URIs for the pixel data on the nodes. -It should be able to return the URI of the pixel data given one of: -\begin{enumerate} -\item an exposure identifier and a chip identifier (raw and processed - pixel data from the telescope); -\item a calibration identifier (detrend pixel data); and -\item a sky cell identifier (summed static sky, reduced and difference - pixel data). -\end{enumerate} - -The IPSD will also contain a history of data management commands and -actions. - -\tbd{Is there a reason why this is not a part of the Metadata DB?} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{IPP Pixel Server Maintenance (IPSM)} - -The IPSM initiates the execution of bulk data management processing -stages. It may have an automated component which, e.g., monitors the -disk space on each of the nodes and redistributes them if they become -unbalanced. However, the main intent is that it is used by a human -operator to reorgainise the data, e.g., after a new data optimisation -plan has been formulated, or to delete old data. - -The IPSM passes processing stages to the Controller which executes -them on the specified nodes. - -The IPSM allows four types of operation: -\begin{itemize} -\item Retrieve external data --- to manually trigger the copying of - external data (routine copying of the pixel data from OATS is - handled by the Scheduler). The IPSM generates {\em retrieve data} - stages which are passed to the Controller for execution. -\item Delete data --- to delete old data. The IPSM looks up the - location of the data in the IPSD and generates {\em delete data} - stages which are passed to the Controller for execution. -\item Replicate data --- to backup and rearrange data. The IPSM - generates {\em copy data} stages which are passed to the Controller - for execution. Note that this mode differs from the ``copy external - data'' mode in that it copies data already within the IPS. -\item Move data --- to reorganise storage. The IPSM executes a - replication followed by a deletion. -\end{itemize} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{IPP Pixel Server I/O Library (IPSIOL)} - -The IPSIOL provides a mechanism for reading and writing pixel data to -the IPS. The existence of the IPSIOL insulates the processing stages -from the details of how the pixel data are stored (i.e., the -processing stages need not worry whether the data is stored locally or -remotely). It will generally be used on the nodes and the IPSDLO, -although other components will also make use of it. - -The IPSIOL will be able to: -\begin{itemize} -\item Open a file specified by a URI --- it may simply open the file - if it exists on the particular node, or it may retrieve the file - over the network. -\item Write a file specified by a URI --- it may simply write the file - if it exists on the particular node, or it may copy the file over - the network. It should also register with the IPSD that a file - specified by a URI has been written. -\item Delete a file specified by a URI --- it may simply delete the - file if it exists on the particular node, or it may delete the file - over the network. -\item Interface with the IPSD to return a URI given one of the - identifiers in \S\ref{sec:ipsd}. -\end{itemize} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Pixel Data Flow Examples} - -For examples of the operation of the IPS, below we sketch out the -intended sequence of events for common operations. - -Reads during processing: -\begin{enumerate} -\item A processing stage has been passed (from the Scheduler) the URI - for an image that it needs to load into memory. -\item The processing stage uses the IPSIOL to open the image. -\item The processing stage reads the image into local memory in the - usual manner. -\item The processing stage closes the image using the IPSIOL. -\end{enumerate} - -Writes during processing: -\begin{enumerate} -\item A processing stage has been passed (from the Scheduler) the URI - for an image that needs to be saved, e.g., a subtracted image. -\item The processing stage uses the IPSIOL to open the image. -\item The processing stage writes the image in the usual manner. -\item The processing stage closes the image using the IPSIOL. -\end{enumerate} - -Note how the IPSIOL has insulated the processing stage from the details -of the reading and writing. - -Maintenance: -\begin{enumerate} -\item A human operator decides that all the pixel data for chip 12 - should be stored on node 3. -\item Operator plugs this into the IPSM. -\item The IPSM queries the IPSD using the IPSIOL. -\item The IPSD returns the URIs for all the pixel data for chip 12. -\item The IPSM generates processing tasks to be executed on the nodes - that will copy the data from the old URIs to a new URI which - specifies node 3. -\item The IPSM generates processing tasks to be executed on the nodes - that deletes the data pointed to by the old URIs. -\item The IPSM reports success to the operator. -\end{enumerate} - -Client Science Pipelines: -\begin{enumerate} -\item A CSP wants some pixel data. -\item The CSP queries the IPSD using the IPSIOL (e.g., asking for a - particular exposure or sky cell). -\item The IPSD returns the URI for the pixel data. -\item The CSP opens the image using the IPSIOL and the URI. -\item The CSP reads the pixel data into memory in the usual manner. -\item The CSP closes the image using the IPSIOL. -\end{enumerate} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Metadata Database} - -The IPP Metadata Database acts as a repository for all non-pixel data -needed by the IPP subsystems. This includes the image metadata, the -environmental data, system configuration data and system reference -data. The Metadata Database is required to save the non-ephemeral -data for the lifetime of the project for future reference and -additional analysis. The Metadata Database may potentially be used in -close coupling with the analysis pipelines to store temporary data -either within or between stages of the analysis. In this scenario, -the analysis pipeline will interact directly with the database. -However, database latency may make this scenario impractical, in which -case the database may be used for long-term storage only. In this -scenario, the data produced by analysis pipelines which is destined -for the Metadata Database may be collected and inserted by a separate, -dedicated process or analysis pipeline collection of processes. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Metadata Tables} - -Table \tbd{NN} lists the Metadata tables identified for the Metadata -Database. - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Metadata Tables} \\ -Weather & Details on the weather, including internal temperatures. \\ -SkyProbe & Analysis of SkyProbe data. \\ -LRProbe & Analysis of LRProbe data. \\ -DIMM & Analysis of DIMM data. \\ -NIR & Analysis of NIR data. \\ -Dome Status & The status of the dome. \\ -Telescope Status & The status of the telescope. \\ -Raw FPAs & Details on raw FPA exposures. \\ -Raw Chips & Details on raw chips. \\ -Raw Cells & Details on raw cells. \\ -Observation Group & Details on a group of observations to be processed. \\ -Chip Guide Stars & Details on guide stars \\ -Science Chip stats & Details on processed chips. \\ -Science Cell stats & Details on processed cells. \\ -Science FPA stats & Details on processed FPAs. \\ -Sky-Detector overlaps & List of overlaps between sky cells and detectors. \\ -Processed Sky-Cell stats & Details on sky cells. \\ -Calibration 1 input stats & Details on input images for Cal 1. \\ -Calibration 1 output stats & Details on output detrend images from Cal 1. \\ -Calibration 2 input stats & Details on input images for Cal 2. \\ -Calibration 2 output stats & Details on output detrend images from Cal 2. \\ -Calibration 3 input stats & Details on input images for Cal 3. \\ -Calibration 3 output stats & Details on output detrend images from Cal 3. \\ -\hline -\end{tabular} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Metadata Table Contents} - -Tables \tbd{NN} -- \tbd{NN} list the basic contents of each of the Metadata tables -listed above. - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Weather} \\ -Time & The time the weather information was measured. \\ -Temperature & The temperature at \tbd{some place. Will likely want temperatures for a range of locations: -external, dome, secondary, primary for starters.} \\ -Humidity & The relative humidity. \\ -Pressure & The (external) atmospheric pressure. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf SkyProbe} \\ -Time & The time the SkyProbe image was taken. \\ -Filter & Filter used for SkyProbe image. \\ -Transparency & The derived transparency. \\ -Error in transparency & The error in the derived transparency. \\ -Number of stars & The number of stars used to measure the transparency. \\ -Astrometry & The astrometry used on the SkyProbe image. \\ -Exposure time & The exposure time of the SkyProbe image. \\ -Sky brightness & The measured sky (surface) brightness, in physical units. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf LRProbe} \\ -Time & The time the LRProbe observation was taken. \\ -A band absorption & The absorption EW of the atmospheric A band. \\ -B band absorption & The absorption EW of the atmospheric B band. \\ -Absorption component 3 & The absorption EW by some other atmospheric component. \\ -Emission 1 & The emission EW of some sky line. \\ -emission 2 & The emission EW of another sky line. \\ -emission 3 & The emission EW of some other sky line. \\ -Number of stars & Number of stars used to measure the absorptions. \\ -Astrometry & The astrometry used on the LRProbe image. \\ -Exposure time & The exposure time of the LRProbe image. \\ -Sky brightness & The measured sky (surface) brightness, in physical units. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf DIMM} \\ -Time & The time the DIMM observation was taken. \\ -$\sigma_x$ & \tbd{The dispersion in $x$}. \\ -$\sigma_y$ & \tbd{The dispersion in $y$}. \\ -FWHM & The seeing full width at half maximum. \\ -Star coordinates & The coordinates of the measured star. \\ -Exposure time & The exposure time of the DIMM observation. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf NIR} \\ -Time & The time the NIR observation was taken. \\ -Sky brightness & The sky (surface) brightness in the NIR observation. \\ -Sky variance & The variance in the sky (surface) brightness. \\ -Astrometry & The astrometry used on the NIR image. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Dome Status} \\ -Time & The time for which the dome status is valid. \\ -Azimuth & The azimuth of the dome. \\ -Open status & Whether the dome is open or not. \\ -Lights status & Whether lights are on in the dome or not. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Telescope Status} \\ -Time & The time for which the telescope status is valid. \\ -Guide status & The status of the guiding. \\ -Altitude & The telescope altitude. \\ -Azimuth & The telescope azimuth. \\ -RA & The telescope Right Ascension (ICRS $\approx$ J2000). \\ -Dec & The telescope Declination (ICRS $\approx$ J2000).\\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Raw FPAs} \\ -Coords & Coordinates of the boresight (i.e. telescope pointing). \\ -Filter & Filter used for the exposure. \\ -Exposure status & Status of the exposure. \\ -Exposure time & Exposure time for the image. \\ -Airmass & Airmass at which the image was taken. \\ -ObsGroup ID & \tbd{The ObsGroup identification number.} \\ -Observer & The name of the observer, or the version of the telescope scheduler software. \\ -Program & The observing program being executed. \\ -Number of chips & The number of chips that comprise the FPA. \\ -NX, NY & \tbd{Assuming the chips are laid out rectilinearly,} the number of chips in each dimension. \\ -Astrometry & The astrometry used for the FPA. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Raw Chips} \\ -i, j & \tbd{Assuming a rectilinear FPA,} the chip number in each dimension. \\ -ID & Chip identification number. \\ -temps & The chip temperature. \\ -Astrometry & The astrometry used for the chip. \\ -Number of cells & The number of component cells. \\ -NX, NY & \tbd{Assuming the cells are rectilinear,} the number of cells in each dimension. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Raw Cells} \\ -Astrometry & The astrometry used for the cell. \\ -Validity & Is the cell working? \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Observation Group} \\ -ID & Identification number for the observation group. \\ -Number of images & Number of images in the observation group. \\ -Type & Type of observation. \\ -Status & Status of the observation group. \\ -\tbd{etc} & \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Chip guide stars} \\ -Chip ID & The identification number for the chip. \\ -Guide Star ID & The identification number for the guide star. \\ -X, Y & The centroided pixel coordinates of the guide star. \\ -RA, DEC & The sky coordinates of the guide star. \\ -$\sigma_{x}$, $\sigma_{y}$ & The dispersion in the centroids over the particular exposure.\\ -$\Delta X_{\rm max}$, $\Delta Y_{\rm max}$ & The maximum deviation in the centroid over the -particular exposure. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Science Chip stats} \\ -Chip ID & The chip identification number. \\ -State & \tbd{The state of the processing.} \\ -Major frame & \tbd{The major frame the chip belongs to.} \\ -ObsGroup & The observation group the science exposure belongs to. \\ -P1 astrom & The Phase 1 astrometry. \\ -P2 astrom & The Phase 2 astrometry. \\ -P3 astrom & The Phase 3 astrometry. \\ -Number of guide stars & Number of guide stars used for the exposure. \\ -Bias correction method & Method used to correct the bias. \\ -Bias stats & Summary statistics for bias (mean, number of parameters, deviation of residuals, -bias section used). \\ -Flat-field image & The flat-field image that was applied. \\ -Kernel convolution parameters & A description of the OT kernel. \\ -Flat-field stats & Summary statistics for flat-field (sigma of sky). \\ -Mask image & The mask image that was applied. \\ -Masking algorithm & \tbd{The algorithm used to mask the bad pixels.} \\ -Fringe images & The fringe model images that were used. \\ -Fringe stats & Summary statistics for fringes (fringe amplitude, sky sigma) \\ -Object detection stats & Summary statistics for object detection (number of objects, depth, other -input parameters). \\ -Updated astrometry & \tbd{Updated astrometry parameters.} \\ -Astrometry stats & Summary statistics for astrometry (number of stars, $sigma_x$, $sigma_y$) \\ -Reference catalog & The reference catalog that was used for the astrometry. \\ -Updated photometry parameters & The parameters used to update the photometry: magnitude zero point -and other corrections. \\ -Photometry stats & Summary statistics for the photometry (number of stars, $sigma_m$) \\ -Reference catalog & The reference catalog that was used for the photometry. \\ -PSF stats & Summary statistics of the PSF. \\ -Chip state & \tbd{The state of the chip?} \\ -Software versions & Versions of each of the modules used in the processing. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Science Cell stats} \\ -Bias stats & Summary statistics for the bias (mean, parameters, dispersion of residuals, biassec) \\ -P1 astrom & The Phase 1 astrometry. \\ -P2 astrom & The Phase 2 astrometry. \\ -P3 astrom & The Phase 3 astrometry. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Science FPA stats} \\ -FPA ID & The FPA identification number. \\ -State & \tbd{The state of the FPA.} \\ -P1 astrom & The Phase 1 astrometry. \\ -P1 astrom stats & Summary statistics for the Phase 1 astrometry (number of stars, $\sigma_x$, $sigma_y$). \\ -P1 reference catalog & The reference catalog that was used for the astrometry. \\ -P1 software versions & The versions of each of the modules used in the Phase 1 processing. \\ -P1 bright stars & Pointers to the bright stars in the field. \\ -P1 ghosts & Pointers to the ghosts in the field. \\ -P1 large objects & Pointers to the large astronomical objects in the field. \\ -P1 PSF model & Description of the PSF model used in Phase 1. \\ -P3 astrom & The Phase 3 astrometry. \\ -P3 astrom stats & Summary statistics for the Phase 3 astrometry (number of stars, $sigma_x$, $sigma_y$). \\ -P3 reference catalog & The reference catalog that was used for the astrometry. \\ -P3 photom & The Phase 3 photometry. \\ -P3 photom stats & Summary statistics for the Phase 3 photometry (number of stars, $sigma_m$). \\ -P3 reference catalog & The reference catalog that was used for the photometry. \\ -P3 PSF model & Description of the PSF model used in Phase 3. \\ -P3 software versions & The versions of each of the modules used in the Phase 3 processing. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Sky-Detector overlaps} \\ -Chip ID & The identification number of the chip. \\ -Sky Cell ID & The identification number of the sky cell. \\ -State & \tbd{The state of the processing?} \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Processed Sky-Cell stats} \\ -Input Chips & Identification numbers of the chips used to produce the sky cell. \\ -PSF adjustments & \tbd{Adjustments to the PSF.} \\ -CR rejection stats & Statistics from the CR rejection (number of CRs, distribution, limiting flux). \\ -Image combination parameters & Parameters used for the image combination. \\ -Difference image parameters & Parameters used for the image differencing. \\ -Average reference image depth / weight & \tbd{The weight of the reference image?} \\ -Difference image object detection stats & Summary statistics of the object detection (number of objects, -depth, other input parameters). \\ -Summed image object detection stats & Summary statistics of the object detection (number of objects, -depth, other input parameters). \\ -Software versions & Software versions of modules used in the sky cell processing. \\ -Processing stats & Summary statistics of the processing (CPU time, etc). \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Calibration 1 input stats} \\ -Input ID & The input chip identification number. \\ -Output ID & The identification number of the output detrend image. \\ -State & \tbd{State of the processing?} \\ -Accepted? & Is the detrend image of acceptable quality? \\ -Image stats & Assorted image statistics (mean flux, exposure time, airmass) \\ -Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Calibration 1 output stats} \\ -Output ID & The identification number of the output detrend image. \\ -Data type & The type of the detrend image (bias | dark | flat) \\ -Number accepted & Number of accepted input images that contributed. \\ -Number rejected & Number of rejected input images (no contribution). \\ -Summary stats & Summary statistics of the combination (deviation, normalisations). \\ -Applicability period & The time period the detrend image is applicable for. \\ -Software versions & The software versions of the modules used in processing. \\ -Processing stats & Summary statistics of the processing (CPU time, etc). \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Calibration 2 input stats} \\ -Input ID & The input chip identification number. \\ -Output ID & The identification number of the output detrend image. \\ -State & \tbd{State of the processing?} \\ -Accepted? & Is the detrend image of acceptable quality? \\ -Image stats & Assorted image statistics (mean flux, exposure time, airmass). \\ -Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ -Applied reduction & \tbd{Reduction method used?} \\ -Applied params & Parameters of reduction. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Calibration 2 output stats } \\ -Output ID & The identification number of the output detrend image. \\ -Data type & The type of the detrend image (bias | dark | flat) \\ -Number accepted & Number of accepted input images that contributed. \\ -Number rejected & Number of rejected input images (no contribution). \\ -Summary stats & Summary statistics of the combination (deviation, normalisations). \\ -Applicability period & The time period the detrend image is applicable for. \\ -Software versions & The software versions of the modules used in processing. \\ -Processing stats & Summary statistics of the processing (CPU time, etc). \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Calibration 3 input stats} \\ -Input ID & The input chip identification number. \\ -Output ID & The identification number of the output detrend image. \\ -State & \tbd{State of the processing?} \\ -Accepted? & Is the detrend image of acceptable quality? \\ -Image stats & Assorted image statistics (mean flux, exposure time, airmass). \\ -Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ -Applied reduction & \tbd{Reduction method used?} \\ -Applied params & Parameters of reduction. \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Calibration 3 output metadata } \\ -Input images & Identification numbers of the input chips. \\ -Input image stats & Summary statistics of the input chips. \\ -Input object summary stats & Summary statistics of the objects on the input chips (number, density, etc) \\ -Object rejection criteria & Parameters of the rejection step. \\ -Phot stats & Summary statistics of the relative photometry (Mcal, dMcal, Klam, etc, bin size) \\ -Residual stats & Summary statistics of the residuals. \\ -Output image params & Parameters of the output image (size, etc) \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Astrometric Reference Generation output metadata } \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{1}{l}{\bf Photometric Reference Generation output metadata } \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Reference Data} \\ -\hline -\end{tabular} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Configuration Data} \\ -\hline -\end{tabular} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Metadata Queries} - -\tbd{How is the Metadata DB queried?} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Object Database} - -The IPP Object Database (IOD) acts as a repository for data on all -astronomical objects. This database is required to provide organized -access to objects on the sky, including the access to the photometry -associated with specific input images, moving objects associated with -specific chips. Detailed requirements for the IOD are described in -\tbd{the IOD subsystem specification document xxx-xxx-xxxx}. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Object DB Tables} - -\begin{tabular}{ll} -\hline -\multicolumn{2}{l}{\bf Object DB Tables} \\ -Images & The images that have objects in the DB. \\ -Objects & The objects --- average properties of multiple detections of the same object. \\ -Detections & Detections of sources in an image. \\ -Non-Detections & Non-detections of objects in an image. \\ -Filters & Filters understood by the system. \\ -Photcodes & \tbd{Transformations between different photometric systems?} \\ -Bright Objects & \tbd{Links to postage stamp images of bright objects.} \\ -Region Tables & \tbd{???} \\ -Average Magnitudes & \tbd{How is this different from an `object'?} \\ -USNO Objects & Objects from the USNO database. \\ -Reference Objects & The reference catalogs for astrometry and photometry. \\ -\hline -\end{tabular} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Object DB Table Contents} - -\tbd{Dunno yet} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Object DB Queries} - -\tbd{Dunno yet} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Controller} - -The IPP Controller is responsible for managing the processing stages. -The Controller manages the parallel processing of these stages in the -IPP computer hardware environment and reports the completion to the -Scheduler. The Controller must be able to manage more than a single -processing thread to make maximum use of available processor -resources. - -The Controller must honour demands that a processing stage run on a -particular Node. Requests that a processing stage run on a particular -node should be honoured if possible. Where no restriction is placed -on the choice of Node choice by the Scheduler, the processing stage -may be run on any available Node. - -The Controller maintains a table of processing nodes available to it -and the status of these Nodes. When the Controller starts, it -attempts to launch a Node Agent on each of the available processing -nodes. Modes which are not responsive are placed into an inactive -state and retried occasionally. - -The Controller also maintains three tables of processing jobs: pending -stages, active stages, and completed stages. The pending stages are -those which have not yet been performed. The active stages are those -currently being performed on one of the remote nodes. The completed -stages are those which have finished, either successfully or with an -error state. The Controller daemon monitors the collection of remote -clients and sends them new pending stages when they become free. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Node Agents} - -A Node Agent runs on each of the individual nodes to perform the -processing stages as directed by the Controller. The Node Agents -communicate with the Controller via a socket connection. - -A processing stage is executed in the UNIX user space, and is run as a fork by the -Node Agent. The Node Agent must monitor the standard error and -standard output of the processing stage and save them in separate buffers. If the -process dies, the Node Agent must detect the crash. The Node Agent -must respond to various commands from the Controller. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Report status} - -The Node Agent returns the state of the Node (idle, busy, done), the -state of the current processing stage\footnote{Note that a processing -stage is considered ``current'' until it is cleared with {\em clear -processing stage} --- even if it has crashed or completed.} (`none', -`busy', `crash', `done'), and the exit status of the current -processing stage (`none', 0--256). - -The three states of the Node indicate that the client has no current -processing stage (`idle'), that it has a processing stage which is -still running (`busy'), or that it has a processing stage which has -completed. - -The processing stage states indicate the there is no current -processing stage (`none'), that the current processing stage is -running (`busy'), that the current processing stage has crashed -(`crash'), or that the current processing stage has exited gracefully -(`done'). The exit state is the exit state reported by the processing -stage (0--256 with 0 indicating a successful completion) or is an -indication that there is no current processing stage (`none'). - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Report stdout} - -Send and flush the current stdout buffer. The Node Agent will return -the complete contents of the stdout buffer via a buffered write and -flush the buffer when it is finished. The Node Agent will not accept -more data on the stdout buffer from the current processing stage until -the send is complete and the buffer is flushed. The daemon must -accept all of the buffer output. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Report stderr} - -Identical to `report stdout', but for stderr. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Kill processing stage} - -The Node Agent should send a kill signal to the current processing -stage. When the processing stage has exited, the Node Agent should -set the processing stage status to `crash' and the Node status to -`done'. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Clear processing stage} - -The Node Agent should set the current processing stage state to `none' -and the Node state to `idle'. If a processing stage is currently -running, it should be killed before the processing stage is cleared. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Start processing stage} - -The Node Agent forks a specified command. The command should be a -standard UNIX command without command line redirection or -backgrounding. For this reason, the Node Agent must provide a layer -of security, for example, by employing SSL authentication. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Matrix} - -\tbd{The Node Agent does not wear a suit, nor does it know kung fu.} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Scheduler} - -The IPP Scheduler is responsible for initiating the various processing -stages (which are executed by the IPP Controller), based on the state -of the survey as reflected by the IPP Metadata Database (IMD). - -The Scheduler shall maintain a list of processing stages, as well as -the required input and dependencies for each of the processing stagesFor example, the -dependencies for copying pixel data from OATS may be: -\begin{itemize} -\item OATS has new pixel data available; -\item The new pixel data has not been copied. -\end{itemize} -Similarly, the dependencies for executing Phase 2 processing on a chip -may be: -\begin{itemize} -\item The chip pixel data has been copied. -\item Phase 1 has run successfully on the metadata for the FPA to which - the chip belongs. -\item A reduced image (i.e., output from Phase 2) does not already - exist. -\end{itemize} - -When the dependencies are satisfied, the Scheduler shall prepare for -execution the particular processing stage on the appropriate data. -The Scheduler must query the Metdata DB for the most appropriate -calibration data, if required. The processing stage should be -filtered through the IPSDLO in order to assign the processing stage to -a particular Node (if desired) and to determine the URIs for the -required inputs. The processing stage is then passed to the -Controller. - -The Scheduler must also be able to send requests for new calibration -data to OATS, including required flat-fields, flat-field correction -observations, or other specialized observations needed to improve the -calibrations. The Scheduler must balance the need for improved -calibrations with the need to process the science images in a timely -manner given the capabilities of the science pipelines. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{System UI} - -A user interface allows a human operator to monitor the Controller and -Scheduler through some user interface (UI). The System UI may -interact with the Controller and Scheduler via a socket connection -using a defined set of commands. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Execute processing stage} - -A new processing stages is sent to the Scheduler. The Scheduler may -filter the processing stages through the IPSDLO, or it may be -prevented from doing so by the user. The Scheduler then passes the -processing stages to the Controller for execution. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Kill processing stage} - -The user may kill an existing processing stage. The Controller is -commanded to kill the particular processing stage. - -\tbd{Should we allow a System UI to kill processing stages sent by -other System UIs?} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Get status} - -The System UI may request the current status of the Controller, -including the list of pending, active, and completed processing stages -and the status of the individual processing stages. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Available Nodes} - -The System UI may view and configure the list of Nodes available to -the Controller (e.g., to remove a Node temporarily for maintenance). - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Processing Stages} - -In this section, we review the processing stages which are executed on -the Nodes. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Overview} - -The processing stages are the software that process data. These -processing stages are divided into five categories which are -summarised in \S\ref{sec:processingStages}. Each of the processing -stages are described below. - -The processing stages are initiated by the Scheduler, parallized and -managed by the Controller, and executed through the Node Agents on the -nodes. Processing stages are purely serial, and so they may be run on -a single node at once without the need for interprocess communication. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Retrieval} - -The retrieval stages simply retrieve pixel data from an external -source (ordinarily OATS at the Summit, but it could conceivably be -some other external source) and store it on the nodes. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Science Image Processing} - -The IPP science image processing stages perform analyses on the -night-sky science images to extract the science data from these -images. These consist of: Phase 1, the image processing preparation -stage; Phase 2, the image reduction stage; Phase 3, the exposure -analysis stage; and Phase 4, the image combination stage. These -pipelines must process the images in a timely manner so that the -incoming data stream will not overload the IPS. The decision to -execute a specific pipeline for a specific dataset is made by the -Scheduler, which sends the infomation to the Controller. The -Controller executes the pipeline for the data on an appropriate -machine and monitors the success or failure of the processing stage. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Phase 1: image processing preparation} - -The Phase 1 system operates on data from each FPA to calculate basic -astrometric information needed by other stages of the analysis. The -analysis includes: - -\begin{itemize} -\item preliminary astrometry based on the guide-star centroids -\item sky-cell / detector-cell overlaps -\end{itemize} - -The input to this analysis is the list of guide-star pixel centroids -and their celestial coordinates as saved in the metadata database, as -well as the FPA and chip organization and geometry, and the basic -optical distortion for the camera. For the sky-cell / detector-cell -overlaps, the sky tiling scheme is required. - -The output consists of calculated astrometric parameters (linear -transformation + static distortion) for each of the FPA chips. On the -basis of this astrometry, the overlap between the detectors and the -sky-cells is calculated. The output of this calculation is a list of -sky-cell / chip links in a database table. This list of links can be -used by the later stages to initiate the analyses. - -The phase 1 analysis is performed on an FPA basis to ensure that -enough reference stars are available for the astrometry calculation. -Phase 1 cannot be usefully calculated on the basis of a major frame -since the telescope positions are independent; no additional -information is available by combining stars from different FPAs. This -analysis does not restrict the definition of a major frame in any way. - -\tbd{Phase 1 command: P1 (exposure)} - -\tbd{Megacam: P1 654321o} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Phase 2 : image reduction : new version} - -\tbd{how long are processed images kept?} - -\tbd{what subsystem deletes processed images?} - -\tbd{does 'remove' mean 'mask' or 'replace'} - -\tbd{what is the absolute astrometry accuracy at phase 2? 0.1 arcsec -== 0.33 pix?} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Concept} - -Phase~2 processing within the \PS{} image processing pipeline is -the de-trend stage, where the images from the detector are processed -to remove instrumental signatures. - -\begin{figure} -\begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/phase2}} -\caption{ \label{phase2} Phase 2 dataflow} -\end{center} -\end{figure} - -Prior to Phase~2, the Phase~1 process operates on an entire telescope -Focal Plane Array to set the boresight astrometric solution using -the guide stars and initial masking of ghost reflections. - -Phase~2 consists of the following modules: -\begin{enumerate} -\item Form OT kernel; -\item Convolve de-trend images with the OT kernel; -\item Mask bad pixels -\item Mask diffraction spikes and optical ghosts; -\item Bias/dark/overscan subtraction; -\item Trim overscan; -\item Non-linearity correction; -\item Flat-field; -\item Subtract sky; -\item Identify CRs by morphology; -\item Determine PSF model; -\item Find and photometer objects in the image; -\item Improved astrometry; and -\item Bright object postage stamps. -\end{enumerate} -These modules are each explained below. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Form OT Kernel} - -The first module for Phase~2 is to form the OT kernel from the image -metadata of pixel shifts made during the exposure. This involves -decoding the metadata and converting it to a data type that can be -used to convolve by. The output is the OT convolution kernel. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Convolve de-trend images} - -This module convolves the de-trend images with the OT convolution kernel -so that they can be used to de-trend the object image. The inputs -are: -\begin{enumerate} -\item The OT convolution kernel --- from the previous module; -\item The appropriate dark frame --- from the IPP Pixel Server; -\item The appropriate flat-field --- from the IPP Pixel Server; -\item The appropriate fringe frame(s) --- from the IPP Pixel Server; and -\item The appropriate static bad pixel mask --- from the IPP Pixel Server. -\end{enumerate} - -The module convolves each of the dark frame, flat-field, and the fringe -frame(s) by the OT convolution kernel. Specific flags in the static -bad pixel mask are grown by the outline of the OT convolution kernel -(see Section \ref{ap:masks}). The output results are: -\begin{enumerate} -\item The convolved flat-field; -\item The convolved fringe frame(s); and -\item The updated pixel mask. -\end{enumerate} -Each of these will be used for a later module. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Overscan Subtraction} - -This module corrects the object exposures for the electronic pedestal -introduced by the readout electronics. The inputs are: -\begin{enumerate} -\item The object image --- from the IPP Pixel Server; -\item The pixel mask --- from the previous module; -\item The overscan and physical detector regions --- from the -Metadata; and -\item Detector characteristics (gain, read noise) --- from the -Metadata. -\end{enumerate} - -The overscan is averaged (either in bulk, or individually by rows) or -fit with a polynomial, and the result is subtracted from the image. -Overscan rows having a standard deviation which exceeds a threshold of -twice (configurable) the detector read noise should be masked. Pixels -saturated in the A/D converter should also be masked, and these -regions grown by an additional pixel to counter CCD ``blooming''. The -output is: -\begin{enumerate} -\item The overscan-subtracted object image; and -\item The updated pixel mask. -\end{enumerate} -These will be used for a subsequent module. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Trim} - -This module trims the object image and each of the calibration frames to -remove the outer edge which was affected by the OT during the -exposure. The inputs, each from previous modules, are: -\begin{enumerate} -\item The overscan-subtracted object image; -\item The corresponding pixel mask; -\item The convolved flat-field; -\item The convolved fringe frame(s); and -\item The dimension of the OT convolution kernel in each direction. -\end{enumerate} - -Each of the input frames (object image, flat-field, fringe frame(s) -and pixel mask) are trimmed by the extent of the OT convolution kernel -in each direction ($+x$, $-x$, $+y$, $-y$). The outputs are trimmed -images for each of the input images, which will be used in later -modules. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Non-Linearity Correction} - -This module corrects images for non-linearity in the detector. The -inputs are: -\begin{enumerate} -\item The trimmed object image --- from a previous module; and -\item The detector non-linearity correction coefficient(s) --- from -the Metadata. -\end{enumerate} - -The module corrects the flux in each pixel for non-linearity by applying -a polynomial correction, with the specified coefficients. The output -is the corrected object image, which is used for a later module. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Flat field} - -This module corrects the object image for variations in sensitivity over -the image. The inputs are: -\begin{enumerate} -\item The object image corrected for non-linearity; -\item The corresponding pixel mask; and -\item The convolved, trimmed flat-field. -\end{enumerate} -Each of these comes from a previous module. - -The module divides the object image by the flat-field, masking pixels -that are non-positive in the flat-field. The outputs are: -\begin{enumerate} -\item The flattened object image; and -\item The updated pixel mask. -\end{enumerate} -Both of these will be used in later modules. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Subtract sky} - -This module subtracts the sky background from the object image. The -inputs are: -\begin{enumerate} -\item The object image --- from the previous module; -\item The list of objects on the image --- from the object database; and -\item The convolved, trimmed fringe frame(s) --- from a previous module. -\end{enumerate} - -The module masks (though {\em not} in the ``official'' pixel mask) all -objects on the image using the astrometric solution from the -boresight, and fits for the sky background, consisting of a polynomial -to model the continuum, and the fringe frame(s) to model the fringes -from sky emission lines. If the concentration of objects in the image -is too high to reliably fit the sky background, the background -solution from an exposure close in time and airmass to the current -object image is used. The output is the sky-subtracted object image, -which is used for the next module. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Identify CRs by morphology} - -This module identifies cosmic rays (or other hot pixels missed in the -static bad pixel mask) on the basis of their morphology. The inputs -are: -\begin{enumerate} -\item The object image; and -\item The corresponding pixel mask. -\end{enumerate} -Both of these come from a previous module. - -The module identifies CRs, the pixels of which are masked in the pixel -mask. The pixels flagged as CRs are then grown by an additional pixel -in each direction. Masked pixels are interpolated over. The outputs -are the updated pixel mask, which is sent to the IPP pixel server for -use in Phase~3, and is also used for the next module; and the object image, -which is sent to the IPP Pixel Server. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Find objects} - -This module finds objects on the object image. The inputs are: -\begin{enumerate} -\item The sky-subtracted object image; and -\item The corresponding pixel mask. -\end{enumerate} -Both of these come from a previous module. - -The module identifies objects on the image, which will be later used to -register images from different focal planes. The output is the -catalog of objects (see Appendix~\ref{ap:catalogs}) identified on -the image, which is sent to the metadata database, associated with the -object image. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Bright object postage stamps} - -This module saves postage stamps of bright objects, so that extra care -with regard to astrometry and photometry can be taken with them at a -later stage. The inputs, each from a previous module, are: -\begin{enumerate} -\item The sky-subtracted object image; -\item The corresponding pixel mask; and -\item The catalog of objects. -\end{enumerate} - -The module makes postage stamps of all objects brighter than a given -instrumental magnitude, along with corresponding pixel masks. The -outputs are these postage stamps and pixel masks, which are sent to -the IPP Pixel Server. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Metadata Required} - -The following metadata associated with the images are required for -Phase~2 operation: -\begin{itemize} -\item The orthogonal transfer (OT) image shifts made during the -exposure --- in order to create a convolution kernel; -\item Time of observation --- for selecting the appropriate detrend -images; -\item Filter --- for selecting the appropriate detrend images; -\item Telescope identification --- for selecting the appropriate -detrend images; -\item Exposure time --- for the photometric calibration; -\item Detector gain --- for calculating photometric errors; and -\item Detector read noise --- for calculating photometric errors and -determining the quality of the overscan; -\end{itemize} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Pixel Masks} -\label{ap:masks} - -This section describes the requirements on Bad Pixel Masks (BPMs). -These will consist of bit masks for each pixel. For Phase 2, flags -are required for at least each of the following pixel attributes: -\begin{enumerate} -\item The pixel is a charge trap; -\item The pixel is a bad column; -\item The pixel is saturated in the A/D converter; -\item The pixel is non-positive in the flat-field; -\item The pixel is part of a row that has excess noise; and -\item The pixel is determined to be a cosmic ray, based on its -morphology. -\end{enumerate} - -Of these, only masks for the charge traps need to be grown by the -extent of the OT convolution kernel. For other pixel types, -orthogonal transfer of the flux in this pixel will not (necessarily) -affect the flux in neighbouring pixels - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Object Catalogs} -\label{ap:catalogs} - -Object catalogs from Phase 2 shall consist of at least the -following elements for each object: -\begin{enumerate} -\item Object centre, with corresponding errors; -\item Object magnitude, with corresponding error; -\item Object isophotal magnitude, with corresponding error; -\item Object FWHM; -\item Object elliptical axis lengths; and -\item Object position angle for ellipse. -\end{enumerate} - -Though further details may be required for catalogs in Phase~4, -the above details are minimum requirements for Phase~2 catalogs. - -\tbd{Phase 2 command: P2 (exposure.ota.fits)} -\tbd{Megacam: P2 654321o.fits[ccd00] - what are output names?} -\tbd{PS FPA is saved as a collection of MEF files. Megacam FPA is - saved as a single MEF file. how to handle this difference?} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Phase 3 : exposure analysis} - -The Phase 3 system operates on the combined Phase 2 results from an -FPA to determine improved solutions for the image calibrations and to -provide the parameters needed by Phase 4. The Phase 3 output is saved -by the IMD, and consists largely of improved values of the -calibrations already determined by Phase 2. The analysis performed by -this pipeline consists of: - -\begin{itemize} -\item improved astrometric solution based on comparison between - objects in the images and the astrometric reference. -\item improved background model based on the full telescope field, or - fields. -\item photometric solution based on comparison to photometric - standards -\end{itemize} - -\begin{figure} -\begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/phase3}} -\caption{ \label{phase3} Phase 3 dataflow} -\end{center} -\end{figure} - -In the Phase 2 analysis, the astrometric solutions were determined -independently for each chip. These solutions are limited by the -assumption of a static distortion and \tbd{by the accuracy of the -astrometric reference}. In the phase 3 analysis, the astrometric -solutions of the $N$ FPA images are improved by \tbd{???}. - -\tbd{what is the expected accuracy of the relative astrometric - solution compared to the absolute astrometric solution?} - -\tbd{for image combination in phase 4, should we use relative - astrometry to map N-1 images to 1, or are we sufficiently accurate - to use absolute astrometry to map N images to the sky-cells?} - -In the Phase 2 analysis, the background is determined based only on -the available sky in a single chip. However, the background -structures are normally correlated on the scale of the FPA, so an -improved background solution can be determined by combining the -information from many chips. \tbd{is the background correlated -between FPAs?} - -\tbd{Phase 3 photometric improvement??} \tbd{Phase 3 determined -accurate relative photometry between the N images which are to be -combined in the Phase 4 analysis. Is this more accurate than the -absolute photometry solution? (probably)} - -In the Phase 4 analysis, the $N$ FPA images are optimally combined to -create a single image of the sky with bad-pixel and cosmic-ray -rejection. This combination requires the calculation of a set of PSF -kernels to convert each of the input images to a single, common PSF. -These PSF kernels are determined from the per-chip PSFs measured in -Phase 2. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Phase 4 : image combination} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Phase 4 Concept} - -Phase 4 processing within the \PS{} image processing pipeline is -the final stage of processing for a science image. It operates on -each sky cell that has overlapping imaging data from the exposure(s) -being processed, and produces the main output image data products of -the pipeline --- the difference images and a deep static sky image --- -along with the associated catalogs of static and variable sources. - -\begin{figure} -\begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/phase4}} -\caption{ \label{phase4} Phase 4 dataflow} -\end{center} -\end{figure} - -Prior to Phase 4, the Phase 3 process produces the following products: -\begin{itemize} -\item bias-subtracted, flattened, sky-subtracted images; -\item photometric calibration; -\item astrometric calibration with mapping to sky cells; and -\end{itemize} -These will each be used by the Phase 4 modules: -\begin{enumerate} -\item Combine Images; -\item Identify Sources; -\item Transient Identification; and -\item Add to Static Sky. -\end{enumerate} -These modules are each explained below. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Combine Images} - -The first module for Phase 4 is to combine the images from each -telescope, rejecting artifacts such as cosmic rays and low altitude -streaks. The inputs to this module are: -\begin{enumerate} -\item the sky-subtracted images that overlap the sky cell (or portions -thereof) --- from the IPP Pixel Server (or directly from Phase 3); -\item a \tbd{linear} map for the image pixels of each detector to the -sky cell pixels --- from Phase 3; -\item photometric calibration (zeropoint) for each image --- from -Phase 3; and -\item a (relative) weighting for each image proportional to the -signal-to-noise (i.e.\ sky noise divided by the square of the seeing) ---- from metadata associated with the images. -\end{enumerate} - -The module maps the detector images to the sky cell using the specified -linear transformations, combines the images with strong rejection -criteria and uses the combined sky cell image to identify artifacts in -the original detector images. It is desirable that the artifacts are -masked in the detector plane (i.e.\ before mapping to the sky cell) so -that they are not smeared out by the mapping; alternatively, the CR -mask needs to be grown by an additional pixel (which is likely -faster). The mapped and masked detector images are then optimally -combined using the specified weighting. Both sets of combinations use -the photometric calibration for the images to set the relative scales -of the input images. The final combination should have the adopted -Universal zeropoint (25 mag, configurable). The limiting magnitude -for the combined sky cell image should also be estimated. - -The outputs from this module are: -\begin{enumerate} -\item The combined sky cell image --- sent to the IPP Pixel Server -and/or the next module; -\item Limiting magnitude of the combined sky cell image --- metadata -associated with the combined sky cell image, and used for a later module -in Phase 4; and -\item Catalog of sources on the combined sky cell image --- sent to -the IPP Object Database. -\end{enumerate} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Identify Sources} - -This module identifies sources in the combined sky cell image. The -input is the combined sky cell image, which is obtained from the IPP Pixel Server -or the previous module. - -Sources are identified on the combined sky cell image by convolving -with the PSF and searching for peaks above the noise. The output -is the catalog of sources on the combined sky cell image, which is to -the IPP Object Database. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Transient Identification} - -This module identifies variable/moving sources. The inputs are: -\begin{enumerate} -\item The combined sky cell image --- from the previous module or the -IPP Pixel Server; and -\item The current static sky image --- from the Sky Image Server. -\end{enumerate} - -The module subtracts the current static sky image from the combined sky -cell image. In order to do so, the PSFs need to be matched. This is -done by convolving the image that has the narrower PSF with the -kernel, which is the ratio of the two PSFs (this should be done with a -fit to the kernel instead of just using the data). It should be -sufficient to assume that the kernel is constant over the sky cell -(otherwise, the sky cell can be broken into smaller sections). - -The subtracted image is scoured for point sources above the noise -threshold, as well as short and long streaks caused by asteroids and -satellites, respectively. It may be neccessary to determine whether -the detection is false by virtue of its PSF (a cosmic ray missed by -the combination script should have a very narrow PSF, at least in one -dimension), or negative pixels surrounding a positive core (caused by -a bad subtraction, in turn caused by a bad kernel). - -If the subtraction is very bad (many false detections), then Phase 4 -for this sky cell should fail neatly, with a flag for the human -supervisor. Otherwise, all variable sources identified in the -subtracted image should be masked in the combined sky cell image. The -pixels from the combined sky cell image for point sources and short -trails (asteroids) should be saved (say, 3 $\times$ FWHM in radius -surrounding the source, configurable). The long trails (satellites) -should be removed in the combined sky cell image and the subtracted -image, from edge to edge. The dividing limit between short and long -trails shall be a configurable parameter, initially set to 15 degrees -per day. - -The module outputs: -\begin{enumerate} -\item Combined sky cell image, with all variable sources masked --- -used for the next module; -\item Subtracted image, with long trails masked --- sent to the IPP -Pixel Server; and -\item Catalog of variable sources --- sent to the IPP Object -Database. -\end{enumerate} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Add to Static Sky} - -This module adds the combined sky cell image into the static sky, so -that a deep image of the sky may be formed. This step should only be -performed if the new data is of sufficient quality that it will not -degrade the static sky image. The inputs are: -\begin{enumerate} -\item The combined sky cell image with variable sources masked --- -from a previous module; -\item The current version of the static sky --- from a previous module, -or the IPP Pixel Server; and -\item Relative weightings, based on the relative signal-to-noise in -each of the images --- estimate made from metadata associated with -each image. -\end{enumerate} - -The sky cell image is added to the static sky. The sky cell image -should already be photometrically accurate (when combined), and -variable sources have been masked, so it is safe to simply add the -images, employing the weightings. Sources should be identified on the -new static sky, and the limiting magnitude of the new static sky image -estimated. - -The output is: -\begin{enumerate} -\item The new static sky image --- sent to the Sky Image Server; -\item The Catalog of sources on the new static sky image --- sent to the IPP Object Database; and -\item The estimated limiting magnitude for the new static sky --- -metadata associated with the the new static sky image. -\end{enumerate} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Notes} - -\begin{itemize} -\item Catalogs should include positional information ($x,y$, with -associated errors), photometry (with associated error), and shape -parameters (FWHM, major and minor axes, position angle). -\item Limiting magnitudes can be obtained by photometering many -regions of blank sky (if possible), and (robustly) estimating the mean -and standard deviation (in counts). The limiting magnitude is the -magnitude corresponding to 3 (configurable) standard deviations above -the mean. -\end{itemize} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Calibration Image Processing} - -The IPP Calibration Image Pipelines perform the tasks needed to -generate high-quality calibration images from the input image -dataset. These operations may be performed on whatever timescales are -appropriate and necessary to maintain the quality and relevance of the -calibration images. There are four distinct types of calibration -image pipelines: the basic detrend creation pipeline, the photometric -correction image creation pipeline, the fringe pattern generation -pipeline, and the sky foreground pattern generation pipeline. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Cal 1: Basic detrend image creation} - -The basic detrend image creation pipeline collects the appropriate -input detrend images (bias, dark, dome flat, etc) and generates a -master image by combining the input images in some optimal way -\tbd{median/sigma-clipping/etc}. The master image is used to -determine input image residuals so that poor input images can be -iteratively rejected. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Cal 2: Fringe pattern and sky foreground model creation} - -The fringe model creation and sky foreground model creation pipelines -use night-sky images with sufficient flux to measure the fringe or sky -models. The input images are processed and optimally combined to yield -a set of correction fringe patterns. The fringe pattern creation and -the sky foreground pattern creation have a similar processing -structure: both require processing of the input images, both determine -a set of principal components as a function of specific input -parameters. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{Cal 3: Photometric flat correction image creation} - -The photometric flat-field correction uses images which have been -dithered with a large range of spatial scales, combined with the -uncorrected flat-field images, to generate a correction to the -flat-field image. This correction compenstates for non-uniform -illumination of the detector during the initial flat-field generation -stage. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Calibration Test Processing} - -The calibration test processing tests observations to determine if the -calibrations need updating. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{CalTest 1: Detrend frame testing} - -A newly-acquired master detrend frame, having been combined (using Cal -1 or Cal 2) are simply differenced from the old detrend frames. If -there exist significant residuals, the newly-acquired detrend frame -is adopted as the detrend frame of choice. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{CalTest 2: Photometric flat correction testing} - -Newly-acquired photometry of many objects (initially, this may be -standard star fields, but once the PS1 catalog is available, it should -be possible to use all photometry acquired over a given time period) -are compared with previously-acquired photometry. If there exist -significant residuals, a new photometric flat correction should be -produced from the newly-acquired photometry. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Reference Catalog Processing} - -The IPP reference catalog pipelines use the data in the IPP Metadata -Database and the IPP Object Database to determined improved -astrometric and photometric calibration references. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{AstroRef: Astrometric Reference Catalog creation} - -This processing stage shall use many observations over a given time -period to fit a consistent global astrometric solution, resulting in a -high quality and internally-consistent astrometric catalog that may be -published. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subparagraph{PhotoRef: Photometric Reference Catalog creation} - -This processing stage shall use many observations over a given time -period to fit a consistent global photometric solution, resulting in a -high quality and internally-consistent photometric catalog that may be -published. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Reference Catalogs} - -The IPP will employ reference catalogs in order to calibrate the -photometry and astrometry. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Astrometric Reference Catalog} - -For PS1, this shall be UCAC. - -For PS4, this shall be the PS1 catalog. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Photometric Reference Catalog} - -For PS1, absolute photometry will not be available until the master -fit which will be performed when all data is taken. For purposes of -relative photometric extinction, the guide star brightnesses should be -sufficient. - -For PS4, the PS1 catalog shall be used. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Modules} - -\tbd{What goes here? There will be modules?} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{\PS{} Library} - -See PSDC-430-007 for the design of the \PS{} Library, PSLib. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Internal Interfaces} - -\tbd{To be updated and expanded.} - -Internal interfaces consist of queries to the IMD or IPS, insertion of -data in the IMD, IPS, or IOD, or processing configuration files. The -science and calibration image processing pipelines make requests for -images from the IPS, metadata from the IMD, and push their results -back onto the IPS and IMD. The reference catalog pipelines make -requests on the IMD and the IOD and push their results back to the -IOD. The scheduler creates input processing configuration files for -the processing pipelines and queries the IMD and IPS and pushes -results back to the IIS. - -FITS Images - -FITS Tables - -XML - -SQL queries - -C:DB interactions - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{External Interfaces} - -\tbd{This whole section to be updated.} - -This subsection describes the interfaces between the IPP and other -\PS{} systems and the external clients. The interfaces are -illustrated in Figure~\ref{fig:functionalities}. Incoming data is -received by either the IPS (pixels), the IMD (metadata), or the IOD -(objects). Requests for data by external clients are also made to -these three databases. Requests for data made by the IPP are -generated by the IPP Scheduler or the science processing pipelines. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{OATS} - -The Summit Pixel Server (SPS) sends raw image data, image metadata, -and enviromental metadata to the IPP. The IPP provides an interface -mechanism by which the SPS can register new images with the IPP, which -sends them to the appropiate subsystem: The image pixel data is sent -to the IPS while the metadata is sent to the IMD. - -The \PS{} Telescope Scheduler (PTS) sends information about the -telescope schedule to the IPP: observing plan for the night, or longer -time scales. The IPP scheduler sends telescope schedule requests to -the PTS (i.e.\ calibration needs). - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Published Static Sky Server} - -The Static Image Server provides segments of the current static sky -image to the IPP on demand. IPP subsystems which require this data -will block until it is available or timeout if it is not. The IPP -provides updated static sky images to the SIS when available. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Object Database} - -The Master Science Object Database receives new object photometry from -the IPP. The IPP IOD acts as a cache for object photometry data; -\tbd{an IPP subsystem will send photometry data in batches on some -timescale. Is this a function of the IOD?} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Moving Object Processing System} - -The Moving Object Processing System interfaces with the IPP to receive -the objects detected in the difference images via queries to the IOD. -The MOPS may interface with the IMD as needed. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Other Client Science Pipelines} - -The client science pipelines may interface with the IPP via requests -for data from the IMD, IOD, or IPS. \tbd{how many clients max? / how -much data?} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Computer Hardware} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Overview} - -This document discusses the likely range of the \PS{} Image -Processing Pipeline (IPP) hardware requirements. The hardware -requirements addressed in this document consist of: - -\begin{itemize} -\item Total Disk Volume -\item Total Processing Power -\item Sustained Switch Bandwidth -\item Sustained Node Network I/O -\item Sustained Disk I/O -\end{itemize} - -Even without the complete IPP design, it is possible to identify the -major drivers on the hardware requirements. The total disk volume -requirements are dominated by the need to store raw images for a -certain period, the need to store calibration images for a longer -period, and the need to store the static sky images. Of the various -analysis pipelines, and depending on the data organization as -discussed below, Phase 2 and Phase 4 present the most significant -demands in terms of data I/O throughput on the network. Phase 2 and -Phase 4 also present the most significant CPU demands. In this -discusion, Phase 2 refers to the per-chip pre-processing in which the -instrumental signature is removed and a first pass object detection is -performed. Phase 4 refers to the multiple chip combination in which -the pre-processed images are merged and combined, in both addition and -subtraction, with the static sky image, and up to three object -detection passes are performed. - -This document does not address the hardware requirements implied by -the Phase 0, 1, or 3 stages, nor the load required by the calibration -image creation stages. In the first instance, the operations are only -performed on the metadata and are extremely minimal both in terms of -data I/O and computation requirements. In the second case, the -processing is less time critical than the per-image processing and is -performed only infrequently (once per night to once per week or -month). This document also does not address any hardware requirements -introduced by the metadata manipulation. The software implementation -for metadata storage (RDBMS, FITS tables, etc) will have a very large -impact and will be evaluated along with the needed hardware at a later -date. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Scenarios} - -We will address the various hardware requirements by referring to a -set of data processing and data organization scenarios. The actual -hardware requirements will depend on design decisions which are not -yet available. It is possible to define the data organization in ways -which will minimize the hardware requirements, but which will increase -the software development effort. We will discuss both the worst-case -data organization scenario, which does not require significant -intelligence in the software systems, and the optimal data -organization scenario, which will require the software to track the -location of data products more carefully. In addition, this document -will address the data requirements of the complete \PS{} pipeline -with 4 telescopes as well as the single-telescope \PS{}-1 scenario -based on the Design Reference Mission [REF]. - -The IPP hardware system must provide both data storage and -computational resources. The IPP requires relativley large amounts of -data storage space, primarily for the image data. Image data is -organized in two categories. First, there is the per-chip data -- -data associated with specific chips, including the raw images, the -calibration images, and temporary processed images at various stages. -Second, there is the data associated with the static sky imagery, -which is in turn organized into smaller sky-cell units. The first -assumption we make is that the hardware is organized into nodes which -provide both data storage and computational resources. The second -assumption we make is that the data storage nodes are divided into two -classes: those which deal with the per-chip data and those that -provide the static sky storage. In addition, we assume that the -computational tasks related to Phase 2 take place on the per-chip -storage nodes and the Phase 4 computation takes place on the static -sky storage nodes. - -Figure~\ref{hardware} shows our basic concept for the hardware -organization for the IPP. This diagram shows the two types of compute -nodes: chip-level processing and storage nodes (dominated by Phase 2) -and static sky processing and storage nodes (mostly Phase 4). Also -shown are two switches used in this configuration; although it is -currently possible to buy a single switch which would have a -sufficient number of GigE ports for both sections of the PS-1 system, -such a two-switch organization may be needed for the full \PS{} -system. In such a case, the interswitch communication must also meet -the required throughput needs. We discuss the hardware requirements -in the assumption that such an organization will be necessary. - -The way in which the images are distributed among the storage and -compute nodes will largely determine the I/O bandwidth requirements. -For data bandwidth requirements calculations, it is necessary to make -some assumptions about the data organization. For the purposes of -this document, we explore two extreme-case options: -\begin{itemize} -\item Random Data Distribution --- Detector \& Sky data is randomly - distributed within the compute node of a given type (ie, chip data - is randomly distributed among the detector compute nodes). -\item Optimal Data Distribution --- Detector \& Sky data is optimally - distributed to compute Detector/Sky nodes (chip processing is always - on a machine with local chip data). -\end{itemize} -A second factor which will have a significant impact on the I/O -requirements is the image storage format for the processed and -calibration images. We have two basic choices: 32 bit floating point -format or 16 bit integer format with appropriate scaling. In the -former case, additional dynamic range is retained, while in the latter -case, we reduce the data volume by a factor of 2. While some may -argue that the higher dynamic range is necessary, arguments can be -made that the 16 bit range is sufficient. (In particular, the 16 bit -data provides a dynamic range far above the expected 1/1000 fractional -accuracy of the flat-field images). A related question is the number -of calibration images needed by the processing system. Since the -complete analysis is not yet defined, this number is difficult to -ascertain. However, we can make a range of assumptions which are -reasonable. We therefore adopt two data volume scenarios to explore -these possibilites: -\begin{itemize} -\item Standard Data Volume - 32 bit data for processed and calibration - images, average of 7 calibration frames per image. -\item Minimal Data Volume - 16 bit data for processed and calibration - images, average of 4 calibration frames per image. -\end{itemize} -In the discussion that follows, we explore the hardware requirements -implied by the collection of four combinations of these two sets of -scenario options. - -\begin{table} -\begin{center} -\caption{Hardware Throughput Tests \label{existing-hardware}} -\begin{tabular}{lrrrr} -\hline -\hline -Test & where \& when & model & result \\ -\hline -node I/O & CFHT 11/2002 & Intel 1000 Gigabit & 35 - 40 MB/s sustained \\ -node I/O & CFHT 2/2004 & Intel 1000 Gigabit & 65 - 70 MB/s sustained \\ -RAID write & CFHT 2/2004 & 3ware RAID cntl + IDE & 110 MB/s sustained \\ -Switch Load & VeriTest & Cisco & 3 GB/s (for 32 ports) \\ -\hline -\end{tabular} -\end{center} -\end{table} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Existing Hardware Throughput} - -We have collected a few representative tests of various pieces of -modern hardware to give a reference for the throughput capabilities. -A number of hardware configurations have been tested at CFHT for the -Elixir project, and we include here their recent reported hardware -RAID-5 I/O speeds and GigE card speeds. We also have included data -from VeriTest studies of Cisco switch throughput, commissioned by -Cisco for a 32 port GigE switch. These tests are summarized in -Table~\ref{existing-hardware}. - -\begin{table}[b] -\begin{center} -\caption{Data Storage Requirements \label{storage}} -\begin{tabular}{lrrrr} -\hline -\hline - & Standard / PS-4 - & Standard / PS-1 - & Minimal / PS-4 - & Minimal / PS-1 \\ -\hline -Raw data & 300 TB & 75 TB & 300 TB & 75 TB \\ -static sky & 512 TB & 64 TB & 256 TB & 32 TB \\ -calibration frames & 175 TB & 18 TB & 17 TB & 5 TB \\ -metadata db & 2 TB & 2 TB & 0.2 TB & 0.2 TB \\ -object db & 60 TB & 4 TB & 60 TB & 4 TB \\ -\hline -totals & 1050 TB & 163 TB & 633 TB & 116 TB \\ -\hline -\end{tabular} -\end{center} -\end{table} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Data Storage Requirements} - -The \PS{} IPP data storage requirements may be divided into five -principal areas: raw image data, static sky image data, master -calibration images, the metadata database, and the object database. -We discuss each of these data items and their impact on the data -storage requirements for the IPP, and identify the impact of the -minimal vs standard data storage requirements as well as the -requirements specifically for PS-1. Table~\ref{storage} summarizes -the data storage requirements in the different scenarios. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Raw Data Storage} - -There are two basic image types which will be acquired: night-time -science images and calibration images. The night-time science images -consist of 1Gpix per image, or 2GB in raw format. At nominal cadence, -the 4 telescopes can obtain images at a sustained rate of 1 image per -30 seconds per telescope for the entire night of 10 hours (36000 -minutes). A total of 100 calibration images per night would be a -substantial overestimate of the typical expectation. Combining these -numbers, we can expect to receive a total of 1300 image per telescope -per night, 5200 image total, or 10.4 TB of data per night. The total -data storage requirements for the raw data are governed by the number -of nights' worth of data we are required to keep online. A reasonable -number is one month to allow a full moon's cycle. Thus, for raw image -storage, we require a total of 300 TB data storage. For PS-1, this -number is simply scaled down by a factor of 4. The choice of the -minimal data volume does not affect these numbers because the raw data -is already stored with 16 bit pixels. - -\tbd{The PS-1 design reference may now require storage of the entire -first year of data, calculated to be 200 TB.} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Static Sky Data Storage} - -The static sky is represented by images with 0.2 arcsec per pixel. -There will be one summed image and one weight image for each of the 6 -filters, each stored in floating point format. At this resolution, -there are 324 Mpix per square degree, and we will observe a potential -total area of 30,000 square degrees. Allowing for 10\% overage for -overlapping tiling, we require a total of 10.7 Gpix to cover the sky -once, or a total of $\sim 512$ TB for the static sky images. In the -minimal data volume scenario, this value is reduced by a factor of 2, -while in PS-1, the reduction is a factor of roughly 8 because we only -intend to store the static sky for the ecliptic plane survey and the -small IPP verification program. - -\tbd{This last point is no longer valid - the PS-1 static sky may -require the entire 3pi.} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Calibration Frame Storage} - -The possible required calibration frames consist of the bias, dark, -and mask images, along with one flat, one flat-correction, and -multiple sky/fringe library frames per filter. In fact, not all types -are needed at all stages. For the standard data volume, we assume an -average of 7 calibration frames per image and filter. This results in -a total of 42 master calibration image per telescope. If we intend to -keep all master calibration frames for the project lifetime, and -generate a new master on a weekly basis (a reasonable time-scale), -then we can expect to require a total of 175 TB of calibration image -by the end of the 5 year lifetime of the project. For the case of -PS-1, the time period is only 2 years, and there is only 1 telescope, -resulting in a factor of 10 reduction in the volume. For the minimal -data case, we reduce the volume by another factor of 3.5. We also note -that this is likely to be a drastic overestimate as we are unlikely to -need to regenerate all master calibration frames on a weekly -time-scale. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Metadata Database Storage} - -The metadata data storage requirements are driven by the need to store -the data for the project lifetime. There are two types of metadata -generated at the summit: data associated with images and environmental -data. The environmental data consists of measurements on a regular -cadence, roughly 1 per minute, of a variety of parameters. We suggest -an expected of 1kB per entry, for a total of 2.6 GB over the lifetime -of the project. PS-1 will represent a smaller amount of data per -minute, and also a factor of 2.5 fewer minutes. We suggest PS-1 may -have a total environmental metadata set smaller by a factor of 5. The -additional systems, such as the DIMM, SkyProbe, NIR Sky Camera, and -the LRProbe will have higher data requirements, but should be -considered as separate, self-contained systems. Their data products -are distilled to a limited number of parameters per minute which are -included in the 1kB given above. Furthermore, items such as -guide-star history, if saved, will be saved with the image data and -represents only a small fraction of the total image data volume. Some -subset of the telescope diagnosic information may be a high volume -data product as well, but only retained by the telescope control -system for the purpose of diagnostic studies. Such data will be -excluded from this analysis. - -The image metadata consists of values associated with the FPA (4), the -chips (240), and the Cells (15360). Aside from the guide star -history, the total data requirements for each of these entries will be -scaled by the number of bytes required for the metadata from each data -level. Clearly, if the Cell entry is allowed to be large, it will -dominate the total Metadata data volume. If we suggest an expected -number of 64~bytes per Cell, 256~B per chips, and 1~kB per FPA, we find a -total metadata volume per exposure of roughly 1~MB, completely -dominated by the Cell metadata. With the exposure rates above, we -find a total of metadata volume of 1.8~TB over the lifetime of the -project. For PS-1, the total volume is reduced by a factor of 2.5 -(for the shorter lifetime) and another factor of 4 (for the lone -telescope). Neither data quantity is affected by the minimal vs -standard data volume choice. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Object Database Storage} - -The hardware requirements for the IPP object database are rather -flexible: the total volume depends critically on the depth to which -the object detection analyses are performed (and thus the total number -of object detections) and the number of object parameters which are -measured. We can make very rough estimates that the total number of -detections over the 5 year lifetime of the project may be in the -vicinity of $5\times10^{11}$. We can conservatively estimate the -number of bytes needed to represent each detection as 128 B, resulting -in a total data storage for the object detections of 60 TB. However, -this number depends strongly on the timescale for which the IPP is -required to maintain all object detections, and may potentially be -significantly reduced. For the case of PS-1, the total number of -detections is likely to be reduced by a factor of 4 for the number of -telescopes, and potentially another significant factor ($\sim 4?$) by -limiting the depth of object detections. Again, the minimal data -volume scenario is irrelevant to the object database volume. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{CPU Requirements} - -Phase 2 and Phase 4 dominate the processing requirement, primarily -because they must keep up with the image delivery rate of 1 per 30 -seconds. We have performed benchmarks of a demonstration version for -both the Phase 2 and Phase 4 analyses. - -For the Phase 2, a substantial fraction of the processing time is -consumed by the need to perform FFTs on the images in order to -convolve them with the guide-star kernel, and in the smoothing used -for the object detection process. Additional processing time is -needed by the object detection, deblending, and analysis. Experiments -with the FFTW package show that FFTs may be performed on Intel -processors at rates of approximately 0.25~GHz-sec / Mpix for data sets -of order 1 Megapixel. The FFTs required for the Phase 2 analysis are -performed on the 512$^2$ pixel cells, so these numbers may roughly be -scaled linearly to determine the total time required for chip -processing. A single FFT on a full chip, with 64 cells, therefore -requires roughly 4~GHz-sec. For the full Phase 2 analysis, there are -roughly 4 single direction FFTs required excluding those associated -with object detection; thus the total processing time for these FFTs -is approximately 16~GHz-sec. The addtional analysis steps, excluding -object detection and characterization, account for a small fraction of -this compute time, which we estimate at 10\%. The object detection -stage depends somewhat on the depth to which the analysis is -performed, and the number of measurements made per object. Typical -analysis performed by the Sextractor routine, which performs a -substantial number of per-object analyses, requires 27~GHz-sec for a -full chip, including the FFTs used for smoothing. We can therefore -assume a total of 50~GHz-sec per chip for the Phase 2 processing. -This converts to a total of 12,000~GHz-sec for a complete major frame. - -For Phase 4, the main computational tasks are combining the multiple -images, with cosmic-ray rejection, and performing the object detection -tasks. Nick Kaiser has done tests of the Phase 4 image combine and -rejection stages, and finds a total processing time of roughly -96~GHz-sec for a full stack of 4 chips. If we add in an additional -34~GHz-sec for detailed object detection and image differencing, we -find a conservative estimage of 130~GHz-sec for a 4-image chip stack, -equivalent to 7800~GHz-sec for a major frame. - -For PS-1, the data processing will clearly require a smaller amount of -computational resources because of the lower image rate. However, the -total number of GHz-sec required for the complete analysis of 4 input -images and the combination with the static sky will remain -more-or-less the same. Some reduction in the load may be gained by -reducing the complexity and depth of analysis for PS-1. Depending on -the details and depth of the analysis, we may reduce the computational -load by a factor of 2. - -\begin{table} -\begin{center} -\caption{Data Scenarios (MB per Chip or Sky-cell) \label{scenarios}} -\begin{tabular}{lrrrr} -\hline -\hline - & Random / Standard & Random / Minimal & Optimal / Standard & Optimal / Minimal \\ -\hline -{\em Phase 2 input} & & & & \\ -from summit & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB \\ -input image & 32 MB & 32 MB & {\bf 32 MB} & {\bf 32 MB} \\ -calibration & $7 \times 64$ MB & $4 \times 32$ MB & {\bf 7 $\times$ 64 MB} & {\bf 4 $\times$ 32 MB} \\ -mask image & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ -\hline -network I/O: & 560 MB & 232 MB & 64 MB & 64 MB \\ -disk I/O: & (560 MB) & (232 MB) & 496 MB & 168 MB \\ - & & & & \\ -{\em Phase 2 output} & & & & \\ -output image & 64 MB & 32 MB & {\bf 64 MB} & {\bf 32 MB} \\ -output mask & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ -image to P4 & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB \\ -mask to P4 & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB \\ -\hline -network I/O: & 200 MB & 100 MB & 120 MB & 60 MB \\ -disk I/O: & (80 MB) & (40 MB) & 80 MB & 40 MB \\ - & & & & \\ -{\em Phase 4} & & & & \\ -input images & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & & \\ -input masks & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & & \\ -static sky & 64 MB & 64 MB & & \\ -static weight & 64 MB & 32 MB & & \\ -\hline -input: & 608 MB & 336 MB & & \\ -output: & 192 MB & 128 MB & & \\ -\hline -\multicolumn{5}{l}{\em Bold-faced entries are access to local-disk} \\ -\multicolumn{5}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\ -\end{tabular} -\end{center} -\end{table} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Per-Node I/O Requirements} - -Data I/O per node is defined as the number of bytes per second passed -through the node's network adapter. The data throughput for each node -depends strongly on the scenarios identified above. In this section, -we identify the data which is passed between nodes for each of the -different scenarios. Table~\ref{scenarios} lists the per-node data -I/O for the four scenarios. - -For PS-4, there are only 30 seconds of compute time allowed for each -of the Phase 2 and Phase 4 analyses. We use the data I/O volumes and -some assumptions about expected network and disk bandwidth to estimate -the I/O and processing timeline for the four scenarios. From this -analysis, we can judge the total CPU requirements in terms of GHz, not -just GHz-sec. We have assumed that GigE network adapters are capable -of delivering data at 50MB/sec sustained and that a disk RAID can -deliver sustained 100 MB/sec reads and writes. These numbers are -conservative estimates based on recent tests discussed above. Using -these assumptions, Table~\ref{throughput} lists the time allocations -for the complete set of scenarios for the case of PS-4. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Random / Standard Data Scenario} - -In the Random Data Distribution scenario, there is a single CPU -allocated to each chip in the detector farm and a single CPU for each Sky -cell process. The chip data are stored across random machines in the -detector farm, with the result that every Phase 2 processing requires -network access to the data. For each science chip which is -observed, each detector node will read from the network a total of 560 MB -(the 2 raw images for data storage and the 7 calibration frames, along -with one mask and one raw input image) and write a total of 200 MB -(one processed image and the mask along with the 1.5 processed images -and masks for the Phase 4 analysis). Given the assumption of 50 MB/s -from the network adapter, the total data volume implies an I/O period -of 15.2 seconds. Note that the disk I/O is parallel with the network -I/O and substantially underfills the disk bandwidth. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Random / Minimal Data Scenario} - -In the Random-Minimal, there is a single CPU allocated to each chip in -the detector farm and a single CPU for each Sky cell process, and the -chip data are stored across random machines in the detector farm. -However, the calibration and the processed science images are stored -at 2 bytes per pixel, the mask is set at 4 bits per pixel, and only 4 -calibration images are assumed. For each science chip which is -observed, each detector node will read from the network a total of 232 MB -(the 2 raw images for data storage and the 4 calibration frames, along -with one mask and one raw input image) and write a total of 100 MB -(one processed image and the mask along with the 1.5 processed images -for the Phase 4 analysis). Given the assumption of 50 MB/s from the -network adapter, the total data volume implies an I/O period of 6.6 -seconds. Again, note that the disk I/O is parallel with the network -I/O and substantially underfills the disk bandwidth. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Optimal / Standard Data Scenario} - -In the Optimal Data Distribution scenario, there is a single CPU -allocated to each chip in the detector farm and a single CPU for each -Sky cell process. In addition, all data for the specified chip are -stored on local disks attached to the same computer as the CPU, with -the result that all Phase 2 I/O is made to a local disk. For each -science chip which is observed, each detector node will read from the -network a total of 2 raw images (one for the original image, one for -the backup copy) and write an average of roughly 1.5 processed images -and masks to the Phase 4 machines for a total of 184 MB of network -I/O. During the processing stage, the detector node will read from -disk a total of 496 MB (7 calibration frames at 64 MB each, one 16 MB -mask, and one raw science image at 32 MB) and write a total of 80 MB -(one processed image at 64 MB and one mask at 8 MB). Given the -assumptions for the network and disk bandwidths (50 MB/s and 100 MB/s -respectively), the data volumes imply a total I/O period of 9.5 -seconds. In this instance, the network I/O is presumed to be -sequential with the disk I/O. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Optimal / Minimal Data Scenario} - -In the Optimal / Minimal Scenario, the minimal data sizes are used -with the optimal data distribution scheme. In this case, we reduce -the disk I/O volume to 168 read and 40 MB write, and the network -traffic to 124 MB. Given the assumptions for the network and disk -bandwidths, the data volumes imply a total I/O period of 4.6 seconds. -Again, the network I/O is presumed to be sequential with the disk I/O. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Phase 4 Node I/O Requirements / Standard Data Volume} - -Although it is easy to arrange the detector data in such a way that -the majority of I/O is performed locally, it is not as easy to arrange -this for the Static Sky data used by the Phase 4 analysis. We -therefore make the assumption that the Phase 4 analysis will require -all input detector data to be loaded across the network, as well as -all Static Sky data. This is somewhat of an overestimate as some of -the Static Sky data will be processed by machines with the data stored -locally, and clever Static-Sky data organization schemes can enhance -this chance. - -In the Phase 4 analysis, the images from the 4 separate telescopes are -combined into a single image, confronted with the appropriate segment -of the static sky, with output difference image and updated static sky -image. If we restrict input access to the individual chip cells, the -maximum read overhead is 50\% (need to read a 10x10 set of cells for -an 8x8 input image). If the processing is performed on Static Sky -segments equivalent in size to the chips, the input data is 608 MB (384 -MB of processed science image, 96 MB of mask images, 64 MB of static -sky image and 64 MB of static sky weight map) while the output data is -192 MB (static sky, weight map, and difference image, each 64 MB). -Thus, we require a total of 800 MB network I/O. Given the network -bandwidth, this implies an I/O period of 16 seconds for Phase 4. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Phase 4 Node I/O Requirements / Minimal Data Volume} - -In the minimal data volume scenario, the Phase 4 analysis volume is -significantly reduced. The total volume of input data is 336 MB (192 -MB of processed science image, 48 MB of input mask, 64 MB of static -sky image and 32 MB of static sky weight map) while the output data is -128 MB (64 MB static sky, 32 MB weight map, and 32 MB difference -image). Thus, we require a total of 464 MB network I/O, which implies -an I/O period of 9.3 seconds. - -\begin{table} -\begin{center} -\caption{Data Throughput for 4 Scenarios \label{throughput}} -\begin{tabular}{lrrrr} -\hline -\hline -& -\multicolumn{1}{c}{Random / Standard} & -\multicolumn{1}{c}{Random / Minimal} & -\multicolumn{1}{c}{Optimal / Standard} & -\multicolumn{1}{c}{Optimal / Minimal} \\ -\hline -Phase 2 per-node network I/O & 15.2 s & 6.6 s & 3.7 s & 2.5 s \\ -Phase 2 per-node disk I/O (read) & (5.6 s) & (2.3 s) & 5.0 s & 1.7 s \\ -Phase 2 per-node disk I/O (write) & (0.8 s) & (0.4 s) & 0.8 s & 0.4 s \\ -Phase 2 CPU total & 14 s : 860 GHz & 23 s : 520 GHz & 20 s : 600 GHz & 25 s : 480 GHz \\ -Phase 4 per-node I/O & 16 s & 9.3 s & & \\ -Phase 4 CPU total & 14 s : 490 GHz & 20 s : 390 GHz & & \\ -Phase 2 switch load & 6.1 GB/s & 2.7 GB/s & 1.5 GB/s & 1.0 GB/s \\ -Phase 4 switch load & 0.8 GB/s & 0.5 GB/s & 0.8 GB/s & 0.5 GB/s \\ -Phase 2 to Phase 4 switch load & 1.1 GB/s & 0.6 GB/s & 1.1 GB/s & 0.6 GB/s \\ -Summit to Phase 2 switch load & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s \\ -\hline -\end{tabular} -\end{center} -\end{table} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Switch I/O Requirements} - -The switch I/O requirements are defined by the total number of bytes -per second serviced by the two switches in the system. For the -analysis of the Switch I/O requirements, the choice of data -distribution again has a major impact. We again test the four -scenarios discussed above: Random Data Distribution, Random / Minimal, -Optimal Data Distribution, and Optimal / Minimal. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Random / Standard Data Scenario} - -In the Random Data Distribution scenario, each detector node needs to -read a total of 560 MB from the network and write a total of 200 MB -every 30 seconds. With 240 detector nodes, this corresponds to a -total bandwidth of 6080 MB/sec, or 49 Gb/sec. Note that this includes -the bandwidth needed to copy data from the summit and make two copies -on the detector machines, as well as the bandwidth to send the processed -image portions to the Phase 4 machines. The Phase 4 processing adds -an additional 320 MB of network I/O per Sky-Cell group, and there are -roughly 60-70 Sky-cells per exposure set. Thus the Phase 4 processing -adds an additional 750 MB/sec network bandwidth. In the architecture -defined in Figure \tbd{NN}, the Sky nodes and the detector nodes are each -attached to separate switches. An additional bandwidth requirement is -derived by the need to exchange data between these switches in for -Phase 4. The total amount of data exchanged between these switches is -480 MB per Sky-cell, for a total bandwidth of 1120 MB/sec. In -addition, the connection to the summit is a single, separate line -which needs to support the bandwidth requirement of copying all intial -raw images. In our simple model, each raw image is copied twice, -accounting for a total of 15360 MB every 30 seconds, or a bandwidth -load of 512 MB/sec. (Note that this last is double the actual -bandwidth requirement to the summit: a dedicated local circular buffer -would reduce the need for the second copy to come directly from the -summit.) - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Random / Minimal Data Scenario} - -In the Random / Minimal Scenario, the data volumes are significantly -reduced. The total Phase 2 bandwidth contribution is 332 MB over 30 -seconds for 240 nodes (2656 MB/sec) and the residual Phase 4 bandwidth -load is 224 MB per Sky cell over 30 seconds (522 MB/sec). The -inter-switch communication is now 240 MB per sky cell over 30 seconds, -or 560 MB/sec. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Optimal / Standard Data Scenario} - -In the Optimal Data Distribution, the Phase 2 network bandwidth is -reduced significantly to 184 MB per detector node, for a total of -1.5GB/sec, while the Phase 4 network bandwidth remains unchanged at -750 MB/sec. The inter-switch communication also remains the same at -1.12 GB/sec. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Optimal / Minimal Data Scenario} - -In the Optimal / Minimal Scenario, the total Phase 2 network bandwidth -drops to 124 MB per detector node, for a total of 1.0GB/sec, while the -Phase 4 network bandwidth is 552 MB/sec. The inter-switch -communication remains the same as the Random/Minimal Scenario at 560 -MB/sec. - -\begin{table}[t] -\begin{center} -\caption{\label{NP2} Phase 2 load per major frame (12000 GHz-sec)} -\begin{tabular}{lrrrr} -\hline -\hline -& Random/Standard -& Random/Minimal -& Optimal/Standard -& Optimal/Minimal \\ -\hline -network I/O (GB) & 182 & 80 & 44 & 30 \\ -PS-1 & & & & \\ - I/O (cpu-sec) & 3640 & 1600 & 880 & 600 \\ - CPU (cpu-sec) & 4000 & 4000 & 4000 & 4000 \\ - \# cpus & 64 & 47 & 41 & 38 \\ -PS-4 & & & & \\ - I/O (cpu-sec) & 1820 & 800 & 440 & 300 \\ - CPU (cpu-sec) & 2000 & 2000 & 2000 & 2000 \\ - \# cpus & 127 & 93 & 81 & 77 \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[b] -\begin{center} -\caption{\label{NP4} Phase 4 load per major frame (7800 GHz-sec)} -\begin{tabular}{lrr} -\hline -\hline -& Standard -& Minimal \\ -\hline -network I/O (GB) & 48 & 28 \\ -PS-1 & & \\ - I/O (cpu-sec) & 960 & 557 \\ - CPU (cpu-sec) & 2600 & 2600 \\ - \# cpus & 30 & 26 \\ -PS-4 & & \\ - I/O (cpu-sec) & 480 & 278 \\ - CPU (cpu-sec) & 1300 & 1300 \\ - \# cpus & 59 & 53 \\ -\hline -\end{tabular} -\end{center} -\end{table} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Conclusions} - -Table~\ref{throughput} presents one way of analysing the hardware -requirements, making a specific set of assumptions about the number of -nodes for the two phases and the expected network and disk -bandwidths. The important conclusion in this analysis is the implied -number of GHz per processor, given the assumptions laid out. -Phase 2 is specified to have 240 detector nodes, while Phase 4 is specified -to have roughly 60 static sky nodes. The range of Phase 2 CPU -requirements implies that each CPU needs to have speeds in the range -of 2.0 - 3.6 GHz, which sound very plausible for the year 2007, since -these apply to PS-4. - -Another way to represent this information is to use the total number -of MB I/O and the total number of GHz-sec required for the two stages, -confront these with an assumption for the bandwidth per network -adapter and an assumption for the CPU speed and use those numbers to -calculate the minimum number of nodes (CPUs) needed to sustain the -timing requirements. There are quite a few parameters and options to -choose from. We have assumed that for PS-1, the time between major -frames (4 images combined in Phase 4) is 120 seconds, and 30 seconds -for PS-4. We have also assumed that each CPU has one network adapter -associated with it, and use the numbers of 50 MB/sec for PS-1 era -network adapters and 100 MB/sec for the PS-4 network adapters (since -there has been some steady improvement in GigE hardware over the past -year). We have also assumed each PS-1 CPU is rated at 3 GHz and those -for PS-4 are rated at 6 GHz (somewhat conservative since 3 GHz -machines are already available). Tables~\ref{NP2} and \ref{NP4} show -the load and resulting number of nodes for both Phase 2 and Phase 4 -for both the PS-1 and PS-4 assumptions, using the I/O numbers for all -of the scenarios above. Note that in these discussions, we make the -idealized assumption that the computational and I/O portions of each -process are completely serial. As a result, the CPU is completely -used to perform the I/O during the I/O phase, avoiding any concern -about I/O load on the processor during analysis. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Notes} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Cell vs Chip vs FPA vs Major Frame} - -There are several levels of input data pixel groups: Cell, Chip, Focal -Plane Array (FPA), and Major Frame. It is necessary to make the -association between the data of one level and that of the next in a -way that is reliable and robust to missing elements. If a specific -cell is missing from a chip, that information is known by the -controller an needs to be represented in the metadata. Similarly if a -chip is missing from a mosaic camera, that information is also known -and must be carried though the metadata. A more difficult association -is that between the telescopes to define the major frame. Some -possibilities: - -\begin{enumerate} -\item exposures in a major frame are always synchronized; the -telescopes are required to take exposures in a coordinated fashion and -these linked exposures are identified as being part of a specific -major frame by the TCS or PTS. -\item exposures may be taken in a coordinated fashion, and identified -by the TCS or PTS as part of a specific major frame, but not all -exposures are required to be taken in this fashion. Independent -images are handled by the IPP differently (Phase 3 and Phase 4 are not -appropriate, some varient is required). -\item exposure links are defined more generally on the basis of the -resulting image metadata. The telescopes may have images requested -at the same coordinates and time, and are defined as a major frame on -the basis of the observed time and coordinates. The TCS or PTS might -not be the entity which defines these major-frame associations; this -may be the role of some component in the IPP. Different types of -major frames may be defined depending on the correlation period in -time or space. For example, a major frame in which the telescopes are -pointing at the same position in the sky to within a few pixels and -with exposures taken within a second can be treated with more special -assumptions (minimal differential distortion; moving objects -coincident) than a major frame in which the offsets are larger in -either dimension. -\end{enumerate} - -A decisions between these possibilities will drive some requirements -either on the IPP side or on the PTS/TCS side. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Identifying ghosts, spikes, etc} - -One of the functions currenly defined for Phase 1 is the prediction of -the location of the bright star spikes, ghost images, and regions of -complex astronomical background. Elsewhere in the IPP, these -identifications are used to excise or mark image pixels. How these -regions are defined and saved are is not very clear. I propose that -we use the mask image to mark as bit-flags all of these cosmetic pixel -flagging issues. If we need to save this information, for the short -period that the input science images are kept, then it is only a small -addition of data. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Pending Sky-cell / Detector table} - -Define a pending sky-cell / detector table to define the overlaps and to -give something which the scheduler can query to decide when to -initiate phase 4. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Appendices} - - -\bibliographystyle{plain} -\bibliography{panstarrs} -\end{document} - -%%%%%% Phase 0 has been dropped: identifying the moving objects is not needed - -\paragraph{Phase 0 : night preparation} - -Phase 0 is the night preparation phase of the IPP analysis system. -There may be potentially many pieces of information which apply to the -processing for an entire night and which take substantial time to -calculate. these are pre-calculated by the phase 0 stage and stored -in a database table for reference by other stages of the processing -system. Currently, the only quantity calculated by Phase 0 is the -collection of known moving object ephemerids. - -At various stages in the IPP analysis, it is necessary to know the -location of known moving objects (main belt asteroids, comets, -Kuiper-belt objects, any other classes of asteroids) in relation to -specific images obtained. If moving object orbits were trivial to -calculate, or if the number was limited, this would be a simple -problem of three dimensional intersections. However, complete orbits -are not trivial and there may be tens of thousands to millions of -possible objects of interest. To simplify the task, it is possible to -reduce the parameter space of the search by pre-calculating the orbit -segments of all objects for a given night and saving fiducial points -of the orbit in a database table. Later systems which require the -position of objects in a specific image can use linear interpolation -between these fiducial points to identify the likely objects, and -potentially additional non-linear orbital calculations to refine the -positions. - -The database table of object fiducial positions must include the -following information: - -\begin{itemize} -\item object ID -\item epoch -\item RA at epoch -\item DEC at epoch -\item dRA at epoch -\item dDEC at epoch -\item R magnitude? -\item date of calculation? -\item lifetime? -\end{itemize} - -The input for this calculation is the table of known moving objects -and their orbital elements, and the time range for the calculation. -If the calculation is slow, Phase 0 could be paralellized by object. -If Phase 0 is fast enough (\tbd{minutes?}), the process need not be -parallel. The {\tt lifetime} and {\tt date of calculation} allow old -Phase 0 entries to be removed when they are not needed. \tbd{This -cleaning phase could be a function of Phase 0.} Phase 0 need not be -run only for the current night. Any time a specific set of data is to -be analysed by the later stages, phase 0 should be run for the -appropriate time period. \tbd{Does there need to be a database table -with phase 0 runs and time periods defined? this could be the -reference used by later phases to decide if phase 0 has been run. they -could also trigger the phase 0 run if they notice it has not been run -(a job of the scheduler).} - -\tbd{what is the orbit calculation speed? does it scale with Npts? -what is the number of known objects now? in 5 years?} - - - -%%% phase 2 metadata -\milsection{Metadata} - -The following metadata associated with the images are required for -Phase~2 operation: -\begin{itemize} -\item The orthogonal transfer (OT) image shifts made during the -exposure --- in order to create a convolution kernel; -\item Time of observation --- for selecting the appropriate detrend -images; -\item Filter --- for selecting the appropriate detrend images; -\item Telescope identification --- for selecting the appropriate -detrend images; -\item Exposure time --- for the photometric calibration; -\item Detector gain --- for calculating photometric errors and -determining the quality of the overscan; -\item Detector read noise --- for calculating photometric errors and -determining the quality of the overscan; -\end{itemize} - -\milsection{Pixel Masks} -\label{ap:masks} - -This section describes the requirements on Bad Pixel Masks (BPMs). -These will consist in of bit masks for each pixel. For Phase 2, flags -are required for at least each of the following pixel attributes: -\begin{enumerate} -\item The pixel is a charge trap; -\item The pixel is a bad column; -\item The pixel is saturated in the A/D converter; -\item The pixel is non-positive in the flat-field; -\item The pixel is part of a row that has excess noise; and -\item The pixel is determined to be a cosmic ray, based on its -morphology. -\end{enumerate} - -Of these, only masks for the charge traps need to be grown by the -extent of the OT convolution kernel. For other pixel types, -orthogonal transfer of the flux in this pixel will not (necessarily) -affect the flux in neighbouring pixels - -\milsection{Object Catalogs} -\label{ap:catalogs} - -Object catalogs from Phase 2 shall consist of at least the -following elements for each object: -\begin{enumerate} -\item Object centre, with corresponding errors; -\item Object magnitude, with corresponding error; -\item Object isophotal magnitude, with corresponding error; -\item Object FWHM; -\item Object elliptical axis lengths; and -\item Object position angle for ellipse. -\end{enumerate} - -Though further details may be required for catalogs in Phase~4, -the above details are minimum requirements for Phase~2 catalogs. - Index: /trunk/doc/design/ippSDRS.tex =================================================================== --- /trunk/doc/design/ippSDRS.tex (revision 771) +++ /trunk/doc/design/ippSDRS.tex (revision 771) @@ -0,0 +1,3369 @@ +%%% $Id: ippSDRS.tex,v 1.1 2004-05-25 00:38:56 eugene Exp $ +\documentclass[panstarrs]{panstarrs} + +% basic document variables +\title{Pan-STARRS Image Processing Pipeline} +\subtitle{Supplementary Design Requirements Specification} +\shorttitle{IPP SDRS} +\author{Eugene Magnier, Paul Price, Josh Hoblitt} +\group{\PS{} Algorithm Group} +\project{\PS{} Image Processing Pipeline} +\organization{Institute for Astronomy} +\version{DR} +\docnumber{PSDC-430-008} + +% allow paragraphs to be listed in TOC for now +\setcounter{tocdepth}{4} + +\begin{document} +\maketitle + +% -- Revision History -- +\RevisionsStart +% version Date Description +DR.01 & 2004.01.01 & First draft \\ \hline +DR.02 & 2004.03.05 & Second draft \\ \hline +DR.03 & 2004.03.25 & Section reorganization \\ \hline +DR.04 & 2004.04.13 & Most sections fleshed out \\ \hline +DR.05 & 2004.04.29 & Reorganisation for consistency --- PAP. \\ \hline +\RevisionsEnd + +\listoffigures +\pagebreak + +\tableofcontents +\pagebreak +\pagenumbering{arabic} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Scope} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Identification} + +This document establishes additional design requirements, beyond those +specified in the Software Requirement Specification (PSDC-430-005), for +the Pan-STARRS Image Processing Pipeline (IPP) as applied to +Pan-STARRS 1 (PS-1), the initial demonstration telescope to be +constructed on Haleakala by Jan 2006. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{System Overview} + +\PS{} is a survey telescope system being developed by the University +of Hawaii Institute for Astronomy (IfA), the Maui High Performance +Computing Center (MHPCC), Science Applications International +Corporation (SAIC), and Massachusetts Institute of Technology (MIT) +Lincoln Laboratory. The baseline system will consist of four 1.8m +telescopes, each with a 1 gigapixel camera capable of sustained image +rates of 2 per minute. A single initial test telescope (PS-1) will +be constructed on Haleakala and will see first light at the beginning +of 2006. The full four-telescope system (PS-4) will follow PS-1 by +roughly 2 years. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Document Overview} + +The Pan-STARRS document naming scheme is PSDC-NNN-MMM-VV, where the VV +entry specifies the document version number. Where documents are +identified without the version number, the latest official version in +that series is implied. + +Open Issues and TBDs in this document are marked \tbd{in bold, red +type with surrounding square brackets}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\DocumentsInternalSection +PSDC-130-001 & PS-1 Design Reference Mission \\ \hline +PSDC-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline +PSDC-430-006 & Pan-STARRS IPP ADD \\ \hline +PSDC-430-007 & Pan-STARRS IPP PSLib SDR \\ \hline +\DocumentsExternalSection +Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ +\DocumentsEnd + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{System Design Decisions} + +Since \PS{} is a survey project, all data from the telescopes will be +uniformly analysed by the \PS{} Image Processing Pipeline (IPP) and +the appropriate resulting data products made available to internal and +external science analysis systems as they become available. The +processing performed by the IPP on the science images will consist of +detrending and object detection for the individual images, combination +of multiple overlapping images and further object detection, +subtraction of a reference (static-sky) image and detection of +residual objects, update of the static sky images, and detailed object +analysis of the static sky images. In addition, the IPP will produce +improved astrometric and photometric reference catalogs on an +occasional basis as needed. The output data products from the IPP +consist of the calibration images, reduced images from the individual +telescopes, combined images, difference images, the static sky image, +object photometry, and reference astrometry and photometry. + +The IPP interacts closely with other \PS{} systems responsible for +other aspects of the \PS{} operation, including the summit systems +(OATS), the science object database, the Moving Object Processing +System (MOPS), and potentially other client science pipelines. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{System Overview} + +The \PS{} Image Processing Pipeline (IPP) consists of a collection of +computer hardware and software organized to perform the tasks required +to process images from the \PS{} telescopes. The primary goal of the +IPP is to process the science images from the \PS{} telescopes and +make the results available to other systems within \PS{}. To achieve +this goal, the IPP must also perform other analysis functions to +generate the calibrations needed in the science image processing and +to occasionally use the derived data to generate improved astrometric +and photometric reference catalogs. + +In order to meet these broad goals, the IPP must have the following +capabilities: +\begin{itemize} +\item Store a large amount of image data, and other derived data +products (metadata and extracted objects); +\item Provide access mechanisms to these data products (both to the +subsystems of the IPP and in some cases to external users); +\item Continuously accept new image data and metadata from the +telescope system; +\item Execute various analysis processes using these data products; +and +\item Provide the decision-making logic needed to guide the data +processing, and to automatically launch the data processing tasks on +an appropriate timescale. +\end{itemize} +The IPP therefore includes subsystems which provide the data storage +framework, the data analysis framework, and the scheduling of the +analysis processes. The data storage subsystems also provide +interface mechanisms to the external \PS{} systems. + +The IPP architecture can be viewed in several possible ways. We first +consider the software architecture components needed by the IPP. +These subsystems provide the infrastructure for the data storage and +the data processing. Next, we consider the analysis pipelines which +make up the major processing tasks that must be performed by the IPP. +Finally, we consider the hardware organization required to efficiently +and cost-effectively achieve the necessary computing and storage +requirements. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{System Architecture} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Architectural Components} + +In Figure~\ref{fig:functionalities} we show the functionality of the +IPP. + +The Observatory and Telescope System (\textbf{OATS}) system at the +summit periodically produces metadata (e.g.\ weather measurements, +observations completed) and pixel data (the image pixels from the +cameras). The \textbf{Pollster} regularly (e.g., twice per minute) +polls OATS for the existence of new data. If new data exists, the +Pollster writes it to the \textbf{Metadata DB}, which maintains a +table of observations that have been obtained and whether these +observations are reduced, not reduced, or being reduced. The +\textbf{Scheduler} regularly (e.g., twice per minute) polls the +Metadata DB for observations that match predefined criteria that are +required to run reduction processes. For example, the Phase 1 +processing requires that Phase 0 has been run on a focal plane +metadata, and also requires that the observations are available and +have not yet been processed. If the criteria are met, the appropriate +stage is passed to the \textbf{Localiser} which, checks the +\textbf{Pixel DB} to determine if the stage should be performed on a +particular node. The Localiser passes the reduction stage to be +processed, along with the preferred (or mandatory) node that should +execute the reduction stage, to the \textbf{Controller}. It is the +Controller's responsibility to maintain the list of reduction stages +to be processed and execute these stages on the \textbf{Nodes}. The +Nodes may retrieve the pixel data from OATS, they write to the Pixel +DB the location of the products of the reduction and report their +completion to the Controller. + +External systems, such as the Moving Object Processing System +(\textbf{MOPS}) and other Client Science Pipelines (\textbf{CSPs}) +read the Metadata DB and the Object DB. They may also write to the +Object DB the classification of particular objects (e.g., identify an +object as an asteroid). Also, the MOPS and CSPs may also query the +Pixel DB for the location of pixel data and copies data from the +Nodes. + +\begin{figure} +\psfig{file=pics/IPPfunctionalities,width=15cm,angle=0} +\caption{The functionalities of the architectural design. See the text +for further explanation.} +\label{fig:functionalities} +\end{figure} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{OATS} + +The Observatory And Telescope System (OATS) is not a part of the IPP, +but interfaces are required with it in order to allow the Pollster to +get the list of observations not in the Metadata DB, and the nodes to +retrieve pixel data. Also, the Scheduler may report the need for new +calibration data. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Pollster} + +The Pollster is a program that polls OATS at regular intervals for the +existence of observations not contained in the Metadata DB. New +weather and image metadata are written to the Metadata DB. + +There is no reason why this architectural component cannot be +contained within another (such as the Scheduler), but it is shown here +as separate for simplicity. + +A polling model is adopted so that OATS' interface may be kept as +simple as possible --- OATS should not be concerned with whether the +IPP has received notifications. Under this polling model, it is +specifically the responsibility of the IPP to retrieve from OATS the +metadata that is not not already in the Metadata DB. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Metadata DB} + +The Metadata DB stores and maintains the metadata\footnote{Note that +metadata is any data which is not pixel data or object data.}, +including the list of images taken by the telescope system and whether +these images have been processed. The Metadata DB is regularly polled +by the Scheduler to determine what images are ready to be processed. + +Both the Scheduler and the Pollster update the status of the Metadata +DB --- the Pollster as new images become available at the Summit, and +the Scheduler as images are processed. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Scheduler} + +The Scheduler is responsible for determining the processing stages +that are required to be run on any data. Examples of these processing +stages are ``Copy the pixels from the summit'' and ``Run Phase 2 +processing on chip 12 of exposure 123''. + +Processing stages to be executed are passed to the Localiser, which +returns to the Scheduler the list of processing stages with node +assignments to each of the stages. This list of processing stages +with node assignments is passed to the Controller for execution. + +Processing stages which have executed are reported by the Controller, +which updates the Metadata DB appropriately. + +The Scheduler may also interact with OATS to inform it of the need +for new calibration data. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Localiser} +\label{sec:localiser} + +It is the duty of the Localiser to assign processing stages to +particular nodes. This may be in order to optimise performance by +distributing the stages across the nodes, or in the simplest possible +case, it may make no recommendation upon the node which performs a +particular stage. + +The Localiser may query the Pixel DB in order to identify the location +of calibration data that may be needed for the processing stage to run +(and in all likelihood, assign the processing stage to the same node as +that which holds the calibration data). + +The Localiser may either demand or request that a stage is performed on +a particular node, or make no recommendation, and passes the processing +stage back to the Scheduler. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Controller} + +The Controller's job is to control the execution of the processing +stages on the nodes. It is passed stages by the Localiser, and +executes them on the appropriate nodes. It must detect whether a node +executing a processing stage has died, and re-execute the stage on an +alternate node. + +The completed stages are reported back to the Scheduler. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Pixel DB} +\label{sec:pixeldb} + +The Pixel DB is responsible for storing and maintaining the location +of pixel data in the IPP, including the raw images from the telescope, +the master calibration images, the reference static-sky images, and +any temporary image data products produced by the IPP. It provides +this information upon request to the Localiser. + +Note that this design assumes that the pixel data will be stored on +the same nodes that will be doing the processing. + +The Pixel DB will be periodically ``published'' as the quality of the +data is assured. The external world will only have access to the +published version of the Pixel DB. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Nodes} + +The Nodes perform the grunt work of executing the processing stages as +directed by the Controller. When the processing stage has completed, +they report back to the Controller. + +They may retrieve pixel data from OATS (the Summit) and write it to +local disk when directed to do so by the Controller. They also may +access the Metadata DB to read configurations, weather information +etc, and to write summary statistics etc. They may also access the +Object DB to read objects of interest, and to write objects from the +processing stage. + +As they write products, the Nodes register with the Pixel DB that they +have written the requested output (so that the Pixel DB is aware that +the data has been written and is not merely scheduled to be written). +The Nodes do not need to read from the Pixel DB, since everything +(where to read input pixels from, where to write output pixels to) is +specified by the Localiser. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Object DB} + +The Object DB is a facility to store all of the information about +astronomical objects, including individual measurements of objects on +the images, the summary information about those objects, and reference +object data\footnote{Note that this is (possibly) a separate entity +from the object database being developed by SAIC.}. + +The Nodes, CSPs and MOPS may read objects from the Object DB, and the +Nodes may write objects (either new objects or updates), and the CSPs +and MOPS may write certain fields of objects (e.g., the external +identifiers and class of object). + +The Object DB will be periodically ``published'' as the quality of the +data is assured. The external world will only have access to the +published version of the Object DB. The published version of the +Object DB will likely be the DB being developed by SAIC. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{CSPs and MOPS} + +The Client Science Programs (CSPs) and the Moving Object Processing +System (MOPS) are not a part of the IPP, but are external systems. We +include them here to show the required interfaces. + +The CSPs and MOPS may query the Pixel DB, the Metadata DB and the +Object DB. In addition, they may write certain fields to the object +DB (e.g., the external identifiers and class of object) as they +process objects, and they may retrieve pixel data from the Nodes. + +Since ``CSPs'' is a vague term, we now give some examples which may +help to illustrate the functionality. + +One example of a CSP is a web front-end to retrieve (published) images +and objects from the Pixel DB and Object DB. + +Another example would be a program interested in searching for +transiting extrasolar planets. Such a program may periodically poll +the Metadata DB for new processed observations in its region of +interest (such as the Galactic Plane), retrieve the object photometry +of all high signal-to-noise stars in the processed observations, and +attempt to identify a planetary transit in progress. + +Yet another example would be a Stationary Transient Object Pipeline, +which would periodically poll the Metadata DB for new processed +observations, and query the Object DB for variable sources which were +identified twice (so that they are not moving objects). + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Related/Connected components} + +The Pollster may be contained within the Scheduler (i.e., the +Scheduler may initiate and/or schedule as a processing stage the +Pollster), but this is not assumed to be so in this document; this +decision is left to the implementation. + +The Localiser is strongly coupled to the Pixel DB, and throughout this +document, these are both referred to as components of the ``IPP Pixel +Server''. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Responsibility} + +The IPP team will develop and have responsibility for maintaining +these systems. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Processing Stages} +\label{sec:processingStages} + +We now consider the collection of IPP processing stages which are +executed by the Controller on the Nodes. We define a ``stage'' to be +the largest complete task which may be performed in serial without +interation between parallel threads. + +Depending on the particular stage, it may process individual images, +collections of images, or on derived data products. Because of the +nature of the image data, many of the analysis stages can be run in +parallel because, for example, the analysis of a chip in one image +does not depend on the results from another chip. + +The data analysis stages are divided into several categories as follows: + +\begin{enumerate} +\item Retrieval Stage --- pixel data are retrieved from OATS (the + Summit). +\item Science Image Processing Stages + \begin{enumerate} + \item Phase 1: image processing preparation --- estimates + first-order astrometric and photometric solutions required to + process each major frame. + \item Phase 2: image reduction --- produces calibrated chips from + raw chips. + \item Phase 3: exposure analysis --- processes an FPA to produce + unified and consistent backgrounds, photometry and astrometry for + the component chips. + \item Phase 4: image combination --- processes sky cells overlapped + by a major frame. + \end{enumerate} +\item Calibration Image Processing Stages + \begin{enumerate} + \item Cal 1: Basic master-detrend creation --- combination of simple + detrend images (e.g., bias, dome flat etc). + \item Cal 2: Sky-model/fringe-mode generation --- combination of + more-complicated detrend images (e.g., fringe, scattered light + etc). + \item Cal 3: Flat-field correction image creation --- analysis of + photometry from multiple dithered FPAs. + \end{enumerate} +\item Calibration Test Processing Stage + \begin{enumerate} + \item CalTest 1: Detrend frame testing --- tests whether new + calibration frames are required. + \item CalTest 2: Photometric float correction testing --- tests + whether a new photometric flat correction is required. + \end{enumerate} +\item Reference Catalog Processing Stages + \begin{enumerate} + \item Astrometry reference catalog generation --- processing of the + astrometric data to determine and apply a consistent global + solution. + \item Photometry reference catalog generation --- processing of the + photometric data to determine and apply a consistent global + solution. + \end{enumerate} +\end{enumerate} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Hardware Systems} + +The basic IPP hardware organization is shown in Figure~\ref{hardware}. +The overall hardware organization, with a Detector subcluster and a +Static Sky subcluster, is largely chosen to reduce the I/O load during +the pre-reduction analysis of the raw science images. In addition, we +have specified distinct machines to maintain the object and metadata +databases. \tbd{This last aspect is largely theoretical until we have +defined the details of these databases; it may be more appropriate +depending on the eventual solutions to distribute these database +elements across the Detector and Static Sky subclusters.} + +\begin{figure} +\begin{center} +\resizebox{8cm}{!}{\includegraphics{pics/hardware}} +\caption{ \label{hardware} IPP Hardware Organization} +\end{center} +\end{figure} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Software Hierarchy} + +In order to facilitate testing and development, and to encourage +flexibility, the IPP will be built in a layered fashion. The lowest +level functions will be written in C and collected together into a +\PS{} library. These library functions will be used to write more +complex modules. The modules will be written in C but will make use +of the SWIG tool to make their functionality available within other +frameworks. In particular, the modules can be tied together with a +simple framework (an `engine') or with detailed flow-control through +the use of a high-level language such as Perl, Python, or Tcl +employing the SWIG interfaces. For the high-level functions in the +operational system, the IPP will make use of \tbd{Python} as the +scripting language to provide the required flow-control to tie the +modules together. + +This approach satisfies the requirement that complicated low-level +analysis steps run fast, while preserving flexibility for coding the +high-level wrappers for which the speed requirements are not so +stringent. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{External Libraries} + +\PS{} will employ several external libraries to save duplicating +functionality that is already available. These external libraries +will be wrapped by the \PS{} Library, insulating the project from the +implementation details of the external libraries. Examples of the +external libraries are FFTW and SLALib. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{\PS{} Library} + +The \PS{} Library will consist of C structures describing the basic +data types needed by the IPP and C functions which perform the basic +data manipulation operations. Note that a subset of the library +functions will be provided with SWIG interfaces as well to allow for +their use in the creation of the processing stages. Examples of the +\PS{} Library are fourier transforms and transforming between pixel +and celestial coordinates. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Modules} + +The IPP analysis stages are broken down into modules which represent +specific functional operations. The modules will be written in C +using the \PS{} Library functions and will be grouped into a \PS{} +Module Library. The modules will be provided with SWIG interfaces to +all public APIs for their use in processing stages. Examples of +modules are overscan subtraction and image combination. Some modules +(e.g.\ find objects on an image) will be used by multiple stages. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Stages} + +The major IPP processing tasks are organized into stages, which +consist of multiple modules. Each stage represents a collection of +complex operations performed on a single data entity. Each stage +therefore represents the maximum amount of effort which can be +performed in serial without interaction between parallel threads. The +stages will be written in \tbd{Python}, linking the modules together. +Examples of stages are Phase 2 (detrend images) and Phase 4 (combine +images from multiple telescopes and search for transients). + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Orchestration} + +High-level components such as the Scheduler, the Controller and the +Localiser are for process control. As such, they shall be written in +\tbd{Python} in order to maintain flexibility. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{System Interfaces} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{System Architectural Design} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Architectural Components} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Pollster} + +The Pollster simply polls OATS on a regular basis for metadata +(including telescope exposures) which is not known by the IPP (i.e., +already written in the Metadata DB). On the discovery of such metadata, +it is written to the Metadata DB. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Pixel Server} + +The IPP Pixel Server (IPS) is a repository for all image pixel data +required by the IPP, and fulfills the roles of the Pixel DB +(\S\ref{sec:pixeldb}) and the Localiser (\S\ref{sec:localiser}). In +addition, it also provides components for managing the distribution of +data, and accessing the data. + +Images may reside in the IPS for different periods depending on their +use and type. Data stored by the IPS include the raw images, the +calibration images, intermediate processing stage images as needed, +final processed images, difference images, and image subsections, +\tbd{along with the associated metadata}. The IPS must retain images +as long as they are needed, up to the lifetime of the project. In +order to achieve the I/O requirements, the IPS may maintain the pixel +data distributed across the processor nodes in an organized fashion, +i.e.\ associating specific machines with specific detectors. The IPS +interacts with the IPP Metadata Database to allow other systems or +subsystems to identify the available images meeting specified +criteria. IPS specifications are described in the IPS subsystem +specification. + +In addition to storing the pixel data, the IPS is responsible for +acquiring new image data and metadata from the Summit Pixel Server and +making it available for processing by the IPP System. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{IPP Pixel Server Components} + +The IPP Pixel Server (IPS) fulfills the roles of the Pixel DB +(\S\ref{sec:pixeldb}) and the Localiser (\S\ref{sec:localiser}), and +consists of the following components: + +\begin{enumerate} +\item IPP Pixel Server Data Locality Optimizer (IPSDLO) +\item IPP Pixel Server Database (IPSD) +\item IPP Pixel Server Maintainance (IPSM) +\item IPP Pixel Server I/O Library (IPSIOL) +\end{enumerate} + +This assumes that the pixel data will be stored on the nodes. Each +image shall have a unique Universal Resource Identifier (URI) which +specifies the location of the pixel data. As an example, consider a +cluster with cross-mounted disks --- in this case, the filename +incorporating the full path would serve as the URI. + +The components of the IPS and their relation to other components (both +within the IPS and without) are showin in Figure~\ref{fig:ips}. + +\begin{figure} +\psfig{file=pics/IPS,width=15cm,angle=0} +\caption{The components of the IPS. In addition to the IPSDLO, IPSD +and IPSM, the IPSIOL is also a component of the IPS; use of the IPSIOL +is shown as dotted arrows in the interactions. Note that the nodes use +the IPSIOL to pass pixel data between each other.} +\label{fig:ips} +\end{figure} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{IPP Pixel Server Data Locality Optimizer (IPPDLO)} + +Processing stages generated by the Scheduler are passed through the +IPSDLO which does the following: +\begin{enumerate} +\item assigns tasks to specific nodes; +\item identifies the URI of the required input data; and +\item identifies the URI the output data should be written to. +\end{enumerate} + +This allows the choice of processing node to be optimized so that it +resides on the node which will process it, as well as allowing the +output to be written to the node which requires it for the next +processing stage. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{IPP Pixel Server Database (IPSD)} +\label{sec:ipsd} + +The IPSD maintains a database of URIs for the pixel data on the nodes. +It should be able to return the URI of the pixel data given one of: +\begin{enumerate} +\item an exposure identifier and a chip identifier (raw and processed + pixel data from the telescope); +\item a calibration identifier (detrend pixel data); and +\item a sky cell identifier (summed static sky, reduced and difference + pixel data). +\end{enumerate} + +The IPSD will also contain a history of data management commands and +actions. + +\tbd{Is there a reason why this is not a part of the Metadata DB?} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{IPP Pixel Server Maintenance (IPSM)} + +The IPSM initiates the execution of bulk data management processing +stages. It may have an automated component which, e.g., monitors the +disk space on each of the nodes and redistributes them if they become +unbalanced. However, the main intent is that it is used by a human +operator to reorgainise the data, e.g., after a new data optimisation +plan has been formulated, or to delete old data. + +The IPSM passes processing stages to the Controller which executes +them on the specified nodes. + +The IPSM allows four types of operation: +\begin{itemize} +\item Retrieve external data --- to manually trigger the copying of + external data (routine copying of the pixel data from OATS is + handled by the Scheduler). The IPSM generates {\em retrieve data} + stages which are passed to the Controller for execution. +\item Delete data --- to delete old data. The IPSM looks up the + location of the data in the IPSD and generates {\em delete data} + stages which are passed to the Controller for execution. +\item Replicate data --- to backup and rearrange data. The IPSM + generates {\em copy data} stages which are passed to the Controller + for execution. Note that this mode differs from the ``copy external + data'' mode in that it copies data already within the IPS. +\item Move data --- to reorganise storage. The IPSM executes a + replication followed by a deletion. +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{IPP Pixel Server I/O Library (IPSIOL)} + +The IPSIOL provides a mechanism for reading and writing pixel data to +the IPS. The existence of the IPSIOL insulates the processing stages +from the details of how the pixel data are stored (i.e., the +processing stages need not worry whether the data is stored locally or +remotely). It will generally be used on the nodes and the IPSDLO, +although other components will also make use of it. + +The IPSIOL will be able to: +\begin{itemize} +\item Open a file specified by a URI --- it may simply open the file + if it exists on the particular node, or it may retrieve the file + over the network. +\item Write a file specified by a URI --- it may simply write the file + if it exists on the particular node, or it may copy the file over + the network. It should also register with the IPSD that a file + specified by a URI has been written. +\item Delete a file specified by a URI --- it may simply delete the + file if it exists on the particular node, or it may delete the file + over the network. +\item Interface with the IPSD to return a URI given one of the + identifiers in \S\ref{sec:ipsd}. +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Pixel Data Flow Examples} + +For examples of the operation of the IPS, below we sketch out the +intended sequence of events for common operations. + +Reads during processing: +\begin{enumerate} +\item A processing stage has been passed (from the Scheduler) the URI + for an image that it needs to load into memory. +\item The processing stage uses the IPSIOL to open the image. +\item The processing stage reads the image into local memory in the + usual manner. +\item The processing stage closes the image using the IPSIOL. +\end{enumerate} + +Writes during processing: +\begin{enumerate} +\item A processing stage has been passed (from the Scheduler) the URI + for an image that needs to be saved, e.g., a subtracted image. +\item The processing stage uses the IPSIOL to open the image. +\item The processing stage writes the image in the usual manner. +\item The processing stage closes the image using the IPSIOL. +\end{enumerate} + +Note how the IPSIOL has insulated the processing stage from the details +of the reading and writing. + +Maintenance: +\begin{enumerate} +\item A human operator decides that all the pixel data for chip 12 + should be stored on node 3. +\item Operator plugs this into the IPSM. +\item The IPSM queries the IPSD using the IPSIOL. +\item The IPSD returns the URIs for all the pixel data for chip 12. +\item The IPSM generates processing tasks to be executed on the nodes + that will copy the data from the old URIs to a new URI which + specifies node 3. +\item The IPSM generates processing tasks to be executed on the nodes + that deletes the data pointed to by the old URIs. +\item The IPSM reports success to the operator. +\end{enumerate} + +Client Science Pipelines: +\begin{enumerate} +\item A CSP wants some pixel data. +\item The CSP queries the IPSD using the IPSIOL (e.g., asking for a + particular exposure or sky cell). +\item The IPSD returns the URI for the pixel data. +\item The CSP opens the image using the IPSIOL and the URI. +\item The CSP reads the pixel data into memory in the usual manner. +\item The CSP closes the image using the IPSIOL. +\end{enumerate} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Metadata Database} + +The IPP Metadata Database acts as a repository for all non-pixel data +needed by the IPP subsystems. This includes the image metadata, the +environmental data, system configuration data and system reference +data. The Metadata Database is required to save the non-ephemeral +data for the lifetime of the project for future reference and +additional analysis. The Metadata Database may potentially be used in +close coupling with the analysis pipelines to store temporary data +either within or between stages of the analysis. In this scenario, +the analysis pipeline will interact directly with the database. +However, database latency may make this scenario impractical, in which +case the database may be used for long-term storage only. In this +scenario, the data produced by analysis pipelines which is destined +for the Metadata Database may be collected and inserted by a separate, +dedicated process or analysis pipeline collection of processes. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Metadata Tables} + +Table \tbd{NN} lists the Metadata tables identified for the Metadata +Database. + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Metadata Tables} \\ +Weather & Details on the weather, including internal temperatures. \\ +SkyProbe & Analysis of SkyProbe data. \\ +LRProbe & Analysis of LRProbe data. \\ +DIMM & Analysis of DIMM data. \\ +NIR & Analysis of NIR data. \\ +Dome Status & The status of the dome. \\ +Telescope Status & The status of the telescope. \\ +Raw FPAs & Details on raw FPA exposures. \\ +Raw Chips & Details on raw chips. \\ +Raw Cells & Details on raw cells. \\ +Observation Group & Details on a group of observations to be processed. \\ +Chip Guide Stars & Details on guide stars \\ +Science Chip stats & Details on processed chips. \\ +Science Cell stats & Details on processed cells. \\ +Science FPA stats & Details on processed FPAs. \\ +Sky-Detector overlaps & List of overlaps between sky cells and detectors. \\ +Processed Sky-Cell stats & Details on sky cells. \\ +Calibration 1 input stats & Details on input images for Cal 1. \\ +Calibration 1 output stats & Details on output detrend images from Cal 1. \\ +Calibration 2 input stats & Details on input images for Cal 2. \\ +Calibration 2 output stats & Details on output detrend images from Cal 2. \\ +Calibration 3 input stats & Details on input images for Cal 3. \\ +Calibration 3 output stats & Details on output detrend images from Cal 3. \\ +\hline +\end{tabular} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Metadata Table Contents} + +Tables \tbd{NN} -- \tbd{NN} list the basic contents of each of the Metadata tables +listed above. + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Weather} \\ +Time & The time the weather information was measured. \\ +Temperature & The temperature at \tbd{some place. Will likely want temperatures for a range of locations: +external, dome, secondary, primary for starters.} \\ +Humidity & The relative humidity. \\ +Pressure & The (external) atmospheric pressure. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf SkyProbe} \\ +Time & The time the SkyProbe image was taken. \\ +Filter & Filter used for SkyProbe image. \\ +Transparency & The derived transparency. \\ +Error in transparency & The error in the derived transparency. \\ +Number of stars & The number of stars used to measure the transparency. \\ +Astrometry & The astrometry used on the SkyProbe image. \\ +Exposure time & The exposure time of the SkyProbe image. \\ +Sky brightness & The measured sky (surface) brightness, in physical units. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf LRProbe} \\ +Time & The time the LRProbe observation was taken. \\ +A band absorption & The absorption EW of the atmospheric A band. \\ +B band absorption & The absorption EW of the atmospheric B band. \\ +Absorption component 3 & The absorption EW by some other atmospheric component. \\ +Emission 1 & The emission EW of some sky line. \\ +emission 2 & The emission EW of another sky line. \\ +emission 3 & The emission EW of some other sky line. \\ +Number of stars & Number of stars used to measure the absorptions. \\ +Astrometry & The astrometry used on the LRProbe image. \\ +Exposure time & The exposure time of the LRProbe image. \\ +Sky brightness & The measured sky (surface) brightness, in physical units. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf DIMM} \\ +Time & The time the DIMM observation was taken. \\ +$\sigma_x$ & \tbd{The dispersion in $x$}. \\ +$\sigma_y$ & \tbd{The dispersion in $y$}. \\ +FWHM & The seeing full width at half maximum. \\ +Star coordinates & The coordinates of the measured star. \\ +Exposure time & The exposure time of the DIMM observation. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf NIR} \\ +Time & The time the NIR observation was taken. \\ +Sky brightness & The sky (surface) brightness in the NIR observation. \\ +Sky variance & The variance in the sky (surface) brightness. \\ +Astrometry & The astrometry used on the NIR image. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Dome Status} \\ +Time & The time for which the dome status is valid. \\ +Azimuth & The azimuth of the dome. \\ +Open status & Whether the dome is open or not. \\ +Lights status & Whether lights are on in the dome or not. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Telescope Status} \\ +Time & The time for which the telescope status is valid. \\ +Guide status & The status of the guiding. \\ +Altitude & The telescope altitude. \\ +Azimuth & The telescope azimuth. \\ +RA & The telescope Right Ascension (ICRS $\approx$ J2000). \\ +Dec & The telescope Declination (ICRS $\approx$ J2000).\\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Raw FPAs} \\ +Coords & Coordinates of the boresight (i.e. telescope pointing). \\ +Filter & Filter used for the exposure. \\ +Exposure status & Status of the exposure. \\ +Exposure time & Exposure time for the image. \\ +Airmass & Airmass at which the image was taken. \\ +ObsGroup ID & \tbd{The ObsGroup identification number.} \\ +Observer & The name of the observer, or the version of the telescope scheduler software. \\ +Program & The observing program being executed. \\ +Number of chips & The number of chips that comprise the FPA. \\ +NX, NY & \tbd{Assuming the chips are laid out rectilinearly,} the number of chips in each dimension. \\ +Astrometry & The astrometry used for the FPA. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Raw Chips} \\ +i, j & \tbd{Assuming a rectilinear FPA,} the chip number in each dimension. \\ +ID & Chip identification number. \\ +temps & The chip temperature. \\ +Astrometry & The astrometry used for the chip. \\ +Number of cells & The number of component cells. \\ +NX, NY & \tbd{Assuming the cells are rectilinear,} the number of cells in each dimension. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Raw Cells} \\ +Astrometry & The astrometry used for the cell. \\ +Validity & Is the cell working? \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Observation Group} \\ +ID & Identification number for the observation group. \\ +Number of images & Number of images in the observation group. \\ +Type & Type of observation. \\ +Status & Status of the observation group. \\ +\tbd{etc} & \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Chip guide stars} \\ +Chip ID & The identification number for the chip. \\ +Guide Star ID & The identification number for the guide star. \\ +X, Y & The centroided pixel coordinates of the guide star. \\ +RA, DEC & The sky coordinates of the guide star. \\ +$\sigma_{x}$, $\sigma_{y}$ & The dispersion in the centroids over the particular exposure.\\ +$\Delta X_{\rm max}$, $\Delta Y_{\rm max}$ & The maximum deviation in the centroid over the +particular exposure. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Science Chip stats} \\ +Chip ID & The chip identification number. \\ +State & \tbd{The state of the processing.} \\ +Major frame & \tbd{The major frame the chip belongs to.} \\ +ObsGroup & The observation group the science exposure belongs to. \\ +P1 astrom & The Phase 1 astrometry. \\ +P2 astrom & The Phase 2 astrometry. \\ +P3 astrom & The Phase 3 astrometry. \\ +Number of guide stars & Number of guide stars used for the exposure. \\ +Bias correction method & Method used to correct the bias. \\ +Bias stats & Summary statistics for bias (mean, number of parameters, deviation of residuals, +bias section used). \\ +Flat-field image & The flat-field image that was applied. \\ +Kernel convolution parameters & A description of the OT kernel. \\ +Flat-field stats & Summary statistics for flat-field (sigma of sky). \\ +Mask image & The mask image that was applied. \\ +Masking algorithm & \tbd{The algorithm used to mask the bad pixels.} \\ +Fringe images & The fringe model images that were used. \\ +Fringe stats & Summary statistics for fringes (fringe amplitude, sky sigma) \\ +Object detection stats & Summary statistics for object detection (number of objects, depth, other +input parameters). \\ +Updated astrometry & \tbd{Updated astrometry parameters.} \\ +Astrometry stats & Summary statistics for astrometry (number of stars, $sigma_x$, $sigma_y$) \\ +Reference catalog & The reference catalog that was used for the astrometry. \\ +Updated photometry parameters & The parameters used to update the photometry: magnitude zero point +and other corrections. \\ +Photometry stats & Summary statistics for the photometry (number of stars, $sigma_m$) \\ +Reference catalog & The reference catalog that was used for the photometry. \\ +PSF stats & Summary statistics of the PSF. \\ +Chip state & \tbd{The state of the chip?} \\ +Software versions & Versions of each of the modules used in the processing. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Science Cell stats} \\ +Bias stats & Summary statistics for the bias (mean, parameters, dispersion of residuals, biassec) \\ +P1 astrom & The Phase 1 astrometry. \\ +P2 astrom & The Phase 2 astrometry. \\ +P3 astrom & The Phase 3 astrometry. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Science FPA stats} \\ +FPA ID & The FPA identification number. \\ +State & \tbd{The state of the FPA.} \\ +P1 astrom & The Phase 1 astrometry. \\ +P1 astrom stats & Summary statistics for the Phase 1 astrometry (number of stars, $\sigma_x$, $sigma_y$). \\ +P1 reference catalog & The reference catalog that was used for the astrometry. \\ +P1 software versions & The versions of each of the modules used in the Phase 1 processing. \\ +P1 bright stars & Pointers to the bright stars in the field. \\ +P1 ghosts & Pointers to the ghosts in the field. \\ +P1 large objects & Pointers to the large astronomical objects in the field. \\ +P1 PSF model & Description of the PSF model used in Phase 1. \\ +P3 astrom & The Phase 3 astrometry. \\ +P3 astrom stats & Summary statistics for the Phase 3 astrometry (number of stars, $sigma_x$, $sigma_y$). \\ +P3 reference catalog & The reference catalog that was used for the astrometry. \\ +P3 photom & The Phase 3 photometry. \\ +P3 photom stats & Summary statistics for the Phase 3 photometry (number of stars, $sigma_m$). \\ +P3 reference catalog & The reference catalog that was used for the photometry. \\ +P3 PSF model & Description of the PSF model used in Phase 3. \\ +P3 software versions & The versions of each of the modules used in the Phase 3 processing. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Sky-Detector overlaps} \\ +Chip ID & The identification number of the chip. \\ +Sky Cell ID & The identification number of the sky cell. \\ +State & \tbd{The state of the processing?} \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Processed Sky-Cell stats} \\ +Input Chips & Identification numbers of the chips used to produce the sky cell. \\ +PSF adjustments & \tbd{Adjustments to the PSF.} \\ +CR rejection stats & Statistics from the CR rejection (number of CRs, distribution, limiting flux). \\ +Image combination parameters & Parameters used for the image combination. \\ +Difference image parameters & Parameters used for the image differencing. \\ +Average reference image depth / weight & \tbd{The weight of the reference image?} \\ +Difference image object detection stats & Summary statistics of the object detection (number of objects, +depth, other input parameters). \\ +Summed image object detection stats & Summary statistics of the object detection (number of objects, +depth, other input parameters). \\ +Software versions & Software versions of modules used in the sky cell processing. \\ +Processing stats & Summary statistics of the processing (CPU time, etc). \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Calibration 1 input stats} \\ +Input ID & The input chip identification number. \\ +Output ID & The identification number of the output detrend image. \\ +State & \tbd{State of the processing?} \\ +Accepted? & Is the detrend image of acceptable quality? \\ +Image stats & Assorted image statistics (mean flux, exposure time, airmass) \\ +Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Calibration 1 output stats} \\ +Output ID & The identification number of the output detrend image. \\ +Data type & The type of the detrend image (bias | dark | flat) \\ +Number accepted & Number of accepted input images that contributed. \\ +Number rejected & Number of rejected input images (no contribution). \\ +Summary stats & Summary statistics of the combination (deviation, normalisations). \\ +Applicability period & The time period the detrend image is applicable for. \\ +Software versions & The software versions of the modules used in processing. \\ +Processing stats & Summary statistics of the processing (CPU time, etc). \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Calibration 2 input stats} \\ +Input ID & The input chip identification number. \\ +Output ID & The identification number of the output detrend image. \\ +State & \tbd{State of the processing?} \\ +Accepted? & Is the detrend image of acceptable quality? \\ +Image stats & Assorted image statistics (mean flux, exposure time, airmass). \\ +Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ +Applied reduction & \tbd{Reduction method used?} \\ +Applied params & Parameters of reduction. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Calibration 2 output stats } \\ +Output ID & The identification number of the output detrend image. \\ +Data type & The type of the detrend image (bias | dark | flat) \\ +Number accepted & Number of accepted input images that contributed. \\ +Number rejected & Number of rejected input images (no contribution). \\ +Summary stats & Summary statistics of the combination (deviation, normalisations). \\ +Applicability period & The time period the detrend image is applicable for. \\ +Software versions & The software versions of the modules used in processing. \\ +Processing stats & Summary statistics of the processing (CPU time, etc). \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Calibration 3 input stats} \\ +Input ID & The input chip identification number. \\ +Output ID & The identification number of the output detrend image. \\ +State & \tbd{State of the processing?} \\ +Accepted? & Is the detrend image of acceptable quality? \\ +Image stats & Assorted image statistics (mean flux, exposure time, airmass). \\ +Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ +Applied reduction & \tbd{Reduction method used?} \\ +Applied params & Parameters of reduction. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Calibration 3 output metadata } \\ +Input images & Identification numbers of the input chips. \\ +Input image stats & Summary statistics of the input chips. \\ +Input object summary stats & Summary statistics of the objects on the input chips (number, density, etc) \\ +Object rejection criteria & Parameters of the rejection step. \\ +Phot stats & Summary statistics of the relative photometry (Mcal, dMcal, Klam, etc, bin size) \\ +Residual stats & Summary statistics of the residuals. \\ +Output image params & Parameters of the output image (size, etc) \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Astrometric Reference Generation output metadata } \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{1}{l}{\bf Photometric Reference Generation output metadata } \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Reference Data} \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Configuration Data} \\ +\hline +\end{tabular} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Metadata Queries} + +\tbd{How is the Metadata DB queried?} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Object Database} + +The IPP Object Database (IOD) acts as a repository for data on all +astronomical objects. This database is required to provide organized +access to objects on the sky, including the access to the photometry +associated with specific input images, moving objects associated with +specific chips. Detailed requirements for the IOD are described in +\tbd{the IOD subsystem specification document xxx-xxx-xxxx}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Object DB Tables} + +\begin{tabular}{ll} +\hline +\multicolumn{2}{l}{\bf Object DB Tables} \\ +Images & The images that have objects in the DB. \\ +Objects & The objects --- average properties of multiple detections of the same object. \\ +Detections & Detections of sources in an image. \\ +Non-Detections & Non-detections of objects in an image. \\ +Filters & Filters understood by the system. \\ +Photcodes & \tbd{Transformations between different photometric systems?} \\ +Bright Objects & \tbd{Links to postage stamp images of bright objects.} \\ +Region Tables & \tbd{???} \\ +Average Magnitudes & \tbd{How is this different from an `object'?} \\ +USNO Objects & Objects from the USNO database. \\ +Reference Objects & The reference catalogs for astrometry and photometry. \\ +\hline +\end{tabular} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Object DB Table Contents} + +\tbd{Dunno yet} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Object DB Queries} + +\tbd{Dunno yet} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Controller} + +The IPP Controller is responsible for managing the processing stages. +The Controller manages the parallel processing of these stages in the +IPP computer hardware environment and reports the completion to the +Scheduler. The Controller must be able to manage more than a single +processing thread to make maximum use of available processor +resources. + +The Controller must honour demands that a processing stage run on a +particular Node. Requests that a processing stage run on a particular +node should be honoured if possible. Where no restriction is placed +on the choice of Node choice by the Scheduler, the processing stage +may be run on any available Node. + +The Controller maintains a table of processing nodes available to it +and the status of these Nodes. When the Controller starts, it +attempts to launch a Node Agent on each of the available processing +nodes. Modes which are not responsive are placed into an inactive +state and retried occasionally. + +The Controller also maintains three tables of processing jobs: pending +stages, active stages, and completed stages. The pending stages are +those which have not yet been performed. The active stages are those +currently being performed on one of the remote nodes. The completed +stages are those which have finished, either successfully or with an +error state. The Controller daemon monitors the collection of remote +clients and sends them new pending stages when they become free. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Node Agents} + +A Node Agent runs on each of the individual nodes to perform the +processing stages as directed by the Controller. The Node Agents +communicate with the Controller via a socket connection. + +A processing stage is executed in the UNIX user space, and is run as a fork by the +Node Agent. The Node Agent must monitor the standard error and +standard output of the processing stage and save them in separate buffers. If the +process dies, the Node Agent must detect the crash. The Node Agent +must respond to various commands from the Controller. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Report status} + +The Node Agent returns the state of the Node (idle, busy, done), the +state of the current processing stage\footnote{Note that a processing +stage is considered ``current'' until it is cleared with {\em clear +processing stage} --- even if it has crashed or completed.} (`none', +`busy', `crash', `done'), and the exit status of the current +processing stage (`none', 0--256). + +The three states of the Node indicate that the client has no current +processing stage (`idle'), that it has a processing stage which is +still running (`busy'), or that it has a processing stage which has +completed. + +The processing stage states indicate the there is no current +processing stage (`none'), that the current processing stage is +running (`busy'), that the current processing stage has crashed +(`crash'), or that the current processing stage has exited gracefully +(`done'). The exit state is the exit state reported by the processing +stage (0--256 with 0 indicating a successful completion) or is an +indication that there is no current processing stage (`none'). + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Report stdout} + +Send and flush the current stdout buffer. The Node Agent will return +the complete contents of the stdout buffer via a buffered write and +flush the buffer when it is finished. The Node Agent will not accept +more data on the stdout buffer from the current processing stage until +the send is complete and the buffer is flushed. The daemon must +accept all of the buffer output. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Report stderr} + +Identical to `report stdout', but for stderr. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Kill processing stage} + +The Node Agent should send a kill signal to the current processing +stage. When the processing stage has exited, the Node Agent should +set the processing stage status to `crash' and the Node status to +`done'. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Clear processing stage} + +The Node Agent should set the current processing stage state to `none' +and the Node state to `idle'. If a processing stage is currently +running, it should be killed before the processing stage is cleared. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Start processing stage} + +The Node Agent forks a specified command. The command should be a +standard UNIX command without command line redirection or +backgrounding. For this reason, the Node Agent must provide a layer +of security, for example, by employing SSL authentication. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Matrix} + +\tbd{The Node Agent does not wear a suit, nor does it know kung fu.} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Scheduler} + +The IPP Scheduler is responsible for initiating the various processing +stages (which are executed by the IPP Controller), based on the state +of the survey as reflected by the IPP Metadata Database (IMD). + +The Scheduler shall maintain a list of processing stages, as well as +the required input and dependencies for each of the processing stagesFor example, the +dependencies for copying pixel data from OATS may be: +\begin{itemize} +\item OATS has new pixel data available; +\item The new pixel data has not been copied. +\end{itemize} +Similarly, the dependencies for executing Phase 2 processing on a chip +may be: +\begin{itemize} +\item The chip pixel data has been copied. +\item Phase 1 has run successfully on the metadata for the FPA to which + the chip belongs. +\item A reduced image (i.e., output from Phase 2) does not already + exist. +\end{itemize} + +When the dependencies are satisfied, the Scheduler shall prepare for +execution the particular processing stage on the appropriate data. +The Scheduler must query the Metdata DB for the most appropriate +calibration data, if required. The processing stage should be +filtered through the IPSDLO in order to assign the processing stage to +a particular Node (if desired) and to determine the URIs for the +required inputs. The processing stage is then passed to the +Controller. + +The Scheduler must also be able to send requests for new calibration +data to OATS, including required flat-fields, flat-field correction +observations, or other specialized observations needed to improve the +calibrations. The Scheduler must balance the need for improved +calibrations with the need to process the science images in a timely +manner given the capabilities of the science pipelines. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{System UI} + +A user interface allows a human operator to monitor the Controller and +Scheduler through some user interface (UI). The System UI may +interact with the Controller and Scheduler via a socket connection +using a defined set of commands. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Execute processing stage} + +A new processing stages is sent to the Scheduler. The Scheduler may +filter the processing stages through the IPSDLO, or it may be +prevented from doing so by the user. The Scheduler then passes the +processing stages to the Controller for execution. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Kill processing stage} + +The user may kill an existing processing stage. The Controller is +commanded to kill the particular processing stage. + +\tbd{Should we allow a System UI to kill processing stages sent by +other System UIs?} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Get status} + +The System UI may request the current status of the Controller, +including the list of pending, active, and completed processing stages +and the status of the individual processing stages. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Available Nodes} + +The System UI may view and configure the list of Nodes available to +the Controller (e.g., to remove a Node temporarily for maintenance). + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Processing Stages} + +In this section, we review the processing stages which are executed on +the Nodes. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Overview} + +The processing stages are the software that process data. These +processing stages are divided into five categories which are +summarised in \S\ref{sec:processingStages}. Each of the processing +stages are described below. + +The processing stages are initiated by the Scheduler, parallized and +managed by the Controller, and executed through the Node Agents on the +nodes. Processing stages are purely serial, and so they may be run on +a single node at once without the need for interprocess communication. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Retrieval} + +The retrieval stages simply retrieve pixel data from an external +source (ordinarily OATS at the Summit, but it could conceivably be +some other external source) and store it on the nodes. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Science Image Processing} + +The IPP science image processing stages perform analyses on the +night-sky science images to extract the science data from these +images. These consist of: Phase 1, the image processing preparation +stage; Phase 2, the image reduction stage; Phase 3, the exposure +analysis stage; and Phase 4, the image combination stage. These +pipelines must process the images in a timely manner so that the +incoming data stream will not overload the IPS. The decision to +execute a specific pipeline for a specific dataset is made by the +Scheduler, which sends the infomation to the Controller. The +Controller executes the pipeline for the data on an appropriate +machine and monitors the success or failure of the processing stage. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Phase 1: image processing preparation} + +The Phase 1 system operates on data from each FPA to calculate basic +astrometric information needed by other stages of the analysis. The +analysis includes: + +\begin{itemize} +\item preliminary astrometry based on the guide-star centroids +\item sky-cell / detector-cell overlaps +\end{itemize} + +The input to this analysis is the list of guide-star pixel centroids +and their celestial coordinates as saved in the metadata database, as +well as the FPA and chip organization and geometry, and the basic +optical distortion for the camera. For the sky-cell / detector-cell +overlaps, the sky tiling scheme is required. + +The output consists of calculated astrometric parameters (linear +transformation + static distortion) for each of the FPA chips. On the +basis of this astrometry, the overlap between the detectors and the +sky-cells is calculated. The output of this calculation is a list of +sky-cell / chip links in a database table. This list of links can be +used by the later stages to initiate the analyses. + +The phase 1 analysis is performed on an FPA basis to ensure that +enough reference stars are available for the astrometry calculation. +Phase 1 cannot be usefully calculated on the basis of a major frame +since the telescope positions are independent; no additional +information is available by combining stars from different FPAs. This +analysis does not restrict the definition of a major frame in any way. + +\tbd{Phase 1 command: P1 (exposure)} + +\tbd{Megacam: P1 654321o} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Phase 2 : image reduction : new version} + +\tbd{how long are processed images kept?} + +\tbd{what subsystem deletes processed images?} + +\tbd{does 'remove' mean 'mask' or 'replace'} + +\tbd{what is the absolute astrometry accuracy at phase 2? 0.1 arcsec +== 0.33 pix?} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Concept} + +Phase~2 processing within the \PS{} image processing pipeline is +the de-trend stage, where the images from the detector are processed +to remove instrumental signatures. + +\begin{figure} +\begin{center} +\resizebox{8cm}{!}{\includegraphics{pics/phase2}} +\caption{ \label{phase2} Phase 2 dataflow} +\end{center} +\end{figure} + +Prior to Phase~2, the Phase~1 process operates on an entire telescope +Focal Plane Array to set the boresight astrometric solution using +the guide stars and initial masking of ghost reflections. + +Phase~2 consists of the following modules: +\begin{enumerate} +\item Form OT kernel; +\item Convolve de-trend images with the OT kernel; +\item Mask bad pixels +\item Mask diffraction spikes and optical ghosts; +\item Bias/dark/overscan subtraction; +\item Trim overscan; +\item Non-linearity correction; +\item Flat-field; +\item Subtract sky; +\item Identify CRs by morphology; +\item Determine PSF model; +\item Find and photometer objects in the image; +\item Improved astrometry; and +\item Bright object postage stamps. +\end{enumerate} +These modules are each explained below. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Form OT Kernel} + +The first module for Phase~2 is to form the OT kernel from the image +metadata of pixel shifts made during the exposure. This involves +decoding the metadata and converting it to a data type that can be +used to convolve by. The output is the OT convolution kernel. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Convolve de-trend images} + +This module convolves the de-trend images with the OT convolution kernel +so that they can be used to de-trend the object image. The inputs +are: +\begin{enumerate} +\item The OT convolution kernel --- from the previous module; +\item The appropriate dark frame --- from the IPP Pixel Server; +\item The appropriate flat-field --- from the IPP Pixel Server; +\item The appropriate fringe frame(s) --- from the IPP Pixel Server; and +\item The appropriate static bad pixel mask --- from the IPP Pixel Server. +\end{enumerate} + +The module convolves each of the dark frame, flat-field, and the fringe +frame(s) by the OT convolution kernel. Specific flags in the static +bad pixel mask are grown by the outline of the OT convolution kernel +(see Section \ref{ap:masks}). The output results are: +\begin{enumerate} +\item The convolved flat-field; +\item The convolved fringe frame(s); and +\item The updated pixel mask. +\end{enumerate} +Each of these will be used for a later module. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Overscan Subtraction} + +This module corrects the object exposures for the electronic pedestal +introduced by the readout electronics. The inputs are: +\begin{enumerate} +\item The object image --- from the IPP Pixel Server; +\item The pixel mask --- from the previous module; +\item The overscan and physical detector regions --- from the +Metadata; and +\item Detector characteristics (gain, read noise) --- from the +Metadata. +\end{enumerate} + +The overscan is averaged (either in bulk, or individually by rows) or +fit with a polynomial, and the result is subtracted from the image. +Overscan rows having a standard deviation which exceeds a threshold of +twice (configurable) the detector read noise should be masked. Pixels +saturated in the A/D converter should also be masked, and these +regions grown by an additional pixel to counter CCD ``blooming''. The +output is: +\begin{enumerate} +\item The overscan-subtracted object image; and +\item The updated pixel mask. +\end{enumerate} +These will be used for a subsequent module. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Trim} + +This module trims the object image and each of the calibration frames to +remove the outer edge which was affected by the OT during the +exposure. The inputs, each from previous modules, are: +\begin{enumerate} +\item The overscan-subtracted object image; +\item The corresponding pixel mask; +\item The convolved flat-field; +\item The convolved fringe frame(s); and +\item The dimension of the OT convolution kernel in each direction. +\end{enumerate} + +Each of the input frames (object image, flat-field, fringe frame(s) +and pixel mask) are trimmed by the extent of the OT convolution kernel +in each direction ($+x$, $-x$, $+y$, $-y$). The outputs are trimmed +images for each of the input images, which will be used in later +modules. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Non-Linearity Correction} + +This module corrects images for non-linearity in the detector. The +inputs are: +\begin{enumerate} +\item The trimmed object image --- from a previous module; and +\item The detector non-linearity correction coefficient(s) --- from +the Metadata. +\end{enumerate} + +The module corrects the flux in each pixel for non-linearity by applying +a polynomial correction, with the specified coefficients. The output +is the corrected object image, which is used for a later module. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Flat field} + +This module corrects the object image for variations in sensitivity over +the image. The inputs are: +\begin{enumerate} +\item The object image corrected for non-linearity; +\item The corresponding pixel mask; and +\item The convolved, trimmed flat-field. +\end{enumerate} +Each of these comes from a previous module. + +The module divides the object image by the flat-field, masking pixels +that are non-positive in the flat-field. The outputs are: +\begin{enumerate} +\item The flattened object image; and +\item The updated pixel mask. +\end{enumerate} +Both of these will be used in later modules. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Subtract sky} + +This module subtracts the sky background from the object image. The +inputs are: +\begin{enumerate} +\item The object image --- from the previous module; +\item The list of objects on the image --- from the object database; and +\item The convolved, trimmed fringe frame(s) --- from a previous module. +\end{enumerate} + +The module masks (though {\em not} in the ``official'' pixel mask) all +objects on the image using the astrometric solution from the +boresight, and fits for the sky background, consisting of a polynomial +to model the continuum, and the fringe frame(s) to model the fringes +from sky emission lines. If the concentration of objects in the image +is too high to reliably fit the sky background, the background +solution from an exposure close in time and airmass to the current +object image is used. The output is the sky-subtracted object image, +which is used for the next module. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Identify CRs by morphology} + +This module identifies cosmic rays (or other hot pixels missed in the +static bad pixel mask) on the basis of their morphology. The inputs +are: +\begin{enumerate} +\item The object image; and +\item The corresponding pixel mask. +\end{enumerate} +Both of these come from a previous module. + +The module identifies CRs, the pixels of which are masked in the pixel +mask. The pixels flagged as CRs are then grown by an additional pixel +in each direction. Masked pixels are interpolated over. The outputs +are the updated pixel mask, which is sent to the IPP pixel server for +use in Phase~3, and is also used for the next module; and the object image, +which is sent to the IPP Pixel Server. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Find objects} + +This module finds objects on the object image. The inputs are: +\begin{enumerate} +\item The sky-subtracted object image; and +\item The corresponding pixel mask. +\end{enumerate} +Both of these come from a previous module. + +The module identifies objects on the image, which will be later used to +register images from different focal planes. The output is the +catalog of objects (see Appendix~\ref{ap:catalogs}) identified on +the image, which is sent to the metadata database, associated with the +object image. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Bright object postage stamps} + +This module saves postage stamps of bright objects, so that extra care +with regard to astrometry and photometry can be taken with them at a +later stage. The inputs, each from a previous module, are: +\begin{enumerate} +\item The sky-subtracted object image; +\item The corresponding pixel mask; and +\item The catalog of objects. +\end{enumerate} + +The module makes postage stamps of all objects brighter than a given +instrumental magnitude, along with corresponding pixel masks. The +outputs are these postage stamps and pixel masks, which are sent to +the IPP Pixel Server. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Metadata Required} + +The following metadata associated with the images are required for +Phase~2 operation: +\begin{itemize} +\item The orthogonal transfer (OT) image shifts made during the +exposure --- in order to create a convolution kernel; +\item Time of observation --- for selecting the appropriate detrend +images; +\item Filter --- for selecting the appropriate detrend images; +\item Telescope identification --- for selecting the appropriate +detrend images; +\item Exposure time --- for the photometric calibration; +\item Detector gain --- for calculating photometric errors; and +\item Detector read noise --- for calculating photometric errors and +determining the quality of the overscan; +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Pixel Masks} +\label{ap:masks} + +This section describes the requirements on Bad Pixel Masks (BPMs). +These will consist of bit masks for each pixel. For Phase 2, flags +are required for at least each of the following pixel attributes: +\begin{enumerate} +\item The pixel is a charge trap; +\item The pixel is a bad column; +\item The pixel is saturated in the A/D converter; +\item The pixel is non-positive in the flat-field; +\item The pixel is part of a row that has excess noise; and +\item The pixel is determined to be a cosmic ray, based on its +morphology. +\end{enumerate} + +Of these, only masks for the charge traps need to be grown by the +extent of the OT convolution kernel. For other pixel types, +orthogonal transfer of the flux in this pixel will not (necessarily) +affect the flux in neighbouring pixels + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Object Catalogs} +\label{ap:catalogs} + +Object catalogs from Phase 2 shall consist of at least the +following elements for each object: +\begin{enumerate} +\item Object centre, with corresponding errors; +\item Object magnitude, with corresponding error; +\item Object isophotal magnitude, with corresponding error; +\item Object FWHM; +\item Object elliptical axis lengths; and +\item Object position angle for ellipse. +\end{enumerate} + +Though further details may be required for catalogs in Phase~4, +the above details are minimum requirements for Phase~2 catalogs. + +\tbd{Phase 2 command: P2 (exposure.ota.fits)} +\tbd{Megacam: P2 654321o.fits[ccd00] - what are output names?} +\tbd{PS FPA is saved as a collection of MEF files. Megacam FPA is + saved as a single MEF file. how to handle this difference?} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Phase 3 : exposure analysis} + +The Phase 3 system operates on the combined Phase 2 results from an +FPA to determine improved solutions for the image calibrations and to +provide the parameters needed by Phase 4. The Phase 3 output is saved +by the IMD, and consists largely of improved values of the +calibrations already determined by Phase 2. The analysis performed by +this pipeline consists of: + +\begin{itemize} +\item improved astrometric solution based on comparison between + objects in the images and the astrometric reference. +\item improved background model based on the full telescope field, or + fields. +\item photometric solution based on comparison to photometric + standards +\end{itemize} + +\begin{figure} +\begin{center} +\resizebox{8cm}{!}{\includegraphics{pics/phase3}} +\caption{ \label{phase3} Phase 3 dataflow} +\end{center} +\end{figure} + +In the Phase 2 analysis, the astrometric solutions were determined +independently for each chip. These solutions are limited by the +assumption of a static distortion and \tbd{by the accuracy of the +astrometric reference}. In the phase 3 analysis, the astrometric +solutions of the $N$ FPA images are improved by \tbd{???}. + +\tbd{what is the expected accuracy of the relative astrometric + solution compared to the absolute astrometric solution?} + +\tbd{for image combination in phase 4, should we use relative + astrometry to map N-1 images to 1, or are we sufficiently accurate + to use absolute astrometry to map N images to the sky-cells?} + +In the Phase 2 analysis, the background is determined based only on +the available sky in a single chip. However, the background +structures are normally correlated on the scale of the FPA, so an +improved background solution can be determined by combining the +information from many chips. \tbd{is the background correlated +between FPAs?} + +\tbd{Phase 3 photometric improvement??} \tbd{Phase 3 determined +accurate relative photometry between the N images which are to be +combined in the Phase 4 analysis. Is this more accurate than the +absolute photometry solution? (probably)} + +In the Phase 4 analysis, the $N$ FPA images are optimally combined to +create a single image of the sky with bad-pixel and cosmic-ray +rejection. This combination requires the calculation of a set of PSF +kernels to convert each of the input images to a single, common PSF. +These PSF kernels are determined from the per-chip PSFs measured in +Phase 2. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Phase 4 : image combination} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Phase 4 Concept} + +Phase 4 processing within the \PS{} image processing pipeline is +the final stage of processing for a science image. It operates on +each sky cell that has overlapping imaging data from the exposure(s) +being processed, and produces the main output image data products of +the pipeline --- the difference images and a deep static sky image --- +along with the associated catalogs of static and variable sources. + +\begin{figure} +\begin{center} +\resizebox{8cm}{!}{\includegraphics{pics/phase4}} +\caption{ \label{phase4} Phase 4 dataflow} +\end{center} +\end{figure} + +Prior to Phase 4, the Phase 3 process produces the following products: +\begin{itemize} +\item bias-subtracted, flattened, sky-subtracted images; +\item photometric calibration; +\item astrometric calibration with mapping to sky cells; and +\end{itemize} +These will each be used by the Phase 4 modules: +\begin{enumerate} +\item Combine Images; +\item Identify Sources; +\item Transient Identification; and +\item Add to Static Sky. +\end{enumerate} +These modules are each explained below. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Combine Images} + +The first module for Phase 4 is to combine the images from each +telescope, rejecting artifacts such as cosmic rays and low altitude +streaks. The inputs to this module are: +\begin{enumerate} +\item the sky-subtracted images that overlap the sky cell (or portions +thereof) --- from the IPP Pixel Server (or directly from Phase 3); +\item a \tbd{linear} map for the image pixels of each detector to the +sky cell pixels --- from Phase 3; +\item photometric calibration (zeropoint) for each image --- from +Phase 3; and +\item a (relative) weighting for each image proportional to the +signal-to-noise (i.e.\ sky noise divided by the square of the seeing) +--- from metadata associated with the images. +\end{enumerate} + +The module maps the detector images to the sky cell using the specified +linear transformations, combines the images with strong rejection +criteria and uses the combined sky cell image to identify artifacts in +the original detector images. It is desirable that the artifacts are +masked in the detector plane (i.e.\ before mapping to the sky cell) so +that they are not smeared out by the mapping; alternatively, the CR +mask needs to be grown by an additional pixel (which is likely +faster). The mapped and masked detector images are then optimally +combined using the specified weighting. Both sets of combinations use +the photometric calibration for the images to set the relative scales +of the input images. The final combination should have the adopted +Universal zeropoint (25 mag, configurable). The limiting magnitude +for the combined sky cell image should also be estimated. + +The outputs from this module are: +\begin{enumerate} +\item The combined sky cell image --- sent to the IPP Pixel Server +and/or the next module; +\item Limiting magnitude of the combined sky cell image --- metadata +associated with the combined sky cell image, and used for a later module +in Phase 4; and +\item Catalog of sources on the combined sky cell image --- sent to +the IPP Object Database. +\end{enumerate} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Identify Sources} + +This module identifies sources in the combined sky cell image. The +input is the combined sky cell image, which is obtained from the IPP Pixel Server +or the previous module. + +Sources are identified on the combined sky cell image by convolving +with the PSF and searching for peaks above the noise. The output +is the catalog of sources on the combined sky cell image, which is to +the IPP Object Database. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Transient Identification} + +This module identifies variable/moving sources. The inputs are: +\begin{enumerate} +\item The combined sky cell image --- from the previous module or the +IPP Pixel Server; and +\item The current static sky image --- from the Sky Image Server. +\end{enumerate} + +The module subtracts the current static sky image from the combined sky +cell image. In order to do so, the PSFs need to be matched. This is +done by convolving the image that has the narrower PSF with the +kernel, which is the ratio of the two PSFs (this should be done with a +fit to the kernel instead of just using the data). It should be +sufficient to assume that the kernel is constant over the sky cell +(otherwise, the sky cell can be broken into smaller sections). + +The subtracted image is scoured for point sources above the noise +threshold, as well as short and long streaks caused by asteroids and +satellites, respectively. It may be neccessary to determine whether +the detection is false by virtue of its PSF (a cosmic ray missed by +the combination script should have a very narrow PSF, at least in one +dimension), or negative pixels surrounding a positive core (caused by +a bad subtraction, in turn caused by a bad kernel). + +If the subtraction is very bad (many false detections), then Phase 4 +for this sky cell should fail neatly, with a flag for the human +supervisor. Otherwise, all variable sources identified in the +subtracted image should be masked in the combined sky cell image. The +pixels from the combined sky cell image for point sources and short +trails (asteroids) should be saved (say, 3 $\times$ FWHM in radius +surrounding the source, configurable). The long trails (satellites) +should be removed in the combined sky cell image and the subtracted +image, from edge to edge. The dividing limit between short and long +trails shall be a configurable parameter, initially set to 15 degrees +per day. + +The module outputs: +\begin{enumerate} +\item Combined sky cell image, with all variable sources masked --- +used for the next module; +\item Subtracted image, with long trails masked --- sent to the IPP +Pixel Server; and +\item Catalog of variable sources --- sent to the IPP Object +Database. +\end{enumerate} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Add to Static Sky} + +This module adds the combined sky cell image into the static sky, so +that a deep image of the sky may be formed. This step should only be +performed if the new data is of sufficient quality that it will not +degrade the static sky image. The inputs are: +\begin{enumerate} +\item The combined sky cell image with variable sources masked --- +from a previous module; +\item The current version of the static sky --- from a previous module, +or the IPP Pixel Server; and +\item Relative weightings, based on the relative signal-to-noise in +each of the images --- estimate made from metadata associated with +each image. +\end{enumerate} + +The sky cell image is added to the static sky. The sky cell image +should already be photometrically accurate (when combined), and +variable sources have been masked, so it is safe to simply add the +images, employing the weightings. Sources should be identified on the +new static sky, and the limiting magnitude of the new static sky image +estimated. + +The output is: +\begin{enumerate} +\item The new static sky image --- sent to the Sky Image Server; +\item The Catalog of sources on the new static sky image --- sent to the IPP Object Database; and +\item The estimated limiting magnitude for the new static sky --- +metadata associated with the the new static sky image. +\end{enumerate} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Notes} + +\begin{itemize} +\item Catalogs should include positional information ($x,y$, with +associated errors), photometry (with associated error), and shape +parameters (FWHM, major and minor axes, position angle). +\item Limiting magnitudes can be obtained by photometering many +regions of blank sky (if possible), and (robustly) estimating the mean +and standard deviation (in counts). The limiting magnitude is the +magnitude corresponding to 3 (configurable) standard deviations above +the mean. +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Calibration Image Processing} + +The IPP Calibration Image Pipelines perform the tasks needed to +generate high-quality calibration images from the input image +dataset. These operations may be performed on whatever timescales are +appropriate and necessary to maintain the quality and relevance of the +calibration images. There are four distinct types of calibration +image pipelines: the basic detrend creation pipeline, the photometric +correction image creation pipeline, the fringe pattern generation +pipeline, and the sky foreground pattern generation pipeline. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Cal 1: Basic detrend image creation} + +The basic detrend image creation pipeline collects the appropriate +input detrend images (bias, dark, dome flat, etc) and generates a +master image by combining the input images in some optimal way +\tbd{median/sigma-clipping/etc}. The master image is used to +determine input image residuals so that poor input images can be +iteratively rejected. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Cal 2: Fringe pattern and sky foreground model creation} + +The fringe model creation and sky foreground model creation pipelines +use night-sky images with sufficient flux to measure the fringe or sky +models. The input images are processed and optimally combined to yield +a set of correction fringe patterns. The fringe pattern creation and +the sky foreground pattern creation have a similar processing +structure: both require processing of the input images, both determine +a set of principal components as a function of specific input +parameters. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{Cal 3: Photometric flat correction image creation} + +The photometric flat-field correction uses images which have been +dithered with a large range of spatial scales, combined with the +uncorrected flat-field images, to generate a correction to the +flat-field image. This correction compenstates for non-uniform +illumination of the detector during the initial flat-field generation +stage. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Calibration Test Processing} + +The calibration test processing tests observations to determine if the +calibrations need updating. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{CalTest 1: Detrend frame testing} + +A newly-acquired master detrend frame, having been combined (using Cal +1 or Cal 2) are simply differenced from the old detrend frames. If +there exist significant residuals, the newly-acquired detrend frame +is adopted as the detrend frame of choice. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{CalTest 2: Photometric flat correction testing} + +Newly-acquired photometry of many objects (initially, this may be +standard star fields, but once the PS1 catalog is available, it should +be possible to use all photometry acquired over a given time period) +are compared with previously-acquired photometry. If there exist +significant residuals, a new photometric flat correction should be +produced from the newly-acquired photometry. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Reference Catalog Processing} + +The IPP reference catalog pipelines use the data in the IPP Metadata +Database and the IPP Object Database to determined improved +astrometric and photometric calibration references. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{AstroRef: Astrometric Reference Catalog creation} + +This processing stage shall use many observations over a given time +period to fit a consistent global astrometric solution, resulting in a +high quality and internally-consistent astrometric catalog that may be +published. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subparagraph{PhotoRef: Photometric Reference Catalog creation} + +This processing stage shall use many observations over a given time +period to fit a consistent global photometric solution, resulting in a +high quality and internally-consistent photometric catalog that may be +published. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Reference Catalogs} + +The IPP will employ reference catalogs in order to calibrate the +photometry and astrometry. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Astrometric Reference Catalog} + +For PS1, this shall be UCAC. + +For PS4, this shall be the PS1 catalog. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Photometric Reference Catalog} + +For PS1, absolute photometry will not be available until the master +fit which will be performed when all data is taken. For purposes of +relative photometric extinction, the guide star brightnesses should be +sufficient. + +For PS4, the PS1 catalog shall be used. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Modules} + +\tbd{What goes here? There will be modules?} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{\PS{} Library} + +See PSDC-430-007 for the design of the \PS{} Library, PSLib. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Internal Interfaces} + +\tbd{To be updated and expanded.} + +Internal interfaces consist of queries to the IMD or IPS, insertion of +data in the IMD, IPS, or IOD, or processing configuration files. The +science and calibration image processing pipelines make requests for +images from the IPS, metadata from the IMD, and push their results +back onto the IPS and IMD. The reference catalog pipelines make +requests on the IMD and the IOD and push their results back to the +IOD. The scheduler creates input processing configuration files for +the processing pipelines and queries the IMD and IPS and pushes +results back to the IIS. + +FITS Images + +FITS Tables + +XML + +SQL queries + +C:DB interactions + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{External Interfaces} + +\tbd{This whole section to be updated.} + +This subsection describes the interfaces between the IPP and other +\PS{} systems and the external clients. The interfaces are +illustrated in Figure~\ref{fig:functionalities}. Incoming data is +received by either the IPS (pixels), the IMD (metadata), or the IOD +(objects). Requests for data by external clients are also made to +these three databases. Requests for data made by the IPP are +generated by the IPP Scheduler or the science processing pipelines. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{OATS} + +The Summit Pixel Server (SPS) sends raw image data, image metadata, +and enviromental metadata to the IPP. The IPP provides an interface +mechanism by which the SPS can register new images with the IPP, which +sends them to the appropiate subsystem: The image pixel data is sent +to the IPS while the metadata is sent to the IMD. + +The \PS{} Telescope Scheduler (PTS) sends information about the +telescope schedule to the IPP: observing plan for the night, or longer +time scales. The IPP scheduler sends telescope schedule requests to +the PTS (i.e.\ calibration needs). + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Published Static Sky Server} + +The Static Image Server provides segments of the current static sky +image to the IPP on demand. IPP subsystems which require this data +will block until it is available or timeout if it is not. The IPP +provides updated static sky images to the SIS when available. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Object Database} + +The Master Science Object Database receives new object photometry from +the IPP. The IPP IOD acts as a cache for object photometry data; +\tbd{an IPP subsystem will send photometry data in batches on some +timescale. Is this a function of the IOD?} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Moving Object Processing System} + +The Moving Object Processing System interfaces with the IPP to receive +the objects detected in the difference images via queries to the IOD. +The MOPS may interface with the IMD as needed. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Other Client Science Pipelines} + +The client science pipelines may interface with the IPP via requests +for data from the IMD, IOD, or IPS. \tbd{how many clients max? / how +much data?} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Computer Hardware} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Overview} + +This document discusses the likely range of the \PS{} Image +Processing Pipeline (IPP) hardware requirements. The hardware +requirements addressed in this document consist of: + +\begin{itemize} +\item Total Disk Volume +\item Total Processing Power +\item Sustained Switch Bandwidth +\item Sustained Node Network I/O +\item Sustained Disk I/O +\end{itemize} + +Even without the complete IPP design, it is possible to identify the +major drivers on the hardware requirements. The total disk volume +requirements are dominated by the need to store raw images for a +certain period, the need to store calibration images for a longer +period, and the need to store the static sky images. Of the various +analysis pipelines, and depending on the data organization as +discussed below, Phase 2 and Phase 4 present the most significant +demands in terms of data I/O throughput on the network. Phase 2 and +Phase 4 also present the most significant CPU demands. In this +discusion, Phase 2 refers to the per-chip pre-processing in which the +instrumental signature is removed and a first pass object detection is +performed. Phase 4 refers to the multiple chip combination in which +the pre-processed images are merged and combined, in both addition and +subtraction, with the static sky image, and up to three object +detection passes are performed. + +This document does not address the hardware requirements implied by +the Phase 0, 1, or 3 stages, nor the load required by the calibration +image creation stages. In the first instance, the operations are only +performed on the metadata and are extremely minimal both in terms of +data I/O and computation requirements. In the second case, the +processing is less time critical than the per-image processing and is +performed only infrequently (once per night to once per week or +month). This document also does not address any hardware requirements +introduced by the metadata manipulation. The software implementation +for metadata storage (RDBMS, FITS tables, etc) will have a very large +impact and will be evaluated along with the needed hardware at a later +date. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Scenarios} + +We will address the various hardware requirements by referring to a +set of data processing and data organization scenarios. The actual +hardware requirements will depend on design decisions which are not +yet available. It is possible to define the data organization in ways +which will minimize the hardware requirements, but which will increase +the software development effort. We will discuss both the worst-case +data organization scenario, which does not require significant +intelligence in the software systems, and the optimal data +organization scenario, which will require the software to track the +location of data products more carefully. In addition, this document +will address the data requirements of the complete \PS{} pipeline +with 4 telescopes as well as the single-telescope \PS{}-1 scenario +based on the Design Reference Mission [REF]. + +The IPP hardware system must provide both data storage and +computational resources. The IPP requires relativley large amounts of +data storage space, primarily for the image data. Image data is +organized in two categories. First, there is the per-chip data -- +data associated with specific chips, including the raw images, the +calibration images, and temporary processed images at various stages. +Second, there is the data associated with the static sky imagery, +which is in turn organized into smaller sky-cell units. The first +assumption we make is that the hardware is organized into nodes which +provide both data storage and computational resources. The second +assumption we make is that the data storage nodes are divided into two +classes: those which deal with the per-chip data and those that +provide the static sky storage. In addition, we assume that the +computational tasks related to Phase 2 take place on the per-chip +storage nodes and the Phase 4 computation takes place on the static +sky storage nodes. + +Figure~\ref{hardware} shows our basic concept for the hardware +organization for the IPP. This diagram shows the two types of compute +nodes: chip-level processing and storage nodes (dominated by Phase 2) +and static sky processing and storage nodes (mostly Phase 4). Also +shown are two switches used in this configuration; although it is +currently possible to buy a single switch which would have a +sufficient number of GigE ports for both sections of the PS-1 system, +such a two-switch organization may be needed for the full \PS{} +system. In such a case, the interswitch communication must also meet +the required throughput needs. We discuss the hardware requirements +in the assumption that such an organization will be necessary. + +The way in which the images are distributed among the storage and +compute nodes will largely determine the I/O bandwidth requirements. +For data bandwidth requirements calculations, it is necessary to make +some assumptions about the data organization. For the purposes of +this document, we explore two extreme-case options: +\begin{itemize} +\item Random Data Distribution --- Detector \& Sky data is randomly + distributed within the compute node of a given type (ie, chip data + is randomly distributed among the detector compute nodes). +\item Optimal Data Distribution --- Detector \& Sky data is optimally + distributed to compute Detector/Sky nodes (chip processing is always + on a machine with local chip data). +\end{itemize} +A second factor which will have a significant impact on the I/O +requirements is the image storage format for the processed and +calibration images. We have two basic choices: 32 bit floating point +format or 16 bit integer format with appropriate scaling. In the +former case, additional dynamic range is retained, while in the latter +case, we reduce the data volume by a factor of 2. While some may +argue that the higher dynamic range is necessary, arguments can be +made that the 16 bit range is sufficient. (In particular, the 16 bit +data provides a dynamic range far above the expected 1/1000 fractional +accuracy of the flat-field images). A related question is the number +of calibration images needed by the processing system. Since the +complete analysis is not yet defined, this number is difficult to +ascertain. However, we can make a range of assumptions which are +reasonable. We therefore adopt two data volume scenarios to explore +these possibilites: +\begin{itemize} +\item Standard Data Volume - 32 bit data for processed and calibration + images, average of 7 calibration frames per image. +\item Minimal Data Volume - 16 bit data for processed and calibration + images, average of 4 calibration frames per image. +\end{itemize} +In the discussion that follows, we explore the hardware requirements +implied by the collection of four combinations of these two sets of +scenario options. + +\begin{table} +\begin{center} +\caption{Hardware Throughput Tests \label{existing-hardware}} +\begin{tabular}{lrrrr} +\hline +\hline +Test & where \& when & model & result \\ +\hline +node I/O & CFHT 11/2002 & Intel 1000 Gigabit & 35 - 40 MB/s sustained \\ +node I/O & CFHT 2/2004 & Intel 1000 Gigabit & 65 - 70 MB/s sustained \\ +RAID write & CFHT 2/2004 & 3ware RAID cntl + IDE & 110 MB/s sustained \\ +Switch Load & VeriTest & Cisco & 3 GB/s (for 32 ports) \\ +\hline +\end{tabular} +\end{center} +\end{table} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Existing Hardware Throughput} + +We have collected a few representative tests of various pieces of +modern hardware to give a reference for the throughput capabilities. +A number of hardware configurations have been tested at CFHT for the +Elixir project, and we include here their recent reported hardware +RAID-5 I/O speeds and GigE card speeds. We also have included data +from VeriTest studies of Cisco switch throughput, commissioned by +Cisco for a 32 port GigE switch. These tests are summarized in +Table~\ref{existing-hardware}. + +\begin{table}[b] +\begin{center} +\caption{Data Storage Requirements \label{storage}} +\begin{tabular}{lrrrr} +\hline +\hline + & Standard / PS-4 + & Standard / PS-1 + & Minimal / PS-4 + & Minimal / PS-1 \\ +\hline +Raw data & 300 TB & 75 TB & 300 TB & 75 TB \\ +static sky & 512 TB & 64 TB & 256 TB & 32 TB \\ +calibration frames & 175 TB & 18 TB & 17 TB & 5 TB \\ +metadata db & 2 TB & 2 TB & 0.2 TB & 0.2 TB \\ +object db & 60 TB & 4 TB & 60 TB & 4 TB \\ +\hline +totals & 1050 TB & 163 TB & 633 TB & 116 TB \\ +\hline +\end{tabular} +\end{center} +\end{table} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Data Storage Requirements} + +The \PS{} IPP data storage requirements may be divided into five +principal areas: raw image data, static sky image data, master +calibration images, the metadata database, and the object database. +We discuss each of these data items and their impact on the data +storage requirements for the IPP, and identify the impact of the +minimal vs standard data storage requirements as well as the +requirements specifically for PS-1. Table~\ref{storage} summarizes +the data storage requirements in the different scenarios. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Raw Data Storage} + +There are two basic image types which will be acquired: night-time +science images and calibration images. The night-time science images +consist of 1Gpix per image, or 2GB in raw format. At nominal cadence, +the 4 telescopes can obtain images at a sustained rate of 1 image per +30 seconds per telescope for the entire night of 10 hours (36000 +minutes). A total of 100 calibration images per night would be a +substantial overestimate of the typical expectation. Combining these +numbers, we can expect to receive a total of 1300 image per telescope +per night, 5200 image total, or 10.4 TB of data per night. The total +data storage requirements for the raw data are governed by the number +of nights' worth of data we are required to keep online. A reasonable +number is one month to allow a full moon's cycle. Thus, for raw image +storage, we require a total of 300 TB data storage. For PS-1, this +number is simply scaled down by a factor of 4. The choice of the +minimal data volume does not affect these numbers because the raw data +is already stored with 16 bit pixels. + +\tbd{The PS-1 design reference may now require storage of the entire +first year of data, calculated to be 200 TB.} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Static Sky Data Storage} + +The static sky is represented by images with 0.2 arcsec per pixel. +There will be one summed image and one weight image for each of the 6 +filters, each stored in floating point format. At this resolution, +there are 324 Mpix per square degree, and we will observe a potential +total area of 30,000 square degrees. Allowing for 10\% overage for +overlapping tiling, we require a total of 10.7 Gpix to cover the sky +once, or a total of $\sim 512$ TB for the static sky images. In the +minimal data volume scenario, this value is reduced by a factor of 2, +while in PS-1, the reduction is a factor of roughly 8 because we only +intend to store the static sky for the ecliptic plane survey and the +small IPP verification program. + +\tbd{This last point is no longer valid - the PS-1 static sky may +require the entire 3pi.} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Calibration Frame Storage} + +The possible required calibration frames consist of the bias, dark, +and mask images, along with one flat, one flat-correction, and +multiple sky/fringe library frames per filter. In fact, not all types +are needed at all stages. For the standard data volume, we assume an +average of 7 calibration frames per image and filter. This results in +a total of 42 master calibration image per telescope. If we intend to +keep all master calibration frames for the project lifetime, and +generate a new master on a weekly basis (a reasonable time-scale), +then we can expect to require a total of 175 TB of calibration image +by the end of the 5 year lifetime of the project. For the case of +PS-1, the time period is only 2 years, and there is only 1 telescope, +resulting in a factor of 10 reduction in the volume. For the minimal +data case, we reduce the volume by another factor of 3.5. We also note +that this is likely to be a drastic overestimate as we are unlikely to +need to regenerate all master calibration frames on a weekly +time-scale. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Metadata Database Storage} + +The metadata data storage requirements are driven by the need to store +the data for the project lifetime. There are two types of metadata +generated at the summit: data associated with images and environmental +data. The environmental data consists of measurements on a regular +cadence, roughly 1 per minute, of a variety of parameters. We suggest +an expected of 1kB per entry, for a total of 2.6 GB over the lifetime +of the project. PS-1 will represent a smaller amount of data per +minute, and also a factor of 2.5 fewer minutes. We suggest PS-1 may +have a total environmental metadata set smaller by a factor of 5. The +additional systems, such as the DIMM, SkyProbe, NIR Sky Camera, and +the LRProbe will have higher data requirements, but should be +considered as separate, self-contained systems. Their data products +are distilled to a limited number of parameters per minute which are +included in the 1kB given above. Furthermore, items such as +guide-star history, if saved, will be saved with the image data and +represents only a small fraction of the total image data volume. Some +subset of the telescope diagnosic information may be a high volume +data product as well, but only retained by the telescope control +system for the purpose of diagnostic studies. Such data will be +excluded from this analysis. + +The image metadata consists of values associated with the FPA (4), the +chips (240), and the Cells (15360). Aside from the guide star +history, the total data requirements for each of these entries will be +scaled by the number of bytes required for the metadata from each data +level. Clearly, if the Cell entry is allowed to be large, it will +dominate the total Metadata data volume. If we suggest an expected +number of 64~bytes per Cell, 256~B per chips, and 1~kB per FPA, we find a +total metadata volume per exposure of roughly 1~MB, completely +dominated by the Cell metadata. With the exposure rates above, we +find a total of metadata volume of 1.8~TB over the lifetime of the +project. For PS-1, the total volume is reduced by a factor of 2.5 +(for the shorter lifetime) and another factor of 4 (for the lone +telescope). Neither data quantity is affected by the minimal vs +standard data volume choice. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Object Database Storage} + +The hardware requirements for the IPP object database are rather +flexible: the total volume depends critically on the depth to which +the object detection analyses are performed (and thus the total number +of object detections) and the number of object parameters which are +measured. We can make very rough estimates that the total number of +detections over the 5 year lifetime of the project may be in the +vicinity of $5\times10^{11}$. We can conservatively estimate the +number of bytes needed to represent each detection as 128 B, resulting +in a total data storage for the object detections of 60 TB. However, +this number depends strongly on the timescale for which the IPP is +required to maintain all object detections, and may potentially be +significantly reduced. For the case of PS-1, the total number of +detections is likely to be reduced by a factor of 4 for the number of +telescopes, and potentially another significant factor ($\sim 4?$) by +limiting the depth of object detections. Again, the minimal data +volume scenario is irrelevant to the object database volume. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{CPU Requirements} + +Phase 2 and Phase 4 dominate the processing requirement, primarily +because they must keep up with the image delivery rate of 1 per 30 +seconds. We have performed benchmarks of a demonstration version for +both the Phase 2 and Phase 4 analyses. + +For the Phase 2, a substantial fraction of the processing time is +consumed by the need to perform FFTs on the images in order to +convolve them with the guide-star kernel, and in the smoothing used +for the object detection process. Additional processing time is +needed by the object detection, deblending, and analysis. Experiments +with the FFTW package show that FFTs may be performed on Intel +processors at rates of approximately 0.25~GHz-sec / Mpix for data sets +of order 1 Megapixel. The FFTs required for the Phase 2 analysis are +performed on the 512$^2$ pixel cells, so these numbers may roughly be +scaled linearly to determine the total time required for chip +processing. A single FFT on a full chip, with 64 cells, therefore +requires roughly 4~GHz-sec. For the full Phase 2 analysis, there are +roughly 4 single direction FFTs required excluding those associated +with object detection; thus the total processing time for these FFTs +is approximately 16~GHz-sec. The addtional analysis steps, excluding +object detection and characterization, account for a small fraction of +this compute time, which we estimate at 10\%. The object detection +stage depends somewhat on the depth to which the analysis is +performed, and the number of measurements made per object. Typical +analysis performed by the Sextractor routine, which performs a +substantial number of per-object analyses, requires 27~GHz-sec for a +full chip, including the FFTs used for smoothing. We can therefore +assume a total of 50~GHz-sec per chip for the Phase 2 processing. +This converts to a total of 12,000~GHz-sec for a complete major frame. + +For Phase 4, the main computational tasks are combining the multiple +images, with cosmic-ray rejection, and performing the object detection +tasks. Nick Kaiser has done tests of the Phase 4 image combine and +rejection stages, and finds a total processing time of roughly +96~GHz-sec for a full stack of 4 chips. If we add in an additional +34~GHz-sec for detailed object detection and image differencing, we +find a conservative estimage of 130~GHz-sec for a 4-image chip stack, +equivalent to 7800~GHz-sec for a major frame. + +For PS-1, the data processing will clearly require a smaller amount of +computational resources because of the lower image rate. However, the +total number of GHz-sec required for the complete analysis of 4 input +images and the combination with the static sky will remain +more-or-less the same. Some reduction in the load may be gained by +reducing the complexity and depth of analysis for PS-1. Depending on +the details and depth of the analysis, we may reduce the computational +load by a factor of 2. + +\begin{table} +\begin{center} +\caption{Data Scenarios (MB per Chip or Sky-cell) \label{scenarios}} +\begin{tabular}{lrrrr} +\hline +\hline + & Random / Standard & Random / Minimal & Optimal / Standard & Optimal / Minimal \\ +\hline +{\em Phase 2 input} & & & & \\ +from summit & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB \\ +input image & 32 MB & 32 MB & {\bf 32 MB} & {\bf 32 MB} \\ +calibration & $7 \times 64$ MB & $4 \times 32$ MB & {\bf 7 $\times$ 64 MB} & {\bf 4 $\times$ 32 MB} \\ +mask image & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ +\hline +network I/O: & 560 MB & 232 MB & 64 MB & 64 MB \\ +disk I/O: & (560 MB) & (232 MB) & 496 MB & 168 MB \\ + & & & & \\ +{\em Phase 2 output} & & & & \\ +output image & 64 MB & 32 MB & {\bf 64 MB} & {\bf 32 MB} \\ +output mask & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ +image to P4 & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB \\ +mask to P4 & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB \\ +\hline +network I/O: & 200 MB & 100 MB & 120 MB & 60 MB \\ +disk I/O: & (80 MB) & (40 MB) & 80 MB & 40 MB \\ + & & & & \\ +{\em Phase 4} & & & & \\ +input images & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & & \\ +input masks & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & & \\ +static sky & 64 MB & 64 MB & & \\ +static weight & 64 MB & 32 MB & & \\ +\hline +input: & 608 MB & 336 MB & & \\ +output: & 192 MB & 128 MB & & \\ +\hline +\multicolumn{5}{l}{\em Bold-faced entries are access to local-disk} \\ +\multicolumn{5}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\ +\end{tabular} +\end{center} +\end{table} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Per-Node I/O Requirements} + +Data I/O per node is defined as the number of bytes per second passed +through the node's network adapter. The data throughput for each node +depends strongly on the scenarios identified above. In this section, +we identify the data which is passed between nodes for each of the +different scenarios. Table~\ref{scenarios} lists the per-node data +I/O for the four scenarios. + +For PS-4, there are only 30 seconds of compute time allowed for each +of the Phase 2 and Phase 4 analyses. We use the data I/O volumes and +some assumptions about expected network and disk bandwidth to estimate +the I/O and processing timeline for the four scenarios. From this +analysis, we can judge the total CPU requirements in terms of GHz, not +just GHz-sec. We have assumed that GigE network adapters are capable +of delivering data at 50MB/sec sustained and that a disk RAID can +deliver sustained 100 MB/sec reads and writes. These numbers are +conservative estimates based on recent tests discussed above. Using +these assumptions, Table~\ref{throughput} lists the time allocations +for the complete set of scenarios for the case of PS-4. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Random / Standard Data Scenario} + +In the Random Data Distribution scenario, there is a single CPU +allocated to each chip in the detector farm and a single CPU for each Sky +cell process. The chip data are stored across random machines in the +detector farm, with the result that every Phase 2 processing requires +network access to the data. For each science chip which is +observed, each detector node will read from the network a total of 560 MB +(the 2 raw images for data storage and the 7 calibration frames, along +with one mask and one raw input image) and write a total of 200 MB +(one processed image and the mask along with the 1.5 processed images +and masks for the Phase 4 analysis). Given the assumption of 50 MB/s +from the network adapter, the total data volume implies an I/O period +of 15.2 seconds. Note that the disk I/O is parallel with the network +I/O and substantially underfills the disk bandwidth. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Random / Minimal Data Scenario} + +In the Random-Minimal, there is a single CPU allocated to each chip in +the detector farm and a single CPU for each Sky cell process, and the +chip data are stored across random machines in the detector farm. +However, the calibration and the processed science images are stored +at 2 bytes per pixel, the mask is set at 4 bits per pixel, and only 4 +calibration images are assumed. For each science chip which is +observed, each detector node will read from the network a total of 232 MB +(the 2 raw images for data storage and the 4 calibration frames, along +with one mask and one raw input image) and write a total of 100 MB +(one processed image and the mask along with the 1.5 processed images +for the Phase 4 analysis). Given the assumption of 50 MB/s from the +network adapter, the total data volume implies an I/O period of 6.6 +seconds. Again, note that the disk I/O is parallel with the network +I/O and substantially underfills the disk bandwidth. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Optimal / Standard Data Scenario} + +In the Optimal Data Distribution scenario, there is a single CPU +allocated to each chip in the detector farm and a single CPU for each +Sky cell process. In addition, all data for the specified chip are +stored on local disks attached to the same computer as the CPU, with +the result that all Phase 2 I/O is made to a local disk. For each +science chip which is observed, each detector node will read from the +network a total of 2 raw images (one for the original image, one for +the backup copy) and write an average of roughly 1.5 processed images +and masks to the Phase 4 machines for a total of 184 MB of network +I/O. During the processing stage, the detector node will read from +disk a total of 496 MB (7 calibration frames at 64 MB each, one 16 MB +mask, and one raw science image at 32 MB) and write a total of 80 MB +(one processed image at 64 MB and one mask at 8 MB). Given the +assumptions for the network and disk bandwidths (50 MB/s and 100 MB/s +respectively), the data volumes imply a total I/O period of 9.5 +seconds. In this instance, the network I/O is presumed to be +sequential with the disk I/O. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Optimal / Minimal Data Scenario} + +In the Optimal / Minimal Scenario, the minimal data sizes are used +with the optimal data distribution scheme. In this case, we reduce +the disk I/O volume to 168 read and 40 MB write, and the network +traffic to 124 MB. Given the assumptions for the network and disk +bandwidths, the data volumes imply a total I/O period of 4.6 seconds. +Again, the network I/O is presumed to be sequential with the disk I/O. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Phase 4 Node I/O Requirements / Standard Data Volume} + +Although it is easy to arrange the detector data in such a way that +the majority of I/O is performed locally, it is not as easy to arrange +this for the Static Sky data used by the Phase 4 analysis. We +therefore make the assumption that the Phase 4 analysis will require +all input detector data to be loaded across the network, as well as +all Static Sky data. This is somewhat of an overestimate as some of +the Static Sky data will be processed by machines with the data stored +locally, and clever Static-Sky data organization schemes can enhance +this chance. + +In the Phase 4 analysis, the images from the 4 separate telescopes are +combined into a single image, confronted with the appropriate segment +of the static sky, with output difference image and updated static sky +image. If we restrict input access to the individual chip cells, the +maximum read overhead is 50\% (need to read a 10x10 set of cells for +an 8x8 input image). If the processing is performed on Static Sky +segments equivalent in size to the chips, the input data is 608 MB (384 +MB of processed science image, 96 MB of mask images, 64 MB of static +sky image and 64 MB of static sky weight map) while the output data is +192 MB (static sky, weight map, and difference image, each 64 MB). +Thus, we require a total of 800 MB network I/O. Given the network +bandwidth, this implies an I/O period of 16 seconds for Phase 4. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Phase 4 Node I/O Requirements / Minimal Data Volume} + +In the minimal data volume scenario, the Phase 4 analysis volume is +significantly reduced. The total volume of input data is 336 MB (192 +MB of processed science image, 48 MB of input mask, 64 MB of static +sky image and 32 MB of static sky weight map) while the output data is +128 MB (64 MB static sky, 32 MB weight map, and 32 MB difference +image). Thus, we require a total of 464 MB network I/O, which implies +an I/O period of 9.3 seconds. + +\begin{table} +\begin{center} +\caption{Data Throughput for 4 Scenarios \label{throughput}} +\begin{tabular}{lrrrr} +\hline +\hline +& +\multicolumn{1}{c}{Random / Standard} & +\multicolumn{1}{c}{Random / Minimal} & +\multicolumn{1}{c}{Optimal / Standard} & +\multicolumn{1}{c}{Optimal / Minimal} \\ +\hline +Phase 2 per-node network I/O & 15.2 s & 6.6 s & 3.7 s & 2.5 s \\ +Phase 2 per-node disk I/O (read) & (5.6 s) & (2.3 s) & 5.0 s & 1.7 s \\ +Phase 2 per-node disk I/O (write) & (0.8 s) & (0.4 s) & 0.8 s & 0.4 s \\ +Phase 2 CPU total & 14 s : 860 GHz & 23 s : 520 GHz & 20 s : 600 GHz & 25 s : 480 GHz \\ +Phase 4 per-node I/O & 16 s & 9.3 s & & \\ +Phase 4 CPU total & 14 s : 490 GHz & 20 s : 390 GHz & & \\ +Phase 2 switch load & 6.1 GB/s & 2.7 GB/s & 1.5 GB/s & 1.0 GB/s \\ +Phase 4 switch load & 0.8 GB/s & 0.5 GB/s & 0.8 GB/s & 0.5 GB/s \\ +Phase 2 to Phase 4 switch load & 1.1 GB/s & 0.6 GB/s & 1.1 GB/s & 0.6 GB/s \\ +Summit to Phase 2 switch load & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s \\ +\hline +\end{tabular} +\end{center} +\end{table} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Switch I/O Requirements} + +The switch I/O requirements are defined by the total number of bytes +per second serviced by the two switches in the system. For the +analysis of the Switch I/O requirements, the choice of data +distribution again has a major impact. We again test the four +scenarios discussed above: Random Data Distribution, Random / Minimal, +Optimal Data Distribution, and Optimal / Minimal. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Random / Standard Data Scenario} + +In the Random Data Distribution scenario, each detector node needs to +read a total of 560 MB from the network and write a total of 200 MB +every 30 seconds. With 240 detector nodes, this corresponds to a +total bandwidth of 6080 MB/sec, or 49 Gb/sec. Note that this includes +the bandwidth needed to copy data from the summit and make two copies +on the detector machines, as well as the bandwidth to send the processed +image portions to the Phase 4 machines. The Phase 4 processing adds +an additional 320 MB of network I/O per Sky-Cell group, and there are +roughly 60-70 Sky-cells per exposure set. Thus the Phase 4 processing +adds an additional 750 MB/sec network bandwidth. In the architecture +defined in Figure \tbd{NN}, the Sky nodes and the detector nodes are each +attached to separate switches. An additional bandwidth requirement is +derived by the need to exchange data between these switches in for +Phase 4. The total amount of data exchanged between these switches is +480 MB per Sky-cell, for a total bandwidth of 1120 MB/sec. In +addition, the connection to the summit is a single, separate line +which needs to support the bandwidth requirement of copying all intial +raw images. In our simple model, each raw image is copied twice, +accounting for a total of 15360 MB every 30 seconds, or a bandwidth +load of 512 MB/sec. (Note that this last is double the actual +bandwidth requirement to the summit: a dedicated local circular buffer +would reduce the need for the second copy to come directly from the +summit.) + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Random / Minimal Data Scenario} + +In the Random / Minimal Scenario, the data volumes are significantly +reduced. The total Phase 2 bandwidth contribution is 332 MB over 30 +seconds for 240 nodes (2656 MB/sec) and the residual Phase 4 bandwidth +load is 224 MB per Sky cell over 30 seconds (522 MB/sec). The +inter-switch communication is now 240 MB per sky cell over 30 seconds, +or 560 MB/sec. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Optimal / Standard Data Scenario} + +In the Optimal Data Distribution, the Phase 2 network bandwidth is +reduced significantly to 184 MB per detector node, for a total of +1.5GB/sec, while the Phase 4 network bandwidth remains unchanged at +750 MB/sec. The inter-switch communication also remains the same at +1.12 GB/sec. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Optimal / Minimal Data Scenario} + +In the Optimal / Minimal Scenario, the total Phase 2 network bandwidth +drops to 124 MB per detector node, for a total of 1.0GB/sec, while the +Phase 4 network bandwidth is 552 MB/sec. The inter-switch +communication remains the same as the Random/Minimal Scenario at 560 +MB/sec. + +\begin{table}[t] +\begin{center} +\caption{\label{NP2} Phase 2 load per major frame (12000 GHz-sec)} +\begin{tabular}{lrrrr} +\hline +\hline +& Random/Standard +& Random/Minimal +& Optimal/Standard +& Optimal/Minimal \\ +\hline +network I/O (GB) & 182 & 80 & 44 & 30 \\ +PS-1 & & & & \\ + I/O (cpu-sec) & 3640 & 1600 & 880 & 600 \\ + CPU (cpu-sec) & 4000 & 4000 & 4000 & 4000 \\ + \# cpus & 64 & 47 & 41 & 38 \\ +PS-4 & & & & \\ + I/O (cpu-sec) & 1820 & 800 & 440 & 300 \\ + CPU (cpu-sec) & 2000 & 2000 & 2000 & 2000 \\ + \# cpus & 127 & 93 & 81 & 77 \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[b] +\begin{center} +\caption{\label{NP4} Phase 4 load per major frame (7800 GHz-sec)} +\begin{tabular}{lrr} +\hline +\hline +& Standard +& Minimal \\ +\hline +network I/O (GB) & 48 & 28 \\ +PS-1 & & \\ + I/O (cpu-sec) & 960 & 557 \\ + CPU (cpu-sec) & 2600 & 2600 \\ + \# cpus & 30 & 26 \\ +PS-4 & & \\ + I/O (cpu-sec) & 480 & 278 \\ + CPU (cpu-sec) & 1300 & 1300 \\ + \# cpus & 59 & 53 \\ +\hline +\end{tabular} +\end{center} +\end{table} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Conclusions} + +Table~\ref{throughput} presents one way of analysing the hardware +requirements, making a specific set of assumptions about the number of +nodes for the two phases and the expected network and disk +bandwidths. The important conclusion in this analysis is the implied +number of GHz per processor, given the assumptions laid out. +Phase 2 is specified to have 240 detector nodes, while Phase 4 is specified +to have roughly 60 static sky nodes. The range of Phase 2 CPU +requirements implies that each CPU needs to have speeds in the range +of 2.0 - 3.6 GHz, which sound very plausible for the year 2007, since +these apply to PS-4. + +Another way to represent this information is to use the total number +of MB I/O and the total number of GHz-sec required for the two stages, +confront these with an assumption for the bandwidth per network +adapter and an assumption for the CPU speed and use those numbers to +calculate the minimum number of nodes (CPUs) needed to sustain the +timing requirements. There are quite a few parameters and options to +choose from. We have assumed that for PS-1, the time between major +frames (4 images combined in Phase 4) is 120 seconds, and 30 seconds +for PS-4. We have also assumed that each CPU has one network adapter +associated with it, and use the numbers of 50 MB/sec for PS-1 era +network adapters and 100 MB/sec for the PS-4 network adapters (since +there has been some steady improvement in GigE hardware over the past +year). We have also assumed each PS-1 CPU is rated at 3 GHz and those +for PS-4 are rated at 6 GHz (somewhat conservative since 3 GHz +machines are already available). Tables~\ref{NP2} and \ref{NP4} show +the load and resulting number of nodes for both Phase 2 and Phase 4 +for both the PS-1 and PS-4 assumptions, using the I/O numbers for all +of the scenarios above. Note that in these discussions, we make the +idealized assumption that the computational and I/O portions of each +process are completely serial. As a result, the CPU is completely +used to perform the I/O during the I/O phase, avoiding any concern +about I/O load on the processor during analysis. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Notes} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Cell vs Chip vs FPA vs Major Frame} + +There are several levels of input data pixel groups: Cell, Chip, Focal +Plane Array (FPA), and Major Frame. It is necessary to make the +association between the data of one level and that of the next in a +way that is reliable and robust to missing elements. If a specific +cell is missing from a chip, that information is known by the +controller an needs to be represented in the metadata. Similarly if a +chip is missing from a mosaic camera, that information is also known +and must be carried though the metadata. A more difficult association +is that between the telescopes to define the major frame. Some +possibilities: + +\begin{enumerate} +\item exposures in a major frame are always synchronized; the +telescopes are required to take exposures in a coordinated fashion and +these linked exposures are identified as being part of a specific +major frame by the TCS or PTS. +\item exposures may be taken in a coordinated fashion, and identified +by the TCS or PTS as part of a specific major frame, but not all +exposures are required to be taken in this fashion. Independent +images are handled by the IPP differently (Phase 3 and Phase 4 are not +appropriate, some varient is required). +\item exposure links are defined more generally on the basis of the +resulting image metadata. The telescopes may have images requested +at the same coordinates and time, and are defined as a major frame on +the basis of the observed time and coordinates. The TCS or PTS might +not be the entity which defines these major-frame associations; this +may be the role of some component in the IPP. Different types of +major frames may be defined depending on the correlation period in +time or space. For example, a major frame in which the telescopes are +pointing at the same position in the sky to within a few pixels and +with exposures taken within a second can be treated with more special +assumptions (minimal differential distortion; moving objects +coincident) than a major frame in which the offsets are larger in +either dimension. +\end{enumerate} + +A decisions between these possibilities will drive some requirements +either on the IPP side or on the PTS/TCS side. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Identifying ghosts, spikes, etc} + +One of the functions currenly defined for Phase 1 is the prediction of +the location of the bright star spikes, ghost images, and regions of +complex astronomical background. Elsewhere in the IPP, these +identifications are used to excise or mark image pixels. How these +regions are defined and saved are is not very clear. I propose that +we use the mask image to mark as bit-flags all of these cosmetic pixel +flagging issues. If we need to save this information, for the short +period that the input science images are kept, then it is only a small +addition of data. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Pending Sky-cell / Detector table} + +Define a pending sky-cell / detector table to define the overlaps and to +give something which the scheduler can query to decide when to +initiate phase 4. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Appendices} + + +\bibliographystyle{plain} +\bibliography{panstarrs} +\end{document} + +%%%%%% Phase 0 has been dropped: identifying the moving objects is not needed + +\paragraph{Phase 0 : night preparation} + +Phase 0 is the night preparation phase of the IPP analysis system. +There may be potentially many pieces of information which apply to the +processing for an entire night and which take substantial time to +calculate. these are pre-calculated by the phase 0 stage and stored +in a database table for reference by other stages of the processing +system. Currently, the only quantity calculated by Phase 0 is the +collection of known moving object ephemerids. + +At various stages in the IPP analysis, it is necessary to know the +location of known moving objects (main belt asteroids, comets, +Kuiper-belt objects, any other classes of asteroids) in relation to +specific images obtained. If moving object orbits were trivial to +calculate, or if the number was limited, this would be a simple +problem of three dimensional intersections. However, complete orbits +are not trivial and there may be tens of thousands to millions of +possible objects of interest. To simplify the task, it is possible to +reduce the parameter space of the search by pre-calculating the orbit +segments of all objects for a given night and saving fiducial points +of the orbit in a database table. Later systems which require the +position of objects in a specific image can use linear interpolation +between these fiducial points to identify the likely objects, and +potentially additional non-linear orbital calculations to refine the +positions. + +The database table of object fiducial positions must include the +following information: + +\begin{itemize} +\item object ID +\item epoch +\item RA at epoch +\item DEC at epoch +\item dRA at epoch +\item dDEC at epoch +\item R magnitude? +\item date of calculation? +\item lifetime? +\end{itemize} + +The input for this calculation is the table of known moving objects +and their orbital elements, and the time range for the calculation. +If the calculation is slow, Phase 0 could be paralellized by object. +If Phase 0 is fast enough (\tbd{minutes?}), the process need not be +parallel. The {\tt lifetime} and {\tt date of calculation} allow old +Phase 0 entries to be removed when they are not needed. \tbd{This +cleaning phase could be a function of Phase 0.} Phase 0 need not be +run only for the current night. Any time a specific set of data is to +be analysed by the later stages, phase 0 should be run for the +appropriate time period. \tbd{Does there need to be a database table +with phase 0 runs and time periods defined? this could be the +reference used by later phases to decide if phase 0 has been run. they +could also trigger the phase 0 run if they notice it has not been run +(a job of the scheduler).} + +\tbd{what is the orbit calculation speed? does it scale with Npts? +what is the number of known objects now? in 5 years?} + + + +%%% phase 2 metadata +\milsection{Metadata} + +The following metadata associated with the images are required for +Phase~2 operation: +\begin{itemize} +\item The orthogonal transfer (OT) image shifts made during the +exposure --- in order to create a convolution kernel; +\item Time of observation --- for selecting the appropriate detrend +images; +\item Filter --- for selecting the appropriate detrend images; +\item Telescope identification --- for selecting the appropriate +detrend images; +\item Exposure time --- for the photometric calibration; +\item Detector gain --- for calculating photometric errors and +determining the quality of the overscan; +\item Detector read noise --- for calculating photometric errors and +determining the quality of the overscan; +\end{itemize} + +\milsection{Pixel Masks} +\label{ap:masks} + +This section describes the requirements on Bad Pixel Masks (BPMs). +These will consist in of bit masks for each pixel. For Phase 2, flags +are required for at least each of the following pixel attributes: +\begin{enumerate} +\item The pixel is a charge trap; +\item The pixel is a bad column; +\item The pixel is saturated in the A/D converter; +\item The pixel is non-positive in the flat-field; +\item The pixel is part of a row that has excess noise; and +\item The pixel is determined to be a cosmic ray, based on its +morphology. +\end{enumerate} + +Of these, only masks for the charge traps need to be grown by the +extent of the OT convolution kernel. For other pixel types, +orthogonal transfer of the flux in this pixel will not (necessarily) +affect the flux in neighbouring pixels + +\milsection{Object Catalogs} +\label{ap:catalogs} + +Object catalogs from Phase 2 shall consist of at least the +following elements for each object: +\begin{enumerate} +\item Object centre, with corresponding errors; +\item Object magnitude, with corresponding error; +\item Object isophotal magnitude, with corresponding error; +\item Object FWHM; +\item Object elliptical axis lengths; and +\item Object position angle for ellipse. +\end{enumerate} + +Though further details may be required for catalogs in Phase~4, +the above details are minimum requirements for Phase~2 catalogs. + Index: /trunk/doc/design/ippSRS.tex =================================================================== --- /trunk/doc/design/ippSRS.tex (revision 771) +++ /trunk/doc/design/ippSRS.tex (revision 771) @@ -0,0 +1,2140 @@ +%%% $Id: ippSRS.tex,v 1.1 2004-05-25 00:38:56 eugene Exp $ +\documentclass[panstarrs]{panstarrs} + +% basic document variables +\title{Pan-STARRS Image Processing Pipeline} +\subtitle{Software Requirements Specification} +\shorttitle{IPP SRS} +\author{Eugene Magnier, Paul A. Price, Josh Hoblitt} +\group{Pan-STARRS Algorithm Group} +\project{Pan-STARRS Image Processing Pipeline} +\organization{Institute for Astronomy} +\version{DR} +\docnumber{PSDC-430-005} + +% allow paragraphs to be listed in TOC for now +\setcounter{tocdepth}{4} + +\begin{document} +\maketitle + +% -- Revision History -- +\RevisionsStart +% version Date Description +DR.01 & 2003.01.01 & First draft \\ \hline +DR.02 & 2003.03.10 & Second draft \\ \hline +DR.03 & 2003.04.13 & Most paragraphs fleshed out \\ \hline +\RevisionsEnd + +\listoffigures +\pagebreak + +\tableofcontents +\pagebreak +\pagenumbering{arabic} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Scope} + +\subsection{Identification} + +This document establishes the system requirements for the Pan-STARRS +Image Processing Pipeline (IPP) as applied to Pan-STARRS 1 (PS-1), the +initial demonstration telescope to be constructed on Haleakala by Jan +2006. + +\subsection{System Overview} + +\tbd{description of the Pan-STARRS System and PS-1.} + +\subsection{Document Overview} + +The Pan-STARRS document naming scheme is PSDC-NNN-MMM-VV, where the VV +entry specifies the document version number. Where documents are +identified without the version number, the latest official version in +that series is implied. + +Open Issues and TBDs in this document are marked \tbd{in bold, red +with surrounding square brackets}. + +All timing measurements are to execution time as measured on a +\tbd{Reference Pan-Starrs Computation Node} and assumed to be not +limited by network bandwidth. + +\subsubsection{Definitions} + +\paragraph{``Must''} When used in this specification, the word +``must'' refers to an explicit requirement of a system component or +the complete system. + +\paragraph{``Should''} When used in this specification, the word +``should'' refers to a desired chracteristic of a system component or +the complete system. + +\paragraph{``Will''} When used in this specification, the word +``will'' provides information about a characteristic of a related +system component or a complete related system. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\DocumentsInternalSection +PSDC-430-xxx & PS-1 Design Reference Mission \\ \hline +PSDC-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline +PSDC-430-006 & Pan-STARRS IPP ADD \\ \hline +PSDC-430-007 & Pan-STARRS IPP PSLib SDR \\ \hline +\DocumentsExternalSection +Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ +\DocumentsEnd + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Requirements} + +\subsection{Required States} + +The IPP must have 3 states: active, paused, and interactive. + +\subsubsection{Active State} +\label{req:active-state} + +In active state, the IPP must accept images and metadata from OATS and +automatically perform the complete set of image processing tasks, +including both calibration and science image processing. The IPP must +respond to requests for data from the client science pipelines +\tbd{and IPP monitoring team}. + +\subsubsection{Paused State} +\label{req:paused-state} + +In paused state, the IPP must refuse data and metadata from OATS and +data requests from the client science pipelines. + +\subsubsection{Interactive State} +\label{req:interactive-state} + +In interactive state, the IPP must accept data and metadata from OATS, +but must not automatically process the data. The IPP must respond to +user commands to initiate portions of the data analysis. + +\subsection{System Capability Requirements} +\label{req:system-capabilities} + +The IPP must perform the following tasks: + +\begin{enumerate} + +\item Accept raw images from OATS at a sustained rate of 1 exposure + per 30 seconds. + +\item Accept metadata from OATS at a sustained rate of \tbd{XXX MB / sec}. + +\item Produce high-quality calibration images from the raw calibration + images. The calibration images must not introduce systematic + uncertainties greater than \tbd{0.2\%}. \tbd{Requirements on the + speed of processing the calibration images.} + +\item Pre-process the science images with the high-quality calibration + images. + +\item Merge multiple pre-processed science images -- from multiple + telescopes or from sequential, dithered exposures -- into single, + cleaned, stacked images. + +\item Subtract a static sky image from the cleaned, stacked images to + produce an image of only the transient events. + +\item Excise the significant transients and outliers from the + pre-processed science images and merge the cleaned images into the + static sky image. + +\item Detect objects on the four types of images: pre-processed + images, the stacked image, the difference image, and the static sky + image. + +\item Determine astrometry of the detected objects relative to an + astrometric reference to an accuracy of \tbd{30 mas}. + +\item Determine photometry of the detected objects relative to a + photometric reference to an accuracy of \tbd{5 millimag} relative + photometry and \tbd{10 millimag} absolute photometry in photometric + weather. + +\item Produce a high-quality astrometric reference catalog from the + extracted objects on a time-scale of 6 months. The astrometric + reference must have an absolute accuracy of \tbd{30 mas} and a local + relative accuracy of \tbd{10 mas}. Proper motions of all nearly + stationary objects must be determined with an accuracy of \tbd{XXX + mas / year}. + +\item Produce a high-quality photometric reference catalog from the + extracted objects on a time-scale of 6 months. The photometric + reference must have an consistency across the sky of \tbd{5 + millimag} and an absolute calibration to the external system defined + by \tbd{SDSS} of \tbd{10 millimag}. + +\item Publish the static sky images to the Pan-STARRS published static + sky server on a time-scale of \tbd{1 month}. + +\item Publish the detected objects to the Pan-STARRS published object + database on a time-scale of \tbd{1 week}. + +\item Provide access to external Pan-STARRS clients to the detected + objects on time-scales of \tbd{1 minute} after the image is + processed. + +\end{enumerate} + +\subsubsection{Software Coding Requirements} + +\paragraph{Languages} +\label{req:languages} + +Source code must be in C. All source code must be compiled with `gcc' +version v2.95 or higher. + +Scripting language must be \tbd{Python, version TBD}. + +\paragraph{Interfaces} +\label{req:interfaces} + +Access to low-level Library functions must be provided via C APIs +consisting of the function calls and the defined data structures and +other data types. Access to high-level functions must be provided +via C APIs as well as SWIG interfaces, where specified. Access to +processing jobs must be available via the UNIX shell. + +\paragraph{Coding Standards} + +The C code must comply with ANSI Standard C99. Because the delivered +code is required to run on UNIX machines, the delivered code must be +in compliance with the language-independent UNIX operating system +standard POSIX (Open Group Based Specifications Issue 6, IEEE Std +1003.1, 2003). Source code files must use the UNIX line-break +convention (line-feed only). C coding style must adhere to the +standard defined in the document 'Pan-STARRS C-coding standard' +(PSDC-430-004). \tbd{Python coding must follow the Python standard +defined in the document TBD}. + +\paragraph{Naming Conventions} + +Header files must have names starting \code{ps} or \code{p_ps} for +private interface definitions. The latter must appear in a +subdirectory \code{private} of whichever directory is being searched +for the public header files. + +Functions visible at global scope which are part of the public API +must have names begining with \code{ps}, and follow the naming +conventions in the coding standard. Functions that are visible at +global scope but which are not part of the public interface must have +names begining with \code{p_ps}. Functions that are local to a file +must \textit{not} start \code{ps} (or \code{p_ps}). + +Variables visible at global scope which are part of the public API +must have names begining with \code{ps}, and follow the naming +conventions in the coding standard. Variables that are visible at +global scope but which are not part of the public interface must have +names begining with \code{p_ps}. Variables that are local to a file +must \textit{not} start \code{ps} (or \code{p_ps}). + +The names of all enumerated types and C-preprocessor symbols (but not +variables declared \code{const}) must start with \code{PS_}, in the +case of public symbols, or \code{P_PS_}, for private symbols. The +rest of the name must be uppercase with words separated by underscores +(\code{_}). An exception is the case of system utilities implemented +as macros, in which case the names must conform to the convention for +function names. + +When defining a function to convert from one type to another, the name +must be of the form \code{psOldToAlloc}, e.g.\hfil\break +\code{psEquatorialToEcliptic} (\emph{not} +\code{psEquatorial2Ecliptic}). + +\paragraph{C Programming Guidelines} + +Functions that assign to a variable must list that argument +\textit{first}, following the pattern of \code{strcpy}; e.g. +\begin{verbatim} +void psAddToVector(restrict psVec *outVec, const restrict psVec *inVec, + int val); +\end{verbatim} + +Type definitions should always be accompanied by prototypes for their +constructors and destructors, following these guidelines: + +\begin{itemize} +\item The constructor name should consist of the type name followed by +\code{Alloc}; e.g. a type \code{psImage} would be created by a +function +\begin{verbatim} +psImage *psImageAlloc(int nrow, int ncol); +\end{verbatim} + +\item The type should be freed with a destructor named \code{typeFree}, e.g. +\begin{verbatim} +void psImageFree(psImage *img); +\end{verbatim} + +\item The constructor must never return \code{NULL}, and no code calling the +constructor should ever check the return value. + +\item The destructor must not return a value. + +\item The destructor must handle being passed \code{NULL} by simply +returning immediately. This must not be treated as an error +condition. + +\item Constructors and Destructors should use the memory reference + counter facilities of the PSLib memory management system. + +\end{itemize} + +\paragraph{Commenting and Documentation} + +Commenting of delivered C and Python code must follow the C and +Python coding standards and must provide tags for Doxygen +interpretation of the comments and program structures. + +Documentation for the IPP consists of source code documentation and +user documentation. Source code documentation must be generated with +Doxygen from the in-line comments and must be provided as HTML, +Latex, and man pages. User documentation includes the API usage for +the modules and library functions as well as user interface +description for the higher-level architectural systems. User +documentation must be delivered as PDF documents. + +\paragraph{Version Control} + +Source code version control must be implemented with CVS. + +\paragraph{CSCI Deliverable} + +All final source code generated for the IPP is to be delivered via +CVS, including the test code. CVS revision history must be included +and made available via CVS. + +\paragraph{Platform architectures and operating systems} + +Makefiles must be provided with appropriate flags set so that all +code compiles without warnings under 'gcc -Wall' for the following +platform architectures and operating systems: + +\begin{itemize} +\item x86/Linux +\item PPC/OS-X +\end{itemize} + +The requirement of compiling without warnings includes the allowance +that the output may be filtered to exclude known, specified warnings, +such as those caused by lex-generated code. + +Although the code must compile successfully under both listed +operating systems, unit testing should only be performed for the +x86/Linux combination. + +\paragraph{Software Configuration} + +\tbd{deferred} + +\subsubsection{Architectural Components} + +In order to achieve the required functionality, it is necessary to +divide the IPP into a number of clearly-defined software elements, +listed as follows: + +\begin{enumerate} + +\item {\bf Pixel Server:} This component is a large data store for all + images used by the IPP, including the raw images from the telescope, + the master calibration images, the reference static-sky images, and + any temporary image data products produced by the IPP. The Pixel + Server is required to meet all of the image storage needs identified + in the top-level requirements above. The Pixel Server must accept + the incoming data and store it until it is no longer needed by other + portions of the IPP. + +\item {\bf Photometry \& Astrometry Database (PnA):} This component is + required to store and manipulate astronomical objects detected in + various images, as identified above, including individual + measurements of objects on the images, the summary information about + those objects, and reference object data. + +\item {\bf Metadata Database:} This component is required to store the + data which is not directly related to images or astronomical objects + as needed to perform the analysis specified above. + +\item {\bf Analysis Stages:} Specific programs are required to perform + the processing steps listed above. These can be divided into + well-defined analysis stages, each of which operates on a particular + unit of data, such as a single OTA image or a collection of + astronomical objets. + +\item {\bf Controller:} In order to perform the analysis stages + required by the IPP, it is necessary to use distributed computing + processes on a large number of computers. The Controller is + required to manage the collection of analysis stages performed on + these machines. + +\item {\bf Scheduler:} This component is a decision-making mechanism + required to guide the operation of the IPP: to evaluate the + currently available collection of data, to identify the necessary + analysis, and to assign the analysis tasks to the Controller. + +\end{enumerate} + +The relationship between these software elements is shown in +Figure~\ref{overview}. This figure also shows the interactions +between the IPP and other Pan-STARRS systems. + +\begin{figure} +\begin{center} +\resizebox{8cm}{!}{\includegraphics{pics/overview.ps}} +\caption{ \label{overview} IPP System Overview} +\end{center} +\end{figure} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\paragraph{Pixel Server} + +The IPP Pixel Server \tbd{rename as Image Server?} is a large data +store for all images used by the IPP. The Pixel Server is required to +store all of the images needed by the IPP for the length of time they +are required; total data volume is specified in detail in the hardware +summary, but is in the vicinity of \tbd{700 TB}. + +The IPP Pixel Server must maintain a record of all images currently +available in the repository \tbd{and all no longer available}. This +record must include the image name, location (which machine), the +state of the image (available, deleted), the image size, the image +type, and the existence and location of secondary copies of the image. +This information need not include other metadata such as the image +summary statistics or the state of the image processing for the image, +as these aspects are included in the Metadata DB. + +The IPP Pixel Server must store images as FITS files on disk. Raw +images from the telescope must be stored as individual OTA images for +each file, with multiple Cell images per file as well as video +sequences from the guide stars. Images of the Static Sky must be +stored in the form of \tbd{triangular segments} to minimize the total +data volume and pixel overlap. + +The IPP Pixel Server must distribute images across a cluster of +machines. The IPP Pixel Server must be capable of honoring requests +to store an image on a specific machine. If such a request cannot be +honored, the IPP Pixel Server must select an appropriate machine and +notify the requesting agent of the new locations. The IPP Pixel +Server must provide a mechanism to maintain multiple (at least two) +copies of each image. + +The IPP Pixel Server must interface with other subsystems of the IPP. +It must provide an interface to other IPP subsystems to identify the +image location (the computer on which it resides). It must provide a +mechanism to serve a specified image to another IPP or Pan-STARRS +subsystem. It must provide a mechanism for deletion of images in the +Pixel Server. It must have a mechanism to accept or retrieve an image +from another Pan-STARRS subsystem, in particular OATS. Communication +of messages between the IPP Pixel Server and other subsystem must be +via \tbd{XML messages} passed via \tbd{some transport}. + +The IPP Pixel Server must accept images at the telescope maximum rate +of 1 full-camera image every 30 seconds. The IPP Pixel Server must +therefore accept notifications and process retrievals at a rate of 64 +raw OTAs in 30 seconds. + +\tbd{O/S, language, SQL, ODBC requirements?} + +\tbd{hardware requirements?} + +\tbd{communication protocols?} + +\paragraph{P\&A Database} + +The IPP requires a mechanism to store data related to astronomical +objects derived from various sources with a variety of associations. +The PnA (Photometry and Astrometry) Database serves this function. +The PnA Database deals with two related concepts: {\em objects} and +{\em detections}. The objects are descriptions of astronomical +objects while the detections are the specific measurements of those +objects on an image. A collection of {\em detections} may be used to +derive average quantities which describe a particular {\em object}. + +The PnA Database must store the collections of detections which were +derived from specific images from any of the analysis stages. It must +be possible to determine and locate (perhaps via interactions with the +pixel server) the image from which a specific detection was derived. +It must also be possible to extract all detections derived from a +specific image. These associations must include descriptive +information including the coordinates of the detection on the image. + +The PnA Database must provide a mechanism to associate together +multiple detections of a specific object. Several major classes of +objects will be present, each of which must be handled correctly. + +First, the most distant stars, compact galaxies, and QSOs will have +nearly fixed locations relative to other nearby stars, with only small +deviations for individual measurements. The association between +multiple detections of such objects must be made on the basis of their +coincident positions. The PnA Database must be able to determine the +average position of the object and the deviations of the individual +detections from that average. + +Second, solar system objects do not have a fixed location and +detections of such objects must associated on the basis of their +coincidence with the orbit of the objects. The PnA Database must be +able to associate detections with the orbits of known objects. The +determination of this association is the responsibility of the MOPS +and must be communicated to the IPP PnA Database on \tbd{some +timescale}. The PnD Database must be able to retrieve the detections +associated with the object and to provide the object associated with +the specific detections. This association must include descriptive +information such as the offset of the detection from the predicted +location of the detection based on the orbit. This functionality is +required to allow the PnA Database to ignore known moving object +detections from other types of queries. + +Third, stars in the general vicinity of the solar system fall in +between these first two classes of objects. Their proper motion and +parallax response is significant enough ($>1$ arcsec in 10 years) that +they are not well-described by an average location and a collection of +offsets. These objects must be described by a distance and a proper +motion vector. The PnA Database must be able to find and associate +detections of objects for which either of the parallax or the proper +motion are substantial. + +Fourth, many detections, especially in their initial states, will not +be associated with a specific astronomical object of any of the above +classes and should be treated as orphans. Some of these will be +suprious (not represent real objects), some will be from solar system +objects for which orbits are not yet determined, some will be from +faint stars near the detection limits, some will be from short-term +transients which have only been detected once. The PnA Database must +be able to carry these detections until they have been associated with +one of the objects above. It must be possible to migrate individual +detections associated with an astronomical object back to the orphan +state. + +For every object, and all orphaned detections, it must be possible to +determine the images for which the coordinates were included but for +which no detection was made. The minimum set of information which +must be carried for these non-detections is the image and the +associated object or orphan. + +The PnA Database must store the relationships between various +photometric systems and, in some cases, the evolution of that +relationship. It must be possible, given a determined set of +calibrations, to convert between the measured instrumental magnitude +of a detection with a specific filter, detector, and telescope, and at +particular time and the implied magnitude in the average Pan-STARRS +magnitude systems. It must also be possible, given the magnitudes of +an object in one system to convert those to the magnitudes in another +system; an example of such a conversion is between the average +Pan-STARRS filter systems and the various reference systems +appropriate for those filters. + +The PnA Database must provide interfaces to extract lists of objects +and detections based on various query parameters. It must be possible +to extract all detections associated with a specific object, all +non-detections of that object (or orphan) and summary statistics from +these collections. It must be possible to extract all objects or +detections within specified spatial regions including regions bounded +by great circles (RA,DEC; GLAT,GLON; ELAT,ELON) and regions described +by a location and a search radius. It must be possible to extract the +image parameters associated with a specific detection including image +coordinates of the detection, exposure time, time and date of the +detection, etc. + +\tbd{volume requirements} + +\tbd{speed / access requirements} + +\paragraph{Metadata Database} + +The IPP requires a Metadata Database to store and provide access to +metadata of various types and from various sources. Metadata in the +context of the IPP represents all data which is not included in the +two data stores discussed above (Images and Detection/Objects). +Metadata is generated at the telescope and during the various analysis +stages + +The Metadata Database must store and provide metadata for all raw +images, for processed images, for the calibration images (both raw and +master), for the extracted object lists. Metadata describing the +environmental conditions at the telescope must also be stored and +provided as needed. + +If analysis results are exchanged via the metadata database, it must +provide access to the queried data on timescales of $<2$ seconds to +avoid slowing down the analysis systems. + +\tbd{volume requirements} + +\tbd{does the description of images belong in the Metadata database or + in the Pixel / Image Server?} + +\tbd{queries} + +\subparagraph{Configuration Database -- a subset of the metadata database?} + +The IPP requires a Configuration Database to store and provide access to +information about the IPP itself. Examples of data in the +configuration database include the default parameters for the various +analysis programs, the description of the computing environment, the +process status information, etc. + +\paragraph{Controller} + +The IPP uses a collection of computers to store and process images and +to manipulate collections of detections. These computers perform any +of a large number of analysis stages or other processing tasks without +significant interprocess communication. It is necessary to have a +mechanism which initiates computing tasks on the different computers, +which monitors the tasks as they are executed, which handles the +output and the errors from these tasks, and which reacts to the +failure of any of the computing nodes. The system responsible for the +tasks in the IPP is the Controller. + +The Controller must interact with the collection of computers under +its management and with other subsystems in the IPP. The controller +must accept a variety of inputs from other subsystems, described +below, and respond accordingly. The controller must also provide +information to other subsystems on demand. + +Computers managed by the controller are allowed to be in one of +several states, and the controller must interact with it in an +appropriate way for each of those states. A computer may be {\tt +alive}, {\tt dead} or {\tt off}. If the computer is {\tt alive}, it +responds to commands from the controller and may be used for tasks +subject to other constraints. If it is {\tt dead}, the computer is +not responsive and must not be used for executing tasks. The +controller must identify computers which have died and occasionally +test them to see if they are {\tt alive} again. Computers which are +{\tt off} are not available for tests and must not be tested. +Computers may be set to the {\tt off} or {\tt dead} states by external +subsystems; it is the responsibility of the Controller to return a +computer to the {\tt alive} state if possible. + +Computers which are in the {\tt alive} state may be in one of two +modes: {\tt busy} and {\tt free}. A computer which is {\tt busy} +currently has a task assigned to it. The controller may only assign +one task to one computer at a time\footnote{A physical piece of +hardware may be defined to the Controller as multiple computers to +allow multi-processor nodes to execute more than one simultaneous +task.}. Computers which are in the {\tt free} state may have tasks +assigned to it. The controller must also manage an additional set of +constraint tables for each machine: the allowed tasks. Each computer +may have a list of allowed tasks which may include {\tt all} tasks, +{\tt none} of the tasks, or specified task names. The controller must +only execute the allowed tasks on a machine. + +The Controller must accept tasks from other IPP subsystems. The task +requests must include the specific command to be executed. The +commands must be in the form of a UNIX command which could be +performed on any of the computing nodes. Any input or output data +structures in the commands must be a valid resource regardless of the +node on which the task is executed. Input and output data resources +must be unique where necessary to avoid conflicts. Tasks must be +given an identifier, which must be returned to the requesting agent, +to be used to control the specific task. + +Task requests may specify a desired node for the task execution. The +Controller must attempt to honor the request if the node is {\tt +alive}, but must execute on another node if the requested one is {\tt +dead} or {\tt off}. Even if a node is {\tt alive} the controller must +choose another node if the specified tasks is not allowed on the +requested node. In all other cases, the controller must wait until +executing processes, and processes with higher priority, are completed +before executing the specified task on the requested node. + +Task requests may specify an urgency level. The controller determines +the priority of the task by sorting first by priority and next by the +sequence of the request. An executing task must be completed before +any new task is started, regardless of priority. Tasks may be +assigned a priority of 0 in which case they are maintained in the +queue and never executed. + +The controller must monitor the output streams from the executing +tasks and the exit status of the tasks. \tbd{where do we send the +output logs?}. The status, including the exit status, of each task +must be maintained for other subsystems to query as needed. \tbd{how +long? on disk / database?} + +The controller must accept commands from other IPP subsystems. These +commands include those which govern the processing of specified tasks, +those which govern the behavior of specific computing nodes, and those +which request information from the controller. The controller must be +able to halt the execution of a specified task, delete an unexecuted +task from the task list, change the priority of tasks, change the +requested nodes for tasks. The controller must also be able to stop +the current execution of a task and push it to the end of the queue +and also change its priority. + +The controller must honor requests (normally from the users) to change +the mode of any computing node on demand between {\tt off} and {\tt +dead}. It must also be able to change the list of allowed tasks as +requested by external commands. + +The controller must respond to informational requests regarding the +collection of machines and their states as well as the collection of +tasks and their states. The controller must monitor the execution +times of the different tasks and provide summary statistics. Finally, +the controller must respond to three top-level commands: {\tt finish}, +{\tt stop} and {\tt abort}. When {\tt finish} is requested, no more +new tasks are accepted on the stack of task, and when all tasks in the +stack have completed, the controller must exit. When {\tt stop} is +requested, the currently executing tasks must be completed at which +point the controller must exit, but tasks remaining in the stack which +have not been started are flushed. When {\tt abort} is issued, the +controller immediately kills all executing tasks and exits. + +\paragraph{Scheduler} + +The IPP is responsible for a variety of analysis tasks: several stages +of processing of the science images; routine assessment of the detrend +images used in processing the science images; construction of +replacement detrend images when needed; generation of astrometric and +photometric reference catalogs based on the collected dataset; and the +performance of test analysis programs. At any point, decisions need +to be made about which of these tasks should be performed, based on an +analysis of the contents of the image database tables, the +requirements of the people monitoring the IPP, and the near-term +observing plans. The IPP Scheduler is a mechanism to manage these +various inputs to guide the decisions and initiate the actions. + +The Scheduler acts as an intermediate between several components of +the IPP and also between the IPP and external agents such as the OATS +system and the users who must monitor the behavior of the IPP. + +The Scheduler must send commands to the Controller for execution. It +is the Controller's responsibility to manage the specific analysis +jobs executing on a given processing node. These analyses may include +the process of copying of moving data from OATS to the pixel server +nodes, or it may involve image processing stages performed on the +science images by the appropriate processing nodes, or it may involve +analysis of the data in the PnA object database. In order to isolate +and encapsulate the responsibilities of the Scheduler and the +Controller, the Scheduler must initiate the tasks which the controller +manages; in this way, the controller does not need to have any +information about the details of the tasks which it executes. +Communication between the Scheduler and the Controller must be +bi-directional; the Scheduler must send tasks to the Controller which +the Controller must inform the Scheduler of the outcome of those +tasks. \tbd{it is not specified whether the scheduler and controller +are components of a single software system or interacting but distinct +software components.} + +The Scheduler must take as input the current list of pending images, +both science and calibration, and a description of the current +observing plan or strategy on some time-scale. The Scheduler must +also take input from humans who manage the IPP. + +The Scheduler must choose between several types of analysis stages +based on the contents of those lists and on the requirements of the +users. The list of tasks which the Scheduler must decide between +includes: +\begin{itemize} +\item moving data to and from the pixel server ($\sim 30$ second timescales) +\item running the science analysis stages ($\sim 30$ second timescales) +\item testing the validity of the current detrend images ($\sim$ + nightly) +\item constructing new detrend images ($\sim$ weekly) +\item updating and improving the photometric and astrometric reference + catalogs ($\sim$ yearly). +\end{itemize} + +The Scheduler must choose between tasks which are relevant on several +different time-scales. The time-scale range from 2 times per minute +to once or twice a year, as noted in the list above. The Scheduler +must also make use of the human input in managing such choices. The +human users must be able to specify that a particular task or set of +tasks is of higher or lower priority than the norm. + +The Scheduler must maintain a set of rules defining the dependency of +one type of analysis stage on other analysis products. For example, +the nightly science image processing depends on the existence of valid +detrend images. The Scheduler must be able to recognize the +dependency and initiate the required analysis needed to perform other +analysis tasks. The Scheduler must have the ability to decide between +postponing an analysis task until the required data are available or +to initiate the task using a lower-quality or less appropriate +substitute. For example, in normal circumstances, a science image +must not be processed until the corresponding detrend frame has been +produced. However, if such a frame is unlikely to appear soon, and +the pressure to process the science image is sufficiently high, then +the frame could be processed with an older detrend frame of known +lower quality. The Scheduler must have the ability to choose the +best, if not ideal, reference data for a particular circumstance. + +The Scheduler is responsible for setting the operating mode of the +IPP. When the IPP is in the automatic operating mode, this implies +that the Scheduler is performing the most appropriate tasks at a +particular time. When the IPP is in the interactive mode, the +Scheduler must perform the requested action regardless of the outcome +of the decision trees. In addition, the Scheduler must only perform +the requested actions and not attempt to perform the other +normally-required actions. The only exception to this exclusion is +that, in the interactive mode, data must still be copied from the +summit system. A human-sent command must be able to change the +Scheduler priorities from the automatic to the interactive modes +\tbd{with a CLI or GUI}. An additional IPP mode is the {\em paused +mode}, intended for tests or maintenance, in which case the Scheduler +does not perform even the data copy tasks. Every task is performed on +demand by the user. + +\subsubsection{Analysis Stages} + +\paragraph{Overview} + +We now consider the collection of analysis tasks which must be +performed by the IPP. These tasks represent the core of the required +IPP functionality; the architectural components discussed above can be +viewed as primarily supporting infrastructure to enable the analysis +tasks to be executed on the appropriate data and to store the results. + +Depending on the task, the basic data unit may be individual images, +collections of images, or derived data products such as a collection of +detections of astronomical objects. Because of the granularity of +these data units, many of the analysis tasks can be performed in +parallel because, for example, the intial analysis of an OTA in one +image does not depend on the results from another OTA. We define the +term `analysis stage' to refer to the largest complete analysis task +which may be performed on a single data item. The analysis stages are +divided into three categories, and further subdivided as follows: + +\begin{enumerate} + \item {\bf Science Image Analysis} is performed on the night-sky + science images to extract the science data from these images. The + science image analysis is divided into 4 phases: + + \begin{itemize} + \item {\bf Phase 1:} The image processing preparation phase, in + which basic astrometric analysis of the complete FPA image is + performed. + + \item {\bf Phase 2:} The image reduction phase, in which the + individual detector images (OTAs) are processed as much as possible + without reference to other chips in the same FPA image or other + exposures. + + \item {\bf Phase 3:} The exposure analysis phase, in which the + results of the multiple detectors are combined to improve the + calibrations for the complete FPA images. + + \item {\bf Phase 4:} The image combination phase, in which several + different exposures of the same part of the sky are combined to + produce high-quality difference and summed images. + \end{itemize} + + \item {\bf Calibration Image Analysis} is required to generate the + calibration images used in the science image analysis. There are + three types of calibration images which are produced. \tbd{make this + consistent with other sections which use the basic / other + calibration distinction} + + \begin{enumerate} + \item {\bf Calibration 1:} The basic master-detrend creation images, + which are constructed from a simple stack of multiple input + calibration images. + + \item {\bf Calibration 2:} Sky-model \& fringe-model images, which + are constructed by combining a collection of images which require + substantial processing before the combination. + + \item {\bf Calibration 3:} Flat-field correction image, which is + constructed on the basis of photometry observations of objects from + certain science images. + + \end{enumerate} + + \item {\bf Reference Catalog Creation} is required by the IPP to + generate improved astrometric and photometric reference catalogs on + the basis of Pan-STARRS observations. + +\end{enumerate} + +Figure~\ref{stages} shows the flow of data between the various IPP +software systems and the different analysis stages, each managed by +the Controller. The thick lines represent the flow of pixel data, the +thin lines represent the flow of metadata and object data, and the +grey lines represent the flow of commands. The hatched systems +represent external PanSTARRS systems (OATS, the Sky Server, the SAIC +Object Database, the Moving/Transient Object Pipeline, and other +Client Science Pipelines. + +The individual analysis stages can be accessed as a UNIX command-line +program. Each command represents the action of the stage on a single +quantum of data. These analysis stages are built of lower-level +C-functions wrapped in a higher-level programming language, +\tbd{Python}. + +The decision to execute a specific analysis stage for a specific +dataset is made by the Scheduler, which sends the infomation to the +Controller. The Controller executes the analysis stage for the data +on an appropriate machine and monitors the success or failure of the +job. + +\begin{figure} +\begin{center} +\resizebox{8cm}{!}{\includegraphics{pics/stages.ps}} +\caption{ \label{stages} IPP System Overview} +\end{center} +\end{figure} + +\paragraph{Science Image Analysis} + +The Science Image analysis stages together represent the basic data +analysis required by the IPP. These analysis stages must process the +images in a timely manner so that the incoming data stream will not +overload the Pixel Server. The required processing time is derived +from the rate at which science images are obtained by PS-1. At a +minimum, the Science Image Analysis must keep up with the average +image rate over the course of 1 day. \tbd{The Science image analysis +is required to process images at the maximum science image rate from +PS-1 of 1 image every 30 seconds -- does this fall out of the science +requirements?} \tbd{In order to give time for uncertainties in the +Pan-STARRS system as a whole, the Science Image Analysis must be able +to process all images from a night within 12 hours.} + +\tbd{number of images per night, data volume per image, output +products} + +The science image analysis which must be performed by the IPP consists +of: + +\begin{itemize} +\item detrending the images to remove the instrumental signature + +\item astrometric and photometric calibration of the individual images + +\item merging a collection of several images of the same portion of +the sky obtained over a short period of time (to remove image defects +and gaps) + +\item subtracting the appropriate reference static-sky image + +\item cleaning the image of any transients + +\item adding the cleaned image to the static sky + +\item object detection of images at specific stages +\end{itemize} + +These analysis steps can be grouped into four phases, each of which +deals with a single data unit. We identify and discuss the +requirements of the four phases below. + +\paragraph{Phase 1 : image processing preparation} + +The Phase 1 analysis stage is performed on each science FPA to +calculate basic astrometric \tbd{and photometric} data needed by the +later stages. Phase 1 must use the static (pre-determined) telescope +distortion model and table of nominal OTA positions and rotations, +combined with the guide star pixel and celestial coordinates, to +determine the correct telescope bore-sight, field rotation and +magnification. The astrometric accuracy required from this analysis +stage is \tbd{2 arcsec} across the field, sufficient to match the vast +majority of reference stars with their detections. + +In some circumstances, science images may have no guide stars. This +may occur if the detectors are not run in OTA mode, especially for +short snapshot images of if IPP is being run on non-Pan-STARRS data. +In such a circumstance, the Phase 1 stage must perform extremely basic +object detection, determining the detector coordinates for stars which +are not excessively saturated and which are significantly above the +background level. The threshold levels for this object detection +stage must be configurable. The object extraction must be performed +in less than \tbd{3 seconds}. + +In order for astrometry of an image to succeed, it is necessary that +approximate image coordinates be known. The Phase 1 analysis must be +able to succeed despite initial coordinate errors as large as \tbd{5 +times} the field width. However, the search process must attempt the +near matches first in the assumption that the given coordinates are +accurate. + +A table of the overlaps between the science image to be processed and +the static sky images must be constructed. This table will be used to +guide the processing of the static sky in Phase 4. The overlaps must +be generously calculated so that small errors in astrometry at Phase 1 +will not cause any valid static sky / science image pairs to be missed +because of the astrometric error at this phase. It is acceptable for +a small number of invalid overlaps to be identified as these will be +excluded in Phase 4. Sky cells which do not have sufficient science +image overlap \tbd{$< 10\%$} need not be processed. + +It is not unusual that an image be obtained with invalid coordinates +or without any valid stars. For example, the telescope control system +may make an error and report the wrong time or coordinates. Or, the +image may be obtained in exceptionally poor conditions with no +detected stars. Phase 1 must fail gracefully in these conditions, +reporting an appropriate error. Such images must be identified for +possible human intervention, or future follow-up after metadata +repairs are made. + +\paragraph{Phase 2 : image reduction} + +The Phase~2 analysis is the detrend stage, in which the images from +the detector are processed to remove instrumental signatures. In +addition, basic object detection is performed along with improved +astrometric and photometric calibration. \tbd{what component selects +the appropriate calibration data? is it the phase~2 program, the +individual modules, or the scheduler above it?} In each step of the +analysis process, an image mask and noise map must be carried and +updated when appropriate. The following operations need to occur +within Phase~2 processing: + +\begin{enumerate} +\item Convolve detrend images with the OT kernel, if available +\item Flag bad and saturated pixels +\item Bias correction via overscan subtraction +\item Trim object image to remove overscan and edges corrupted by OT +\item Correct for non-linearity +\item Flat-field correction +\item Sky subtraction +\item Identify CRs +\item Find objects in the image +\item Make postage stamps of bright objects. +\end{enumerate} + +\subparagraph{Convolve detrend images with the OT kernel} + +Detrend images must be convolved by the OT kernel, so that they +accurately represent the detrend images appropriate for the object +images, which have been shifted using OT. The detrend images which +must be convolved include: the flat-field and the +high-spatial-frequency fringe images. \tbd{Must this be a formal +convolution with the analytical OT kernel, or can it be a convolution +with a decomposed kernel?} The appropriate kernel for each cell of an +OTA must be determined from the guide star history. \tbd{what is the +source of the OT kernel? pixel server?} + +\subparagraph{Flag bad and saturated pixels} + +A static bad pixel mask needs to be used to identify pixels which are +bad. Note that bad pixels which are charge traps need to be grown by +the extent of the OT convolution kernel, while those pixels above a +charge trap (i.e.\ bad colums) must not be grown, since they were not +affected by pixel shifting, but only became bad at read-out. + +Pixels saturated in the A/D converter must also be masked, and this +area must be grown by an additional pixel to mask excess charge +spillover. + +The bad pixel mask must be carried with the science images. Different +bits must be set to identify different reasons for masking the pixel. + +\subparagraph{Bias correction via overscan subtraction} + +The image bias must be subtracted. Since different detectors behave in +different ways, several options for modelling the bias must be +available. The bias must be measured from the image overscan region. +The bias subtraction method must be capable of applying a single +constant to the complete image, or to represent the bias as a function +which varies along the overscan. The function to be used must include +a spline or a chebychev polynomial derived from the data values along +the overscan. The values used to determine both the single constant +or the inputs to the spline and polynomial fits must be derived from +groups of pixels on the basis of one of several statistics, including +the sample and robust mean, median, and modes. In the case of a +single constant, all of the overscan pixel values are used in the +calculation of this statistic. In the case of the 1D functional +representation, the input values to the fit must represent the +coordinate along the overscan, with the statistic derived from the +pixels in the perpedicular direction at each location. Sigma-clipping +on the input data values must be an option. \tbd{accuracy of the bias +subtraction?} + +\subparagraph{Trim object image} + +The image must be trimmed to remove the non-imaging pixels, such as +the overscan and any pre-scan pixels, along with those pixels near the +edges that have been compromised due to OT operation. The definition +of the imaging area of the detector must be determined from the camera +configuration data or from the metadata associated with the image, +with the choice a user-configurable option. + +\subparagraph{Correct for non-linearity} + +If required, the object image (after bias correction) must be +corrected for the effects of non-linearity through a provided +polynomial fit to the pixel data values. The choice to apply the +correction must be set by the user. + +\subparagraph{Flat-field correction} + +The object image (after bias correction and non-linearity correction) +must be corrected for sensitivity variations as a function of +position, dividing by a flat-field image. The flat-field images must +be appropriately normalized (see section \ref{mkcal}). The +flat-fielded image must have a consistent photometric zero-point +across the chip, and across the full FPA, to within 0.2\%. + +\subparagraph{Sky \& Fringe subtraction} + +The flux contribution of the sky (from both continuum emission and the +line emission that causes fringing) must be subtracted from the +flat-fielded object image. The subtraction must remove background +(technically, foreground) variations which are not celestial but +generated in the atmosphere or by more localized scattering. This +background subtraction does not address the artifacts generated by +bright stars: bleeding columns, ghosts, or other localized reflection +effects. The background subtraction must remove the variations with +an accuracy such that the residual variations do not introduce, on +average, more than \tbd{0.2\%} photometric scatter or more than +\tbd{1\%} extremely deviant outlier stars (stars for which the +photometry is in error by more than 3\%). \tbd{what is the +requirement on galaxy photometry? morphology determinations?} +\tbd{What is allowed power-spectrum of background variations?} + +\subparagraph{Identify `cosmic rays'} + +Charged particles in the detector frequently cause features which do +not have the morphology of astronomical objects. In a well-sampled +image, these may be easily identified by the sharpness of the image. +In a near critically-sampled image, these `cosmic rays' may be +indistinguishable from stellar objects. The IPP must have the +capability of making the morphological identification of cosmic rays +if the imaging data is suitable. The identified cosmic rays must be +masked with a configurable growth factor (additional pixels beyond the +detected pixels in the feature). \tbd{The determination if the image +can be treated with morphological cosmic ray rejection must be made by +Phase~2.} + +\subparagraph{Find objects in the image} + +Objects on the flat-fielded object image must be found, and general +parameters, including the object centroid, instrumental magnitude, +local background level, and basic shape parameters ($\sigma_{\rm min}, +\sigma_{maj}$) measured. The detection threshold must be +configurable, and be a function of the average background flux or the +image noise map. Minimal object classification must be performed to +distinguish objects which are consistent with a single PSF, objects +which are inconsistent, and objects which are saturated. The +resulting collection of detected objects must be saved along with the +relevant image metadata (\ie filter, exposure time, etc). + +\subparagraph{Astrometry} + +Objects detected in Phase~2 must be matched with known astrometric +reference objects, using reference object coordinates which have been +adjusted for proper motion. The matched objects must be used to +improve the astrometric solutions for the individual OTAs. At this +stage, a user-defined collection of OTA astrometry parameters must be +fitted on the basis of the matched stars. The Cell astrometric +parameters must not be allowed to vary at this stage. The fit must be +robust, rejecting outlier matches (either stars with poorly determined +proper motion or spurious matches). The resulting astrometric +solution must be consistent across the OTA field to within \tbd{0.2 +arcsec}. + +\subparagraph{Postage Stamps} + +The IPP must have the capability of extracting regions surrounding a +specified subset of objects from the flattened images. These postage +stamp images must be saved for additional use by client science +pipelines. The identification of these objects must be on the basis +of a set of rules applied to the object magnitude and position. + +\paragraph{Phase 3 : exposure analysis} + +The Phase 3 analysis stage works with the results from a complete FPA +obtained during Phase 2 to improve the photometric and astrometric +calibrations. + +Phase 3 must use the objects detected in Phase 2, matched with an +appropriate reference catalog, to determine the image photometric zero +point and zero-point variations across the field. If zero-point +variations are significant \tbd{level TBD}, the zero-point variations +must be modeled with a chebychev polynomial correction of order 3 or +less. The complete FPA image must be categorized as photometric or +not \tbd{numerical scale?} on the basis of the zero-point consistency, +the transparency compared with recent long-term measurements in the +filter, and the external indicators of photometricity. + +Phase 3 must use the objects detected in Phase 2, matched with an +appropriate reference catalog, to determine improvements to the +astrometric solutions. The distortion model appropriate to this image +must be determined. The resulting astrometric accuracy must be +limited by the astrometric reference catalog \tbd{30 mas for USNO?} + +\paragraph{Phase 4 : image combination} + +Phase 4 is the image combination stage, in which multiple images of +the same portion of the sky are merged and confronted with the static +sky image. Phase 4 operates on the smallest data unit of the static +sky, the sky cell, along with the associated pixels from a collection +of images which have been processed through phases 1--3. For each sky +cell, the corresponding pixels are extracted from the exposures being +processed and mapped to the projection of the sky cell. The pixels +from the multiple input processed images are combined into a single, +cleaned image. This image is then confronted with the static sky cell +data to produce a difference image. Residual objects in the +difference image, above a threshold are detected and excised from the +original cleaned image. The remaining pixels are added to the +existing static sky image. Object detection must be performed on the +difference and cleaned images. \tbd{when is static sky object +detection \& classification performed?} Phase 4 naturally divides +into several stages, each of which are discussed in detail below. + +\subparagraph{Extract image pixels} + +For the given sky cell, the corresponding set of image pixels must be +determined and extracted from the input images. This process must use +the astrometric information for each OTA and Cell to determine the +exact overlaps. It must not miss any pixels, and it must read no more +than 20\% more pixels than necessary from the input images. + +\subparagraph{Transform pixel coordinates} + +Pixels which have been extracted from the input images must be mapped +to the corresponding pixels in the sky image. The tranformation must +be based on the measured astrometric solution for the input images +relative to the reference catalog used to generate the static sky +image. This warping must use a locally linear astrometric solution to +minimize computational effort. The output image must maintain be +photometric consistent with the input image to within 0.2\%. +\tbd{interpolation method?} + +\subparagraph{Flux matching} + +The multiple input images must have their object fluxes intercompared +using the stars measured in Phase 2 in order to determine the +appropriate photometry scaling factors needed to properly combine them +photometrically. + +\subparagraph{Image outlier pixel rejection} + +Pixels from the group of images which are inconsistent with the +ensemble of pixel values must be identified and flagged. The +resulting collection of pixels must be used to construct a single +output image, cleaned of the outliers. This outlier rejection must be +performed optionally since moving objects will be rejected in images +obtained over a wide range of times. + +\subparagraph{PSF matching} + +The multiple input images must have their PSF mutually matched to +allow for proper image subtraction. + +\subparagraph{Image Subtraction} + +The static sky image must be subtracted from the stacked, cleaned +image. All objects in the difference image must be detected and the +pixels belonging to variable sources flagged in the input image. +Object detection at this stage is the same as that used for Phase 2. + +\subparagraph{Cleaned Input Image} + +The flagged pixels must be excluded from the input images and a new, +cleaned image constructed. This image must have object detection +applied to it. \tbd{parameters} + +\subparagraph{Update static sky} + +The final, cleaned input image must be added to the static sky so that +an incrementally-deeper static sky image may be made. +\tbd{parameters, weight map} + +\subparagraph{Products} + +Phase 4 must produce the following data products at a minimum: +\begin{enumerate} +\item Subtracted image --- the combined image using each of the +telescopes, with the static sky subtracted; +\item New static sky image --- the combined image using each of the +telescopes, with the (old) static sky added; +\item Metadata about the quality of each of these images; and +\item A catalog of variable sources. +\item A catalog of sources from the combined image. +\end{enumerate} + +\subparagraph{Timing} + +It is required that the {\em total} processing for each exposure by +the Pan-STARRS system not take longer than $n \times T_{\rm min}$, +where $T_{\rm min}$ is the minimum time between exposures (30 sec), +and $n$ is a small positive number. Increasing $n$ results in a +proportionally higher expenditure on CPUs, hence it is strongly +desirable that $n \le 2$. + +Since we envision 4 OTAs (each 4k pixels, square) being processed by a +single CPU, we need Phase 4 to process 64 (input) Mpix in +approximately 30 sec (since Phase 4 is the most intensive, it should +receive the lion's share of the time budget), or 2 (input) Mpix per +second. + +\subparagraph{Accuracies} + +Transformations/mappings from detector to sky must preserve both +photometric and astrometric accuracies: +\begin{itemize} +\item Relative photometric accuracy better than \tbd{0.005 mag} +\item Absolute photometric accuracy better than \tbd{0.02 mag} +\item Relative astrometric accuracy better than \tbd{0.01 arcsec} +\item Absolute astrometric accuracy better than \tbd{0.2 arcsec} +\end{itemize} + +\subparagraph{Robustness} + +It is essential that the static sky image (which may have been +painstakingly accumulated over many months) not be corrupted by adding +in transient sources, or data that is of suspect quality (due, e.g., +to an error upstream in the processing). + +\paragraph{Calibration Stages} +\label{mkcal} + +The Calibration analysis stages may be performed on whatever +timescales are appropriate and necessary to maintain the quality and +relevance of the calibration images. We distinguish two major classes +of calibration images which require significantly different techniques +for their construction. We list the specific calibration images which +must be constructed in the calibration analysis stages. The +requirements for each of these stages are discussed in more detail +below. + +\paragraph{Basic Calibration Stages} + +The IPP must generate basic calibration images using the raw bias, +dark, and flat-field (dome or twilight) images obtained by the +telescope as the input. The analysis of these images requires +relatively simple stacking of the input set of images. Outlier +rejection, both of complete input images as well as pixels within the +input stack, must be performed. In addition, each type of image +requires an appropriate normalization which may depend on the data +levels in other detectors in the input set. Each of these calibration +stages must be able to determine from the input stack if the relevant +calibration image needs to be updated and perform an initial test to +see which input images are consistent and valid. + +\subparagraph{bias images} + +Bias images may be needed to correct for structure in the bias. The +IPP must have the capability of constructing a master bias image from +a stack of raw bias frames. The input bias images, representing +offsets from the overscan level, must have the overscan removed, +including 1D structure if needed. The bias construction must +incorporate outlier image and outlier pixel rejection. The statistic +used to determine pixel values must optionally be derived from the +sample mean, median, and mode, robust mean, median, and mode, and the +clipped mean and median. Residual images, in which the master bias is +applied to the input images must be constructed and their statistics +used to exclude any significant outlier input images. + +\subparagraph{dark images} + +Dark images may be needed to correct for structure in the dark +current. The IPP must have the capability of constructing a master +dark image from a stack of raw dark frames. The input dark images +must first be corrected for the bias using whatever method is +appropriate for the science images. The master dark frame must be +specified for a particular exposure time. As such, the input dark +frames must have a limited range of exposure times. The dark frame +construction must incorporate outlier image and outlier pixel +rejection. The statistic used to determine pixel values must +optionally be derived from the sample mean, median, and mode, robust +mean, median, and mode, and the clipped mean and median. Residual +images, in which the master dark image is applied to the input images +must be constructed and their statistics used to exclude any +significant outlier input images. \tbd{The dark frames must be +examined to determine the non-linearity of the measured dark current +-- by what component?}. + +\subparagraph{flat-field images} + +Master flat-field images must be constructed from a collection of +input flat-field images. An appropriate set of input images must be +selected on the basis of their flux levels, time of observations, and +the observing conditions. The input flat-field images must be +processed (bias and dark corrected if needed) and the resulting images +stacked. The master flat-field construction must incorporate image +and pixel outlier rejection. The statistic used to determine pixel +values must optionally be derived from the sample mean, median, and +mode, robust mean, median, and mode, and the clipped mean and median. +Residual images, in which the master flat-field image is applied to +the input images must be constructed and their statistics used to +exclude any significant outlier input images. + +\paragraph{Other Calibration Stages} + +\subparagraph{mask images} + +Initial bad-pixel mask images must be generated on the basis of +comparison between raw flat-field images and a cleaned, stacked +master. The mask creation analysis stage must accept a collection of +flat-field images and identify pixels which are repeatedly +inconsistent from image to image. If too many pixels are +inconsistent, an error must be raised. + +\subparagraph{fringe frames} + +Fringe-correction frames must be generated to remove the fringe +pattern caused by thin-film interference in the top layers of CCDs, +particularly in the redder passbands. Fringe correction frames must +be constructed on the basis of observations of the night-sky in the +appropriate filters. The images must first be flattened to remove the +pixel-to-pixel sensitivity variations of the detector. The +combination of multiple input fringe frames may not be simply stacked +since the amplitude of the fringe pattern varies independently of +other variations in the image. The amplitude of the fringe frames +must be measured and the images scaled to normalize the fringe +amplitude to the range -1 to +1 before combining with one of the +standard combination statistics (mean, median, mode, etc). + +\subparagraph{low-k sky models} + +Large-scale background structure in images which is not caused by +thin-film interference must also be detected and corrected. Models of +this background structure may be the necessary input to the correction +proceedure. The IPP must have the capability of generating image +models of the large-scale structure patterns observed with the +telescope. \tbd{discuss principal components, SVD?} + +\subparagraph{Flat-field correction frame} + +Flat-field images, whether constructed from the dome, twilight, or +night-sky images, rarely will perfectly correct the detector response +in a consistent fashion across the full field of the camera. The IPP +must have the capability of generating flat-field photometric +correction frames on the basis of the measured photometry of objects +which are placed at a variety of locations on the detector in a +sequence of images. + +\subparagraph{Non-linearity correction frames} + +The IPP must have the capability of constructing non-linear correction +frames. These frames are constructed from exposures of a uniform +source with a range of exposure times. The non-linearity correction +frames provide polynomial correction coefficients as a function of +pixel to convert the observed pixel counts to the expected pixel count +from a linear detector. + +\paragraph{Reference Catalog Creation} + +For PS-1, one of the primary goals is the creation of photometric and astrometric +reference catalogs for the general community and for the future +Pan-STARRS requirements. The generation of these catalogs is +inherently a research project, and will require human control and +intervention. The IPP will be required to provide the data access, +manipulation and visualization tools needed to construct these +reference catalogs and to assess their quality. In this section, we +list the requirements of the tools needed for this effort. + +\paragraph{Astrometry Reference Creation} + +The existing astrometric reference catalogs are known to have +limitations at the level of \tbd{NN} milli-arcsec. The internal +accuracy of the Pan-STARRS dataset can potentially be much higher than +the external reference catalogs. The IPP must have the capability of +generating an astrometric reference on the basis of the observations +obtained by the PnA survey. The IPP must provide the analysis tools +needed to generate the master astometric reference catalog. Much of +the required functionality is covered by the PnA Database. + +The necessary ingredients for the construction of the PS-1 Astrometric +Reference Catalog are: the observed coordinates of stars and the +existing astrometric reference catalogs. A variety of reference +catalogs will be required, including: +\begin{itemize} +\item Hipparcos +\item Tycho2 +\item UCAC +\item YBx +\item USNO-Bx +\item 2MASS +\end{itemize} +These catalog must be available and accessible to the PnA Database. +It is necessary to have the tools to convert the reference catalog +object coordinates to all of the possible coordinate frame of +relevance in the telescope and camera system, including: +\begin{itemize} +\item Catalog to mean positions +\item Mean to apparent positions +\item Apparent positions + pointing to focal plane coordinates +\item focal plane to specific detector (OTA) +\item specific detector to detector cell +\end{itemize} + +In addition to the reference catalogs, it will be necessary to +determine and have available for the analysis system a variety of +approximate calibration data, including the telescope and camera +optical distortion models and the variation of the image PSF across +the camera field, as a function of color. + +The final ingredient in the astrometry solution is the observation of +stars with the PS-1 telescope. The object detections are produced by +several of the analysis stages discussed in the Science Analysis +section. The likely measurement of relevance to the astrometric +reference catalog is the object extraction for the individual, +detrended images (section~\ref{foo}). \tbd{is it necessary to have + multiple centroiding methods available?}. The detected objects must +be matched against the reference catalogs, and it must be possible to +determine fit coefficients as a function of simply position, or with +combinations of magnitude or color. The fitting method must include +robust outlier rejection. It is also necessary to have information +about the objects which are detected in the catalog, but not the +science image or vice-versa, as well as an assessment of the +centroiding errors for each object. It must be possible to plot the +fit residuals against a wide variety of parameters, including the +object positions, magnitudes, colors, etc, and to make subset +selections of the objects on the basis of these parameters. . + +An alternative measurement of the stellar positions is derived from +the guide stars, which are much brighter than the typical saturated +stars. It must be possible to compare the coordinates of the guide +stars with the coordinates of the other stars in the image. It must +also be possible to perform the various fitting steps for the guide +stars rather than for the normal image data. + +\paragraph{Photometry Reference Creation} + +The IPP must provide the analysis tools needed to generate a master +photometric reference catalog. The tools needed for generation of the +photometric reference catalogs are similar in essence to those used +for the astrometric reference catalog. It is necessary to confront +the observed objects against the existing reference catalogs to +determine the necessary calibrations. Again, much of the required +functionality is covered by the PnA Database. + +The necessary ingredients for the construction of the PS-1 Photometric +Reference Catalog are: the observed magnitudes of stars and the +existing photometric reference catalogs. A variety of reference +catalogs will be required, including: +\begin{itemize} +\item SDSS +\item CFHT-LS standards +\item Landolt +\item etc +\end{itemize} +These catalog must be available and accessible to the PnA Database. + +The final ingredient in the photometry solution is the observation of +stars with the PS-1 telescope. The object detections are produced by +several of the analysis stages discussed in the Science Analysis +section. The likely measurement of relevance to the photometric +reference catalog is the object extraction for the individual, +detrended images (section~\ref{foo}). It is necessary to have the +tools to convert between different photometric systems, including: +\begin{itemize} +\item instrumental to nominal detector magnitude +\item nominal detector magnitude to average filter system +\item average filter system to reference photometry system +\end{itemize} +These transformations are based on a set of measured coefficients for +the color and airmass dependency of the measurement. In addition to +these types of transformations, it is necessary to have the ability to +measure and apply relative photometry corrections. + +The detected objects must be matched against the reference catalogs, +and it must be possible to determine fit coefficients as a function of +airmass, magnitude, color and detector coordinates, or with +combinations of the above. The fitting method must include robust +outlier rejection. It is also necessary to perform exclusions on the +basis of object locations, instrumental magnitudes, observed and +reference errors, and in particular time of the observations. It must +be possible to plot the fit residuals against a wide variety of +parameters, including the object positions, magnitudes, colors, etc, +and to make subset selections of the objects on the basis of these +parameters. . + +An alternative measurement of the stellar positions is derived from +the guide stars, which are much brighter than the typical saturated +stars. It must be possible to relate the magnitudes of the guide +stars with the magnitudes of the other stars in the image. It must +also be possible to perform the above fitting steps for the guide +stars rather than for the normal image data. + +\subsubsection{Modules} + +In order to encapsulation functionality, the analysis stages are +constructed of a sequence of steps. The analysis stages consist of a +\tbd{python} script which executes a sequence of C-level functions. +The C-level functions called by the \tbd{python} script are called +{\em modules} and represent basic data analysis operations. + +The required set of Pan-STARRS modules and their functionality is +specfied in the document `Pan-STARRS Image Processing Pipeline Modules +Supplementary Design Requirements' (PSDC-430-xxx), and details of +specific apgorithms are specfied in the document `Pan-STARRS Image +Processing Pipeline Algorithm Design Document' (PSDC-430-006). + +\subsubsection{PanSTARRS IPP Library} + +In order to facilitate testing and development, and to encourage +flexibility, the IPP will be built in a layered fashion. The lowest +level functions will be written in C and collected together into a +Pan-STARRS library, \code{PSLib}. + +The Pan-STARRS Data Library will consist of C structures describing +the basic data types needed by the IPP and C functions which perform +the basic data manipulation operations. The library is organized into +four topics: System Utilities, Basic Data Collections, Data +Manipulation, and Astronomy-Specific Functions. + +The required functionality of the Pan-STARRS Data Library is specified +by the document `Pan-STARRS Image Processing Pipeline Library, +Supplementary Design Requirements' (PSDC-430-007), and details of +specified algorithms are specified in the document `Pan-STARRS Image +Processing Pipeline Algorithm Design Document' (the ADD; +PSDC-430-006). + +\subsubsection{Data Sources and Formats} + +\paragraph{Image Formats} + +FITS images + +\paragraph{Table Formats} + +FITS tables + +\paragraph{Other Data Formats} + +XML files + +\paragraph{External Catalogs} + +\begin{itemize} +\item Hipparcos +\item Tycho2 +\item HST-GSC +\item USNO-A +\item UCAC +\item 2Mass +\item USNO-Bx +\item YBx +\end{itemize} + +\paragraph{Analysis Reference Data} + +\begin{itemize} +\item Telescopes +\item Cameras +\item Detectors +\item Filters +\item software basic parameters +\end{itemize} + +\paragraph{Installation Reference Data} + +\begin{itemize} +\item computers +\end{itemize} + +\subsection{External Interfaces} + +\subsection{Internal Interfaces} + +\subsection{Internal Data Requirements} + +\subsection{Computer Hardware} + +\subsubsection{Overview} + +This section discusses the Pan-STARRS Image Processing Pipeline (IPP) +PS-1 hardware requirements. The hardware requirements addressed in +this section consist of: + +\begin{itemize} +\item Total Disk Volume +\item Total Processing Power +\item Sustained Switch Bandwidth +\item Sustained Node Network I/O +\item Sustained Disk I/O +\end{itemize} + +Even without the complete IPP design, it is possible to identify the +major drivers on the hardware requirements. The total disk volume +requirements are dominated by the need to store raw images for a +certain period, the need to store calibration images for a longer +period, and the need to store the static sky images. Of the various +analysis stages, Phase 2 and Phase 4 present the most significant +demands in terms of data I/O throughput on the network. Phase 2 and +Phase 4 also present the most significant CPU demands. In this +discusion, Phase 2 refers to the per-OTA image pre-processing in which +the instrumental signature is removed and a first pass object +detection is performed. Phase 4 refers to the multiple OTA +combination in which the pre-processed images are merged and combined, +in both addition and subtraction, with the static sky image, and up to +three object detection passes are performed. + +This document does not address the hardware requirements implied by +Phase 1 or 3, nor the load required by the calibration or reference +catalog creation stages. In the first instance, the operations are +only performed on the metadata and are extremely minimal both in terms +of data I/O and computation requirements. In the second case, the +processing is less time critical than the per-image processing and is +performed only infrequently (once per night to once per week, month or +year). \tbd{The software implementation for metadata storage (RDBMS, +FITS tables, etc) will have a very large impact and will be evaluated +along with the needed hardware at a later date.} + +We will address the various hardware requirements by referring to an +assumed data processing and data organization scenario. The +organization of the data and certain aspects of the data processing +scheme have very large implications for the hardware requirements. In +this analysis, we assume that data types are chosen to minimize the +data volume and that the data is organized to minimize the I/O +bandwidth needs, as defined below. We address the data requirements +of the single-telescope Pan-STARRS-1 scenario based on the Design +Reference Mission \tbd{REF}. + +\subsubsection{Data Organization} + +The IPP hardware system must provide both data storage and +computational resources. The IPP requires relativley large amounts of +data storage space, primarily for the image data. Image data is +organized in two categories. First, there is the per-OTA data -- data +associated with specific OTAs, including the raw images, the +calibration images, and temporary processed images at various stages. +Second, there is the data associated with the static sky imagery, +which is in turn organized into smaller sky-cell units. The first +assumption we make is that the hardware is organized into nodes which +provide both data storage and computational resources. The second +assumption we make is that the data storage nodes are divided into two +classes: those which deal with the per-OTA data and those that provide +the static sky storage. In addition, we assume that the computational +tasks related to Phase 2 take place on the per-OTA storage nodes and +the Phase 4 computation takes place on the static sky storage nodes. + +Figure~\ref{hardware} shows our basic concept for the hardware +organization for the IPP. This diagram shows the two types of compute +nodes: OTA-level processing and storage nodes (dominated by Phase 2) +and static sky processing and storage nodes (mostly Phase 4). Also +shown are two switches used in this configuration; although it is +currently possible to buy a single switch with sufficient number of +ports, this organization represents a minimal configuration for the +PS-1 IPP hardware. In such a case, the interswitch communication must +also meet the required throughput needs. We discuss the hardware +requirements in the assumption that such an organization will be +necessary. + +The way in which the images are distributed among the storage and +compute nodes will largely determine the I/O bandwidth requirements. +For data bandwidth requirements calculations, it is necessary to make +some assumptions about the data organization. We make the assumption +that the OTA data is optimally distributed to the OTA nodes such that +the OTA processing is always on a machine with local OTA data. This +implies that all OTA data from a specific OTA are targetted to a +specific machine. (see below for discussion of data duplication). + +A second factor which will have a significant impact on the I/O +requirements is the image storage format for the processed and +calibration images. We have two basic choices: 32 bit floating point +format or 16 bit integer format with appropriate scaling. In the +former case, additional dynamic range is retained, while in the latter +case, we reduce the data volume by a factor of 2. Since the science +requirements for PS-1 do not specify a need for dynamic range greater +than 16 bits, we assume all images are stored as 16 bit data. + +A third determining factor is the number of calibration images needed +by the processing system. Since the complete analysis is not yet +defined, this number is difficult to ascertain. However, we can make +a reasonable guess at the total number for scaling purposes. We +assume that each frame requires a total of 4 calibration frames on +average + +\begin{table}[b] +\begin{center} +\caption{Data Storage Requirements \label{storage}} +\begin{tabular}{lrrrr} +\hline +\hline +Raw data & 200 TB \\ +static sky & 256 TB \\ +calibration frames & 5 TB \\ +metadata db & 0.3 TB \\ +object db & 4 TB \\ +\hline +total & 116 TB \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\subsubsection{Data Storage Requirements} + +The Pan-STARRS IPP data storage requirements may be divided into five +principal areas: raw image data, static sky image data, master +calibration images, the metadata database, and the object database. +We discuss each of these data items and their impact on the data +storage requirements for the IPP for PS-1. Table~\ref{storage} +summarizes the data storage requirements in the different scenarios. + +\paragraph{Raw Data Storage} + +There are two basic image types which will be acquired: night-time +science images and calibration images. The night-time science images +consist of 1Gpix per image, or 2GB in raw format. At nominal cadence, +the PS-1 telescope can obtain images at a sustained rate of 1 image +per 30 seconds for the entire night of 10 hours (36000 seconds). A +total of 100 calibration images per night would be a substantial +overestimate of the typical expectation. Combining these numbers, we +can expect to receive a total of 1300 images, or 2.6 TB of data per +night. The total data storage requirements for the raw data are +governed by the number of nights' worth of data we are required to +keep online. \tbd{for the first year, we are required to keep all +images from the PnA and IPV surveys. This amounts to a total of 200 +TB of data}. + +\paragraph{Static Sky Data Storage} + +The static sky is represented by images with 0.2 arcsec per pixel. +There will be one summed image and one weight image for each of the +\tbd{6} filters, each stored with 16 bits of resolution, for a total +of 24 bytes per sky pixel. At this resolution, there are 324 Mpix per +square degree, and we will observe a potential total area of 30,000 +square degrees. Allowing for 10\% overage for overlapping tiling, we +require a total of 10.7 Tpix to cover the sky once, or a total of +$\sim 256$ TB to maintain a single image of the static sky in all 6 +filters. + +\paragraph{Calibration Frame Storage} + +The possible required calibration frames consist of the bias, dark, +and mask images, along with one flat, one flat-correction, and +multiple sky/fringe library frames per filter. In fact, not all types +are needed at all stages. It is very likely that we will not require +bias or dark images, and mask images may be represented by a single +byte per pixel. Nonetheless, it is necessary for us to generate and +store all master calibration frames at least until we prove that they +are not needed. We assume a total of 21 calibration images are +necessary (one flat, fringe, and sky per filter, along with a bias, +dark, and mask). If we intend to keep all master calibration frames +for the project lifetime, and generate a new master on a weekly basis +(a reasonable time-scale), then we can expect to require a total of 5 +TB of calibration image by the end of the 2 years of PS-1. We note +that this is likely to be a drastic overestimate as we are unlikely to +need to regenerate all master calibration frames on a weekly +time-scale. + +\paragraph{Metadata Database Storage} + +The metadata data storage requirements are driven by the need to store +the data for the project lifetime. There are two types of metadata +generated at the summit: data associated with images and environmental +data. The environmental data consists of measurements on a regular +cadence, roughly 1 per minute, of a variety of parameters. We suggest +an expected of 1kB per entry, for a total of 1 GB over the two-year +term of PS-1. The additional systems, such as the DIMM, SkyProbe, NIR +Sky Camera, and the LRProbe will have higher data requirements, but +should be considered as separate, self-contained systems. Their data +products are distilled to a limited number of parameters per minute +which are included in the 1kB given above. Furthermore, items such as +guide-star history, if saved, will be saved with the image data and +represents only a small fraction of the total image data volume. Some +subset of the telescope diagnosic information may be a high volume +data product as well, but only retained by the telescope control +system for the purpose of diagnostic studies. Such data will be +excluded from this analysis. + +The image metadata consists of values associated with the FPA (1), the +OTAs (64), and the Cells (4096). Aside from the guide star history, +the total data requirements for each of these entries will be scaled +by the number of bytes required for the metadata from each data level. +Clearly, if the Cell entry is allowed to be large, it will dominate +the total Metadata data volume. We suggest an expected number of 64 +bytes per Cell, 256 B per OTA, and 1k per FPA, yielding a total +metadata volume per exposure of roughly 0.3 MB, completely dominated +by the Cell metadata. With the exposure rates above, we find a total +of metadata volume of 0.3 TB over the two-year term of PS-1. + +\paragraph{Object Database Storage} + +The hardware requirements for the IPP object database are rather +flexible: the total volume depends critically on the depth to which +the object detection analyses are performed (and thus the total number +of object detections) and the number of object parameters which are +measured. We can make very rough estimates that the total number of +detections over the 2 year lifetime of the project may be in the +vicinity of $10^{11}$. We can conservatively estimate the number of +bytes needed to represent each detection as 128 B, resulting in a +total data storage for the object detections of 12 TB. However, this +number depends strongly on the timescale for which the IPP is required +to maintain all object detections, and may potentially be +significantly reduced. + +\subsubsection{CPU Requirements} + +Phase 2 and Phase 4 dominate the processing requirement, primarily +because they must keep up with the image delivery rate of 1 per 30 +seconds. We have performed benchmarks of a demonstration version for +both the Phase 2 and Phase 4 analyses. + +For the Phase 2, a substantial fraction of the processing time is +consumed by the need to perform FFTs on the images in order to +convolve them with the guide-star kernel, and in the smoothing used +for the object detection process. Additional processing time is +needed by the object detection, deblending, and analysis. Experiments +with the FFTW package show that FFTs may be performed on Intel +processors at rates of approximately 0.25 GHz-sec / Mpix for data sets +of order 1 Megapixel. The FFTs required for the Phase 2 analysis are +performed on the 512$^2$ pixel cells, so these numbers may roughly be +scaled linearly to determine the total time required for OTA +processing. A single FFT on a full OTA, with 64 Cells, therefore +requires roughly 4 GHz-sec. For the full Phase 2 analysis, there are +roughly 4 single direction FFTs required excluding those associated +with object detection; thus the total processing time for these FFTs +is approximately 16 GHz-sec. The addtional analysis steps, excluding +object detection and characterization, account for a small fraction of +this compute time, which we estimate at 10\%. The object detection +stage depends somewhat on the depth to which the analysis is +performed, and the number of measurements made per object. Typical +analysis performed by the Sextractor routine, which performs a +substantial number of per-object analyses, requires 27 GHz-sec for a +full OTA, including the FFTs used for smoothing. We can therefore +assume a total of 50 GHz-sec per OTA for the Phase 2 processing. This +converts to a total of 12800 GHz-sec for a complete major frame. + +For Phase 4, the main computational tasks are combining the multiple +images, with cosmic-ray rejection, and performing the object detection +tasks. Nick Kaiser has done tests of the Phase 4 image combine and +rejection stages, and finds a total processing time of roughly 96 +GHz-sec for a full stack of 4 OTA images. If we add in an additional +34 GHz-sec for detailed object detection and image differencing, we +find a conservative estimage of 130 GHz-sec for a 4-image OTA stack, +equivalent to 7800 GHz-sec for a major frame. + +For PS-1, the typical time for a major frame is $4 \times 30$ seconds. +Some reduction in the load may be gained by reducing the complexity +and depth of analysis for PS-1. Depending on the details and depth of +the analysis, we may reduce the computational load by a factor of 2. + +\begin{table} +\begin{center} +\caption{Data I/O (MB per OTA or Sky-cell) \label{scenarios}} +\begin{tabular}{lrrrr} +\hline +\hline +{\em Phase 2 input} \\ +from summit & $2 \times 32$ MB \\ +input image & {\bf 32 MB} \\ +calibration & {\bf 4 $\times$ 32 MB} \\ +mask image & {\bf 8 MB} \\ +\hline +network I/O: & 64 MB \\ +disk I/O: & 176 MB \\ + & \\ +{\em Phase 2 output} \\ +output image & {\bf 32 MB} \\ +output mask & {\bf 8 MB} \\ +image to P4 & $1.5 \times 32$ MB \\ +mask to P4 & $1.5 \times 8$ MB \\ +\hline +network I/O: & 60 MB \\ +disk I/O: & 40 MB \\ + & \\ +{\em Phase 4} & \\ +input images & $1.5 \times 4 \times 32$ MB \\ +input masks & $1.5 \times 4 \times 8$ MB \\ +static sky & 32 MB \\ +static weight & 32 MB \\ +\hline +input: & 304 MB \\ +output: & 96 MB \\ +\hline +\multicolumn{5}{l}{\em Bold-faced entries are access to local-disk} \\ +\multicolumn{5}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\ +\end{tabular} +\end{center} +\end{table} + +\subsubsection{Per-Node I/O Requirements} + +Data I/O per node is defined as the number of bytes per second passed +through the node's network adapter. The data throughput for each node +depends strongly on the how the data is organized and processed. In +this section, we identify the data which is passed between nodes for +the two stages of the science analysis process. Table~\ref{scenarios} +lists the per-node data I/O for the analysis stages. + +For PS-1, there are 120 seconds of compute time allowed for each of +the Phase 2 and Phase 4 analyses for the collection of four images +which makes up a cannonical major frame. We use the data I/O volumes +and some assumptions about expected network and disk bandwidth to +estimate the I/O and processing timeline for the four scenarios. From +this analysis, we can judge the total CPU requirements in terms of +GHz, not just GHz-sec. We have assumed that GigE network adapters are +capable of delivering data at 50MB/sec sustained and that a disk RAID +can deliver sustained 100 MB/sec reads and writes. These numbers are +conservative estimates based on recent tests discussed below. Using +these assumptions, Table~\ref{throughput} lists the time allocations +for the processing stages. + +\paragraph{Phase 2 Node I/O Requirements} + +In the assumed data distribution scenario, there is a single CPU +allocated to each OTA in the OTA farm and a single CPU for each Sky +cell process. In addition, all data for the specified OTA are stored +on local disks attached to the same computer as the CPU, with the +result that all Phase 2 I/O is made to a local disk. For each science +OTA image which is observed, each OTA node will read from the network +a total of 2 raw images (one for the original image, one for a backup +copy) and write an average of roughly 1.5 processed images and masks +to the Phase 4 machines for a total of 124 MB of network I/O. During +the processing stage, the OTA node will read from disk a total of 176 +MB (4 calibration frames at 32 MB each, one 16 MB mask, and one raw +science image at 32 MB) and write a total of 40 MB (one processed +image at 32 MB and one mask at 8 MB). Given the assumptions for the +network and disk bandwidths (50 MB/s and 100 MB/s respectively), the +data volumes imply a total I/O period of 4.6 seconds. In this +instance, the network I/O is presumed to be sequential with the disk +I/O. + +\paragraph{Phase 4 Node I/O Requirements} + +Although it is easy to arrange the OTA data in such a way that the +majority of I/O is performed locally, it is not as easy to arrange +this for the Static Sky data used by the Phase 4 analysis. We +therefore make the assumption that the Phase 4 analysis will require +all input OTA data to be loaded across the network, as well as all +Static Sky data. This is somewhat of an overestimate as some of the +Static Sky data will be processed by machines with the data stored +locally, and clever Static-Sky data organization schemes can enhance +this chance. + +In the Phase 4 analysis, the images from the 4 separate telescopes are +combined into a single image, confronted with the appropriate segment +of the static sky, with output difference image and updated static sky +image. If we restrict input access to the individual OTA cells, the +maximum read overhead is 50\% (need to read a 10x10 set of cells for +an 8x8 input image). If the processing is performed on Static Sky +segments equivalent in size to the OTAs, the total volume of input +data per node is 304 MB (192 MB of processed science image, 48 MB of +input mask, 32 MB of static sky image and 32 MB of static sky weight +map) while the output data is 96 MB (32 MB static sky, 32 MB weight +map, and 32 MB difference image). Thus, we require a total of 400 MB +network I/O, which implies an I/O period of 8 seconds. + +\begin{table} +\begin{center} +\caption{Data Throughput \label{throughput}} +\begin{tabular}{lrrrr} +\hline +\hline +Phase 2 per-node network I/O & 2.2 s \\ +Phase 2 per-node disk I/O (read) & 1.3 s \\ +Phase 2 per-node disk I/O (write) & 1.2 s \\ +Phase 2 CPU total & 25 s : 128 GHz \\ +Phase 4 per-node I/O & 8 s \\ +Phase 4 CPU total & 112 s : 70 GHz \\ +Phase 2 switch load & 264 MB/s \\ +Phase 4 switch load & 215 MB/s \\ +Phase 2 to Phase 4 switch load & 160 MB/s \\ +Summit to Phase 2 switch load & 70 MB/s \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\subsubsection{Switch I/O Requirements} + +The switch I/O requirements are defined by the total number of bytes +per second serviced by the two switches in the system. + +The Phase 2 network I/O is 124 MB per OTA. With 64 OTAs per image, +and 30 seconds average between images, this implies a total of 264 +MB/s switch bandwidth. The Phase 4 network I/O is 400 MB per sky +cell. With 64 cells and 120 seconds between major frames, this is an +average switch bandwidth of 215 MB/s switch bandwidth. The total +switch-to-switch load is 304 MB per OTA, with an average timescale of +120 seconds. With 64 OTAs, this corresponds to 160 MB/s. The +summit-to-Phase 2 switch load is 70 MB/s. + +\begin{table} +\begin{center} +\caption{Hardware Throughput Tests \label{existing-hardware}} +\begin{tabular}{lrrrr} +\hline +\hline +Test & where \& when & model & result \\ +\hline +node I/O & CFHT 11/2002 & Intel 1000 Gigabit & 35 - 40 MB/s sustained \\ +node I/O & CFHT 2/2004 & Intel 1000 Gigabit & 65 - 70 MB/s sustained \\ +RAID write & CFHT 2/2004 & 3ware RAID cntl + IDE & 110 MB/s sustained \\ +Switch Load & VeriTest & Cisco & 3 GB/s (for 32 ports) \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\subsubsection{Existing Hardware Throughput} + +We have collected a few representative tests of various pieces of +modern hardware to give a reference for the throughput capabilities. +A number of hardware configurations have been tested at CFHT for the +Elixir project, and we include here their recent reported hardware +RAID-5 I/O speeds and GigE card speeds. We also have included data +from VeriTest studies of Cisco switch throughput, commissioned by +Cisco for a 32 port GigE switch. These tests are summarized in +Table~\ref{existing-hardware}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Test Verification} + +A testing regime must be implemented to demonstrate the working state +of the provided software. Certain tests as specified must be +performed by MHPCC, with code release contingent on success. Other +specified tests will be performed by IfA to verify the validity of the +implemented algorithms. The tests include: software configuration +tests, software integrity tests, basic unit tests, and detailed +functional analysis. + +\subsection{Software Configuration Tests} + +MHPCC must test the validity of the software configuration, +specifically to check that the code can be compiled on the specified +platforms and that the compilation produces no errors or warnings, +except as noted and allowed. + +\subsection{Software Integrity Tests} + +MHPCC must test the integrity of the software, specifically to check +that the code does not produce memory leaks, segmentation faults. + +\subsection{Basic Unit Tests} + +MHPCC must perform basic unit tests with sample input data and known +output results, including invalid input data to test error handling. +These tests must exercise the complete range of module options. + +\subsection{Detailed Functional Analysis} + +IfA must perform detailed tests with a wide range of input data and +compare the results with existing software system. + +\subsection{Test Verification Matrix} + +\subsubsection{Pan-STARRS IPP Library} + +See Appendix A \& B of the IPP Library SDR (PSDC-430-007) for the test +verification matricies for the Pan-STARRS IPP Library + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Appendices} + +\bibliographystyle{plain} +\bibliography{panstarrs} +\end{document} + +Requirements Trace Matrix + +active state \ref{req:active-state} +paused state \ref{req:paused-state} +interactive state \ref{req:interactive-state} + +system capabilities + +C for source code \ref{req:languages} +Python for scripts \ref{req:languages} + +SWIG interfaces +C APIs + +POSIX +Pan-STARRS Coding Standard + +Naming Conventions + Index: unk/doc/design/specs.tex =================================================================== --- /trunk/doc/design/specs.tex (revision 770) +++ (revision ) @@ -1,2140 +1,0 @@ -%%% $Id: specs.tex,v 1.7 2004-04-27 18:38:31 eugene Exp $ -\documentclass[panstarrs]{panstarrs} - -% basic document variables -\title{Pan-STARRS Image Processing Pipeline} -\subtitle{Software Requirements Specification} -\shorttitle{IPP SRS} -\author{Eugene Magnier, Paul A. Price, Josh Hoblitt} -\group{Pan-STARRS Algorithm Group} -\project{Pan-STARRS Image Processing Pipeline} -\organization{Institute for Astronomy} -\version{DR} -\docnumber{PSDC-430-005} - -% allow paragraphs to be listed in TOC for now -\setcounter{tocdepth}{4} - -\begin{document} -\maketitle - -% -- Revision History -- -\RevisionsStart -% version Date Description -DR.01 & 2003.01.01 & First draft \\ \hline -DR.02 & 2003.03.10 & Second draft \\ \hline -DR.03 & 2003.04.13 & Most paragraphs fleshed out \\ \hline -\RevisionsEnd - -\listoffigures -\pagebreak - -\tableofcontents -\pagebreak -\pagenumbering{arabic} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Scope} - -\subsection{Identification} - -This document establishes the system requirements for the Pan-STARRS -Image Processing Pipeline (IPP) as applied to Pan-STARRS 1 (PS-1), the -initial demonstration telescope to be constructed on Haleakala by Jan -2006. - -\subsection{System Overview} - -\tbd{description of the Pan-STARRS System and PS-1.} - -\subsection{Document Overview} - -The Pan-STARRS document naming scheme is PSDC-NNN-MMM-VV, where the VV -entry specifies the document version number. Where documents are -identified without the version number, the latest official version in -that series is implied. - -Open Issues and TBDs in this document are marked \tbd{in bold, red -with surrounding square brackets}. - -All timing measurements are to execution time as measured on a -\tbd{Reference Pan-Starrs Computation Node} and assumed to be not -limited by network bandwidth. - -\subsubsection{Definitions} - -\paragraph{``Must''} When used in this specification, the word -``must'' refers to an explicit requirement of a system component or -the complete system. - -\paragraph{``Should''} When used in this specification, the word -``should'' refers to a desired chracteristic of a system component or -the complete system. - -\paragraph{``Will''} When used in this specification, the word -``will'' provides information about a characteristic of a related -system component or a complete related system. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\DocumentsInternalSection -PSDC-430-xxx & PS-1 Design Reference Mission \\ \hline -PSDC-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline -PSDC-430-006 & Pan-STARRS IPP ADD \\ \hline -PSDC-430-007 & Pan-STARRS IPP PSLib SDR \\ \hline -\DocumentsExternalSection -Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ -\DocumentsEnd - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Requirements} - -\subsection{Required States} - -The IPP must have 3 states: active, paused, and interactive. - -\subsubsection{Active State} -\label{req:active-state} - -In active state, the IPP must accept images and metadata from OATS and -automatically perform the complete set of image processing tasks, -including both calibration and science image processing. The IPP must -respond to requests for data from the client science pipelines -\tbd{and IPP monitoring team}. - -\subsubsection{Paused State} -\label{req:paused-state} - -In paused state, the IPP must refuse data and metadata from OATS and -data requests from the client science pipelines. - -\subsubsection{Interactive State} -\label{req:interactive-state} - -In interactive state, the IPP must accept data and metadata from OATS, -but must not automatically process the data. The IPP must respond to -user commands to initiate portions of the data analysis. - -\subsection{System Capability Requirements} -\label{req:system-capabilities} - -The IPP must perform the following tasks: - -\begin{enumerate} - -\item Accept raw images from OATS at a sustained rate of 1 exposure - per 30 seconds. - -\item Accept metadata from OATS at a sustained rate of \tbd{XXX MB / sec}. - -\item Produce high-quality calibration images from the raw calibration - images. The calibration images must not introduce systematic - uncertainties greater than \tbd{0.2\%}. \tbd{Requirements on the - speed of processing the calibration images.} - -\item Pre-process the science images with the high-quality calibration - images. - -\item Merge multiple pre-processed science images -- from multiple - telescopes or from sequential, dithered exposures -- into single, - cleaned, stacked images. - -\item Subtract a static sky image from the cleaned, stacked images to - produce an image of only the transient events. - -\item Excise the significant transients and outliers from the - pre-processed science images and merge the cleaned images into the - static sky image. - -\item Detect objects on the four types of images: pre-processed - images, the stacked image, the difference image, and the static sky - image. - -\item Determine astrometry of the detected objects relative to an - astrometric reference to an accuracy of \tbd{30 mas}. - -\item Determine photometry of the detected objects relative to a - photometric reference to an accuracy of \tbd{5 millimag} relative - photometry and \tbd{10 millimag} absolute photometry in photometric - weather. - -\item Produce a high-quality astrometric reference catalog from the - extracted objects on a time-scale of 6 months. The astrometric - reference must have an absolute accuracy of \tbd{30 mas} and a local - relative accuracy of \tbd{10 mas}. Proper motions of all nearly - stationary objects must be determined with an accuracy of \tbd{XXX - mas / year}. - -\item Produce a high-quality photometric reference catalog from the - extracted objects on a time-scale of 6 months. The photometric - reference must have an consistency across the sky of \tbd{5 - millimag} and an absolute calibration to the external system defined - by \tbd{SDSS} of \tbd{10 millimag}. - -\item Publish the static sky images to the Pan-STARRS published static - sky server on a time-scale of \tbd{1 month}. - -\item Publish the detected objects to the Pan-STARRS published object - database on a time-scale of \tbd{1 week}. - -\item Provide access to external Pan-STARRS clients to the detected - objects on time-scales of \tbd{1 minute} after the image is - processed. - -\end{enumerate} - -\subsubsection{Software Coding Requirements} - -\paragraph{Languages} -\label{req:languages} - -Source code must be in C. All source code must be compiled with `gcc' -version v2.95 or higher. - -Scripting language must be \tbd{Python, version TBD}. - -\paragraph{Interfaces} -\label{req:interfaces} - -Access to low-level Library functions must be provided via C APIs -consisting of the function calls and the defined data structures and -other data types. Access to high-level functions must be provided -via C APIs as well as SWIG interfaces, where specified. Access to -processing jobs must be available via the UNIX shell. - -\paragraph{Coding Standards} - -The C code must comply with ANSI Standard C99. Because the delivered -code is required to run on UNIX machines, the delivered code must be -in compliance with the language-independent UNIX operating system -standard POSIX (Open Group Based Specifications Issue 6, IEEE Std -1003.1, 2003). Source code files must use the UNIX line-break -convention (line-feed only). C coding style must adhere to the -standard defined in the document 'Pan-STARRS C-coding standard' -(PSDC-430-004). \tbd{Python coding must follow the Python standard -defined in the document TBD}. - -\paragraph{Naming Conventions} - -Header files must have names starting \code{ps} or \code{p_ps} for -private interface definitions. The latter must appear in a -subdirectory \code{private} of whichever directory is being searched -for the public header files. - -Functions visible at global scope which are part of the public API -must have names begining with \code{ps}, and follow the naming -conventions in the coding standard. Functions that are visible at -global scope but which are not part of the public interface must have -names begining with \code{p_ps}. Functions that are local to a file -must \textit{not} start \code{ps} (or \code{p_ps}). - -Variables visible at global scope which are part of the public API -must have names begining with \code{ps}, and follow the naming -conventions in the coding standard. Variables that are visible at -global scope but which are not part of the public interface must have -names begining with \code{p_ps}. Variables that are local to a file -must \textit{not} start \code{ps} (or \code{p_ps}). - -The names of all enumerated types and C-preprocessor symbols (but not -variables declared \code{const}) must start with \code{PS_}, in the -case of public symbols, or \code{P_PS_}, for private symbols. The -rest of the name must be uppercase with words separated by underscores -(\code{_}). An exception is the case of system utilities implemented -as macros, in which case the names must conform to the convention for -function names. - -When defining a function to convert from one type to another, the name -must be of the form \code{psOldToAlloc}, e.g.\hfil\break -\code{psEquatorialToEcliptic} (\emph{not} -\code{psEquatorial2Ecliptic}). - -\paragraph{C Programming Guidelines} - -Functions that assign to a variable must list that argument -\textit{first}, following the pattern of \code{strcpy}; e.g. -\begin{verbatim} -void psAddToVector(restrict psVec *outVec, const restrict psVec *inVec, - int val); -\end{verbatim} - -Type definitions should always be accompanied by prototypes for their -constructors and destructors, following these guidelines: - -\begin{itemize} -\item The constructor name should consist of the type name followed by -\code{Alloc}; e.g. a type \code{psImage} would be created by a -function -\begin{verbatim} -psImage *psImageAlloc(int nrow, int ncol); -\end{verbatim} - -\item The type should be freed with a destructor named \code{typeFree}, e.g. -\begin{verbatim} -void psImageFree(psImage *img); -\end{verbatim} - -\item The constructor must never return \code{NULL}, and no code calling the -constructor should ever check the return value. - -\item The destructor must not return a value. - -\item The destructor must handle being passed \code{NULL} by simply -returning immediately. This must not be treated as an error -condition. - -\item Constructors and Destructors should use the memory reference - counter facilities of the PSLib memory management system. - -\end{itemize} - -\paragraph{Commenting and Documentation} - -Commenting of delivered C and Python code must follow the C and -Python coding standards and must provide tags for Doxygen -interpretation of the comments and program structures. - -Documentation for the IPP consists of source code documentation and -user documentation. Source code documentation must be generated with -Doxygen from the in-line comments and must be provided as HTML, -Latex, and man pages. User documentation includes the API usage for -the modules and library functions as well as user interface -description for the higher-level architectural systems. User -documentation must be delivered as PDF documents. - -\paragraph{Version Control} - -Source code version control must be implemented with CVS. - -\paragraph{CSCI Deliverable} - -All final source code generated for the IPP is to be delivered via -CVS, including the test code. CVS revision history must be included -and made available via CVS. - -\paragraph{Platform architectures and operating systems} - -Makefiles must be provided with appropriate flags set so that all -code compiles without warnings under 'gcc -Wall' for the following -platform architectures and operating systems: - -\begin{itemize} -\item x86/Linux -\item PPC/OS-X -\end{itemize} - -The requirement of compiling without warnings includes the allowance -that the output may be filtered to exclude known, specified warnings, -such as those caused by lex-generated code. - -Although the code must compile successfully under both listed -operating systems, unit testing should only be performed for the -x86/Linux combination. - -\paragraph{Software Configuration} - -\tbd{deferred} - -\subsubsection{Architectural Components} - -In order to achieve the required functionality, it is necessary to -divide the IPP into a number of clearly-defined software elements, -listed as follows: - -\begin{enumerate} - -\item {\bf Pixel Server:} This component is a large data store for all - images used by the IPP, including the raw images from the telescope, - the master calibration images, the reference static-sky images, and - any temporary image data products produced by the IPP. The Pixel - Server is required to meet all of the image storage needs identified - in the top-level requirements above. The Pixel Server must accept - the incoming data and store it until it is no longer needed by other - portions of the IPP. - -\item {\bf Photometry \& Astrometry Database (PnA):} This component is - required to store and manipulate astronomical objects detected in - various images, as identified above, including individual - measurements of objects on the images, the summary information about - those objects, and reference object data. - -\item {\bf Metadata Database:} This component is required to store the - data which is not directly related to images or astronomical objects - as needed to perform the analysis specified above. - -\item {\bf Analysis Stages:} Specific programs are required to perform - the processing steps listed above. These can be divided into - well-defined analysis stages, each of which operates on a particular - unit of data, such as a single OTA image or a collection of - astronomical objets. - -\item {\bf Controller:} In order to perform the analysis stages - required by the IPP, it is necessary to use distributed computing - processes on a large number of computers. The Controller is - required to manage the collection of analysis stages performed on - these machines. - -\item {\bf Scheduler:} This component is a decision-making mechanism - required to guide the operation of the IPP: to evaluate the - currently available collection of data, to identify the necessary - analysis, and to assign the analysis tasks to the Controller. - -\end{enumerate} - -The relationship between these software elements is shown in -Figure~\ref{overview}. This figure also shows the interactions -between the IPP and other Pan-STARRS systems. - -\begin{figure} -\begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/overview.ps}} -\caption{ \label{overview} IPP System Overview} -\end{center} -\end{figure} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\paragraph{Pixel Server} - -The IPP Pixel Server \tbd{rename as Image Server?} is a large data -store for all images used by the IPP. The Pixel Server is required to -store all of the images needed by the IPP for the length of time they -are required; total data volume is specified in detail in the hardware -summary, but is in the vicinity of \tbd{700 TB}. - -The IPP Pixel Server must maintain a record of all images currently -available in the repository \tbd{and all no longer available}. This -record must include the image name, location (which machine), the -state of the image (available, deleted), the image size, the image -type, and the existence and location of secondary copies of the image. -This information need not include other metadata such as the image -summary statistics or the state of the image processing for the image, -as these aspects are included in the Metadata DB. - -The IPP Pixel Server must store images as FITS files on disk. Raw -images from the telescope must be stored as individual OTA images for -each file, with multiple Cell images per file as well as video -sequences from the guide stars. Images of the Static Sky must be -stored in the form of \tbd{triangular segments} to minimize the total -data volume and pixel overlap. - -The IPP Pixel Server must distribute images across a cluster of -machines. The IPP Pixel Server must be capable of honoring requests -to store an image on a specific machine. If such a request cannot be -honored, the IPP Pixel Server must select an appropriate machine and -notify the requesting agent of the new locations. The IPP Pixel -Server must provide a mechanism to maintain multiple (at least two) -copies of each image. - -The IPP Pixel Server must interface with other subsystems of the IPP. -It must provide an interface to other IPP subsystems to identify the -image location (the computer on which it resides). It must provide a -mechanism to serve a specified image to another IPP or Pan-STARRS -subsystem. It must provide a mechanism for deletion of images in the -Pixel Server. It must have a mechanism to accept or retrieve an image -from another Pan-STARRS subsystem, in particular OATS. Communication -of messages between the IPP Pixel Server and other subsystem must be -via \tbd{XML messages} passed via \tbd{some transport}. - -The IPP Pixel Server must accept images at the telescope maximum rate -of 1 full-camera image every 30 seconds. The IPP Pixel Server must -therefore accept notifications and process retrievals at a rate of 64 -raw OTAs in 30 seconds. - -\tbd{O/S, language, SQL, ODBC requirements?} - -\tbd{hardware requirements?} - -\tbd{communication protocols?} - -\paragraph{P\&A Database} - -The IPP requires a mechanism to store data related to astronomical -objects derived from various sources with a variety of associations. -The PnA (Photometry and Astrometry) Database serves this function. -The PnA Database deals with two related concepts: {\em objects} and -{\em detections}. The objects are descriptions of astronomical -objects while the detections are the specific measurements of those -objects on an image. A collection of {\em detections} may be used to -derive average quantities which describe a particular {\em object}. - -The PnA Database must store the collections of detections which were -derived from specific images from any of the analysis stages. It must -be possible to determine and locate (perhaps via interactions with the -pixel server) the image from which a specific detection was derived. -It must also be possible to extract all detections derived from a -specific image. These associations must include descriptive -information including the coordinates of the detection on the image. - -The PnA Database must provide a mechanism to associate together -multiple detections of a specific object. Several major classes of -objects will be present, each of which must be handled correctly. - -First, the most distant stars, compact galaxies, and QSOs will have -nearly fixed locations relative to other nearby stars, with only small -deviations for individual measurements. The association between -multiple detections of such objects must be made on the basis of their -coincident positions. The PnA Database must be able to determine the -average position of the object and the deviations of the individual -detections from that average. - -Second, solar system objects do not have a fixed location and -detections of such objects must associated on the basis of their -coincidence with the orbit of the objects. The PnA Database must be -able to associate detections with the orbits of known objects. The -determination of this association is the responsibility of the MOPS -and must be communicated to the IPP PnA Database on \tbd{some -timescale}. The PnD Database must be able to retrieve the detections -associated with the object and to provide the object associated with -the specific detections. This association must include descriptive -information such as the offset of the detection from the predicted -location of the detection based on the orbit. This functionality is -required to allow the PnA Database to ignore known moving object -detections from other types of queries. - -Third, stars in the general vicinity of the solar system fall in -between these first two classes of objects. Their proper motion and -parallax response is significant enough ($>1$ arcsec in 10 years) that -they are not well-described by an average location and a collection of -offsets. These objects must be described by a distance and a proper -motion vector. The PnA Database must be able to find and associate -detections of objects for which either of the parallax or the proper -motion are substantial. - -Fourth, many detections, especially in their initial states, will not -be associated with a specific astronomical object of any of the above -classes and should be treated as orphans. Some of these will be -suprious (not represent real objects), some will be from solar system -objects for which orbits are not yet determined, some will be from -faint stars near the detection limits, some will be from short-term -transients which have only been detected once. The PnA Database must -be able to carry these detections until they have been associated with -one of the objects above. It must be possible to migrate individual -detections associated with an astronomical object back to the orphan -state. - -For every object, and all orphaned detections, it must be possible to -determine the images for which the coordinates were included but for -which no detection was made. The minimum set of information which -must be carried for these non-detections is the image and the -associated object or orphan. - -The PnA Database must store the relationships between various -photometric systems and, in some cases, the evolution of that -relationship. It must be possible, given a determined set of -calibrations, to convert between the measured instrumental magnitude -of a detection with a specific filter, detector, and telescope, and at -particular time and the implied magnitude in the average Pan-STARRS -magnitude systems. It must also be possible, given the magnitudes of -an object in one system to convert those to the magnitudes in another -system; an example of such a conversion is between the average -Pan-STARRS filter systems and the various reference systems -appropriate for those filters. - -The PnA Database must provide interfaces to extract lists of objects -and detections based on various query parameters. It must be possible -to extract all detections associated with a specific object, all -non-detections of that object (or orphan) and summary statistics from -these collections. It must be possible to extract all objects or -detections within specified spatial regions including regions bounded -by great circles (RA,DEC; GLAT,GLON; ELAT,ELON) and regions described -by a location and a search radius. It must be possible to extract the -image parameters associated with a specific detection including image -coordinates of the detection, exposure time, time and date of the -detection, etc. - -\tbd{volume requirements} - -\tbd{speed / access requirements} - -\paragraph{Metadata Database} - -The IPP requires a Metadata Database to store and provide access to -metadata of various types and from various sources. Metadata in the -context of the IPP represents all data which is not included in the -two data stores discussed above (Images and Detection/Objects). -Metadata is generated at the telescope and during the various analysis -stages - -The Metadata Database must store and provide metadata for all raw -images, for processed images, for the calibration images (both raw and -master), for the extracted object lists. Metadata describing the -environmental conditions at the telescope must also be stored and -provided as needed. - -If analysis results are exchanged via the metadata database, it must -provide access to the queried data on timescales of $<2$ seconds to -avoid slowing down the analysis systems. - -\tbd{volume requirements} - -\tbd{does the description of images belong in the Metadata database or - in the Pixel / Image Server?} - -\tbd{queries} - -\subparagraph{Configuration Database -- a subset of the metadata database?} - -The IPP requires a Configuration Database to store and provide access to -information about the IPP itself. Examples of data in the -configuration database include the default parameters for the various -analysis programs, the description of the computing environment, the -process status information, etc. - -\paragraph{Controller} - -The IPP uses a collection of computers to store and process images and -to manipulate collections of detections. These computers perform any -of a large number of analysis stages or other processing tasks without -significant interprocess communication. It is necessary to have a -mechanism which initiates computing tasks on the different computers, -which monitors the tasks as they are executed, which handles the -output and the errors from these tasks, and which reacts to the -failure of any of the computing nodes. The system responsible for the -tasks in the IPP is the Controller. - -The Controller must interact with the collection of computers under -its management and with other subsystems in the IPP. The controller -must accept a variety of inputs from other subsystems, described -below, and respond accordingly. The controller must also provide -information to other subsystems on demand. - -Computers managed by the controller are allowed to be in one of -several states, and the controller must interact with it in an -appropriate way for each of those states. A computer may be {\tt -alive}, {\tt dead} or {\tt off}. If the computer is {\tt alive}, it -responds to commands from the controller and may be used for tasks -subject to other constraints. If it is {\tt dead}, the computer is -not responsive and must not be used for executing tasks. The -controller must identify computers which have died and occasionally -test them to see if they are {\tt alive} again. Computers which are -{\tt off} are not available for tests and must not be tested. -Computers may be set to the {\tt off} or {\tt dead} states by external -subsystems; it is the responsibility of the Controller to return a -computer to the {\tt alive} state if possible. - -Computers which are in the {\tt alive} state may be in one of two -modes: {\tt busy} and {\tt free}. A computer which is {\tt busy} -currently has a task assigned to it. The controller may only assign -one task to one computer at a time\footnote{A physical piece of -hardware may be defined to the Controller as multiple computers to -allow multi-processor nodes to execute more than one simultaneous -task.}. Computers which are in the {\tt free} state may have tasks -assigned to it. The controller must also manage an additional set of -constraint tables for each machine: the allowed tasks. Each computer -may have a list of allowed tasks which may include {\tt all} tasks, -{\tt none} of the tasks, or specified task names. The controller must -only execute the allowed tasks on a machine. - -The Controller must accept tasks from other IPP subsystems. The task -requests must include the specific command to be executed. The -commands must be in the form of a UNIX command which could be -performed on any of the computing nodes. Any input or output data -structures in the commands must be a valid resource regardless of the -node on which the task is executed. Input and output data resources -must be unique where necessary to avoid conflicts. Tasks must be -given an identifier, which must be returned to the requesting agent, -to be used to control the specific task. - -Task requests may specify a desired node for the task execution. The -Controller must attempt to honor the request if the node is {\tt -alive}, but must execute on another node if the requested one is {\tt -dead} or {\tt off}. Even if a node is {\tt alive} the controller must -choose another node if the specified tasks is not allowed on the -requested node. In all other cases, the controller must wait until -executing processes, and processes with higher priority, are completed -before executing the specified task on the requested node. - -Task requests may specify an urgency level. The controller determines -the priority of the task by sorting first by priority and next by the -sequence of the request. An executing task must be completed before -any new task is started, regardless of priority. Tasks may be -assigned a priority of 0 in which case they are maintained in the -queue and never executed. - -The controller must monitor the output streams from the executing -tasks and the exit status of the tasks. \tbd{where do we send the -output logs?}. The status, including the exit status, of each task -must be maintained for other subsystems to query as needed. \tbd{how -long? on disk / database?} - -The controller must accept commands from other IPP subsystems. These -commands include those which govern the processing of specified tasks, -those which govern the behavior of specific computing nodes, and those -which request information from the controller. The controller must be -able to halt the execution of a specified task, delete an unexecuted -task from the task list, change the priority of tasks, change the -requested nodes for tasks. The controller must also be able to stop -the current execution of a task and push it to the end of the queue -and also change its priority. - -The controller must honor requests (normally from the users) to change -the mode of any computing node on demand between {\tt off} and {\tt -dead}. It must also be able to change the list of allowed tasks as -requested by external commands. - -The controller must respond to informational requests regarding the -collection of machines and their states as well as the collection of -tasks and their states. The controller must monitor the execution -times of the different tasks and provide summary statistics. Finally, -the controller must respond to three top-level commands: {\tt finish}, -{\tt stop} and {\tt abort}. When {\tt finish} is requested, no more -new tasks are accepted on the stack of task, and when all tasks in the -stack have completed, the controller must exit. When {\tt stop} is -requested, the currently executing tasks must be completed at which -point the controller must exit, but tasks remaining in the stack which -have not been started are flushed. When {\tt abort} is issued, the -controller immediately kills all executing tasks and exits. - -\paragraph{Scheduler} - -The IPP is responsible for a variety of analysis tasks: several stages -of processing of the science images; routine assessment of the detrend -images used in processing the science images; construction of -replacement detrend images when needed; generation of astrometric and -photometric reference catalogs based on the collected dataset; and the -performance of test analysis programs. At any point, decisions need -to be made about which of these tasks should be performed, based on an -analysis of the contents of the image database tables, the -requirements of the people monitoring the IPP, and the near-term -observing plans. The IPP Scheduler is a mechanism to manage these -various inputs to guide the decisions and initiate the actions. - -The Scheduler acts as an intermediate between several components of -the IPP and also between the IPP and external agents such as the OATS -system and the users who must monitor the behavior of the IPP. - -The Scheduler must send commands to the Controller for execution. It -is the Controller's responsibility to manage the specific analysis -jobs executing on a given processing node. These analyses may include -the process of copying of moving data from OATS to the pixel server -nodes, or it may involve image processing stages performed on the -science images by the appropriate processing nodes, or it may involve -analysis of the data in the PnA object database. In order to isolate -and encapsulate the responsibilities of the Scheduler and the -Controller, the Scheduler must initiate the tasks which the controller -manages; in this way, the controller does not need to have any -information about the details of the tasks which it executes. -Communication between the Scheduler and the Controller must be -bi-directional; the Scheduler must send tasks to the Controller which -the Controller must inform the Scheduler of the outcome of those -tasks. \tbd{it is not specified whether the scheduler and controller -are components of a single software system or interacting but distinct -software components.} - -The Scheduler must take as input the current list of pending images, -both science and calibration, and a description of the current -observing plan or strategy on some time-scale. The Scheduler must -also take input from humans who manage the IPP. - -The Scheduler must choose between several types of analysis stages -based on the contents of those lists and on the requirements of the -users. The list of tasks which the Scheduler must decide between -includes: -\begin{itemize} -\item moving data to and from the pixel server ($\sim 30$ second timescales) -\item running the science analysis stages ($\sim 30$ second timescales) -\item testing the validity of the current detrend images ($\sim$ - nightly) -\item constructing new detrend images ($\sim$ weekly) -\item updating and improving the photometric and astrometric reference - catalogs ($\sim$ yearly). -\end{itemize} - -The Scheduler must choose between tasks which are relevant on several -different time-scales. The time-scale range from 2 times per minute -to once or twice a year, as noted in the list above. The Scheduler -must also make use of the human input in managing such choices. The -human users must be able to specify that a particular task or set of -tasks is of higher or lower priority than the norm. - -The Scheduler must maintain a set of rules defining the dependency of -one type of analysis stage on other analysis products. For example, -the nightly science image processing depends on the existence of valid -detrend images. The Scheduler must be able to recognize the -dependency and initiate the required analysis needed to perform other -analysis tasks. The Scheduler must have the ability to decide between -postponing an analysis task until the required data are available or -to initiate the task using a lower-quality or less appropriate -substitute. For example, in normal circumstances, a science image -must not be processed until the corresponding detrend frame has been -produced. However, if such a frame is unlikely to appear soon, and -the pressure to process the science image is sufficiently high, then -the frame could be processed with an older detrend frame of known -lower quality. The Scheduler must have the ability to choose the -best, if not ideal, reference data for a particular circumstance. - -The Scheduler is responsible for setting the operating mode of the -IPP. When the IPP is in the automatic operating mode, this implies -that the Scheduler is performing the most appropriate tasks at a -particular time. When the IPP is in the interactive mode, the -Scheduler must perform the requested action regardless of the outcome -of the decision trees. In addition, the Scheduler must only perform -the requested actions and not attempt to perform the other -normally-required actions. The only exception to this exclusion is -that, in the interactive mode, data must still be copied from the -summit system. A human-sent command must be able to change the -Scheduler priorities from the automatic to the interactive modes -\tbd{with a CLI or GUI}. An additional IPP mode is the {\em paused -mode}, intended for tests or maintenance, in which case the Scheduler -does not perform even the data copy tasks. Every task is performed on -demand by the user. - -\subsubsection{Analysis Stages} - -\paragraph{Overview} - -We now consider the collection of analysis tasks which must be -performed by the IPP. These tasks represent the core of the required -IPP functionality; the architectural components discussed above can be -viewed as primarily supporting infrastructure to enable the analysis -tasks to be executed on the appropriate data and to store the results. - -Depending on the task, the basic data unit may be individual images, -collections of images, or derived data products such as a collection of -detections of astronomical objects. Because of the granularity of -these data units, many of the analysis tasks can be performed in -parallel because, for example, the intial analysis of an OTA in one -image does not depend on the results from another OTA. We define the -term `analysis stage' to refer to the largest complete analysis task -which may be performed on a single data item. The analysis stages are -divided into three categories, and further subdivided as follows: - -\begin{enumerate} - \item {\bf Science Image Analysis} is performed on the night-sky - science images to extract the science data from these images. The - science image analysis is divided into 4 phases: - - \begin{itemize} - \item {\bf Phase 1:} The image processing preparation phase, in - which basic astrometric analysis of the complete FPA image is - performed. - - \item {\bf Phase 2:} The image reduction phase, in which the - individual detector images (OTAs) are processed as much as possible - without reference to other chips in the same FPA image or other - exposures. - - \item {\bf Phase 3:} The exposure analysis phase, in which the - results of the multiple detectors are combined to improve the - calibrations for the complete FPA images. - - \item {\bf Phase 4:} The image combination phase, in which several - different exposures of the same part of the sky are combined to - produce high-quality difference and summed images. - \end{itemize} - - \item {\bf Calibration Image Analysis} is required to generate the - calibration images used in the science image analysis. There are - three types of calibration images which are produced. \tbd{make this - consistent with other sections which use the basic / other - calibration distinction} - - \begin{enumerate} - \item {\bf Calibration 1:} The basic master-detrend creation images, - which are constructed from a simple stack of multiple input - calibration images. - - \item {\bf Calibration 2:} Sky-model \& fringe-model images, which - are constructed by combining a collection of images which require - substantial processing before the combination. - - \item {\bf Calibration 3:} Flat-field correction image, which is - constructed on the basis of photometry observations of objects from - certain science images. - - \end{enumerate} - - \item {\bf Reference Catalog Creation} is required by the IPP to - generate improved astrometric and photometric reference catalogs on - the basis of Pan-STARRS observations. - -\end{enumerate} - -Figure~\ref{stages} shows the flow of data between the various IPP -software systems and the different analysis stages, each managed by -the Controller. The thick lines represent the flow of pixel data, the -thin lines represent the flow of metadata and object data, and the -grey lines represent the flow of commands. The hatched systems -represent external PanSTARRS systems (OATS, the Sky Server, the SAIC -Object Database, the Moving/Transient Object Pipeline, and other -Client Science Pipelines. - -The individual analysis stages can be accessed as a UNIX command-line -program. Each command represents the action of the stage on a single -quantum of data. These analysis stages are built of lower-level -C-functions wrapped in a higher-level programming language, -\tbd{Python}. - -The decision to execute a specific analysis stage for a specific -dataset is made by the Scheduler, which sends the infomation to the -Controller. The Controller executes the analysis stage for the data -on an appropriate machine and monitors the success or failure of the -job. - -\begin{figure} -\begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/stages.ps}} -\caption{ \label{stages} IPP System Overview} -\end{center} -\end{figure} - -\paragraph{Science Image Analysis} - -The Science Image analysis stages together represent the basic data -analysis required by the IPP. These analysis stages must process the -images in a timely manner so that the incoming data stream will not -overload the Pixel Server. The required processing time is derived -from the rate at which science images are obtained by PS-1. At a -minimum, the Science Image Analysis must keep up with the average -image rate over the course of 1 day. \tbd{The Science image analysis -is required to process images at the maximum science image rate from -PS-1 of 1 image every 30 seconds -- does this fall out of the science -requirements?} \tbd{In order to give time for uncertainties in the -Pan-STARRS system as a whole, the Science Image Analysis must be able -to process all images from a night within 12 hours.} - -\tbd{number of images per night, data volume per image, output -products} - -The science image analysis which must be performed by the IPP consists -of: - -\begin{itemize} -\item detrending the images to remove the instrumental signature - -\item astrometric and photometric calibration of the individual images - -\item merging a collection of several images of the same portion of -the sky obtained over a short period of time (to remove image defects -and gaps) - -\item subtracting the appropriate reference static-sky image - -\item cleaning the image of any transients - -\item adding the cleaned image to the static sky - -\item object detection of images at specific stages -\end{itemize} - -These analysis steps can be grouped into four phases, each of which -deals with a single data unit. We identify and discuss the -requirements of the four phases below. - -\paragraph{Phase 1 : image processing preparation} - -The Phase 1 analysis stage is performed on each science FPA to -calculate basic astrometric \tbd{and photometric} data needed by the -later stages. Phase 1 must use the static (pre-determined) telescope -distortion model and table of nominal OTA positions and rotations, -combined with the guide star pixel and celestial coordinates, to -determine the correct telescope bore-sight, field rotation and -magnification. The astrometric accuracy required from this analysis -stage is \tbd{2 arcsec} across the field, sufficient to match the vast -majority of reference stars with their detections. - -In some circumstances, science images may have no guide stars. This -may occur if the detectors are not run in OTA mode, especially for -short snapshot images of if IPP is being run on non-Pan-STARRS data. -In such a circumstance, the Phase 1 stage must perform extremely basic -object detection, determining the detector coordinates for stars which -are not excessively saturated and which are significantly above the -background level. The threshold levels for this object detection -stage must be configurable. The object extraction must be performed -in less than \tbd{3 seconds}. - -In order for astrometry of an image to succeed, it is necessary that -approximate image coordinates be known. The Phase 1 analysis must be -able to succeed despite initial coordinate errors as large as \tbd{5 -times} the field width. However, the search process must attempt the -near matches first in the assumption that the given coordinates are -accurate. - -A table of the overlaps between the science image to be processed and -the static sky images must be constructed. This table will be used to -guide the processing of the static sky in Phase 4. The overlaps must -be generously calculated so that small errors in astrometry at Phase 1 -will not cause any valid static sky / science image pairs to be missed -because of the astrometric error at this phase. It is acceptable for -a small number of invalid overlaps to be identified as these will be -excluded in Phase 4. Sky cells which do not have sufficient science -image overlap \tbd{$< 10\%$} need not be processed. - -It is not unusual that an image be obtained with invalid coordinates -or without any valid stars. For example, the telescope control system -may make an error and report the wrong time or coordinates. Or, the -image may be obtained in exceptionally poor conditions with no -detected stars. Phase 1 must fail gracefully in these conditions, -reporting an appropriate error. Such images must be identified for -possible human intervention, or future follow-up after metadata -repairs are made. - -\paragraph{Phase 2 : image reduction} - -The Phase~2 analysis is the detrend stage, in which the images from -the detector are processed to remove instrumental signatures. In -addition, basic object detection is performed along with improved -astrometric and photometric calibration. \tbd{what component selects -the appropriate calibration data? is it the phase~2 program, the -individual modules, or the scheduler above it?} In each step of the -analysis process, an image mask and noise map must be carried and -updated when appropriate. The following operations need to occur -within Phase~2 processing: - -\begin{enumerate} -\item Convolve detrend images with the OT kernel, if available -\item Flag bad and saturated pixels -\item Bias correction via overscan subtraction -\item Trim object image to remove overscan and edges corrupted by OT -\item Correct for non-linearity -\item Flat-field correction -\item Sky subtraction -\item Identify CRs -\item Find objects in the image -\item Make postage stamps of bright objects. -\end{enumerate} - -\subparagraph{Convolve detrend images with the OT kernel} - -Detrend images must be convolved by the OT kernel, so that they -accurately represent the detrend images appropriate for the object -images, which have been shifted using OT. The detrend images which -must be convolved include: the flat-field and the -high-spatial-frequency fringe images. \tbd{Must this be a formal -convolution with the analytical OT kernel, or can it be a convolution -with a decomposed kernel?} The appropriate kernel for each cell of an -OTA must be determined from the guide star history. \tbd{what is the -source of the OT kernel? pixel server?} - -\subparagraph{Flag bad and saturated pixels} - -A static bad pixel mask needs to be used to identify pixels which are -bad. Note that bad pixels which are charge traps need to be grown by -the extent of the OT convolution kernel, while those pixels above a -charge trap (i.e.\ bad colums) must not be grown, since they were not -affected by pixel shifting, but only became bad at read-out. - -Pixels saturated in the A/D converter must also be masked, and this -area must be grown by an additional pixel to mask excess charge -spillover. - -The bad pixel mask must be carried with the science images. Different -bits must be set to identify different reasons for masking the pixel. - -\subparagraph{Bias correction via overscan subtraction} - -The image bias must be subtracted. Since different detectors behave in -different ways, several options for modelling the bias must be -available. The bias must be measured from the image overscan region. -The bias subtraction method must be capable of applying a single -constant to the complete image, or to represent the bias as a function -which varies along the overscan. The function to be used must include -a spline or a chebychev polynomial derived from the data values along -the overscan. The values used to determine both the single constant -or the inputs to the spline and polynomial fits must be derived from -groups of pixels on the basis of one of several statistics, including -the sample and robust mean, median, and modes. In the case of a -single constant, all of the overscan pixel values are used in the -calculation of this statistic. In the case of the 1D functional -representation, the input values to the fit must represent the -coordinate along the overscan, with the statistic derived from the -pixels in the perpedicular direction at each location. Sigma-clipping -on the input data values must be an option. \tbd{accuracy of the bias -subtraction?} - -\subparagraph{Trim object image} - -The image must be trimmed to remove the non-imaging pixels, such as -the overscan and any pre-scan pixels, along with those pixels near the -edges that have been compromised due to OT operation. The definition -of the imaging area of the detector must be determined from the camera -configuration data or from the metadata associated with the image, -with the choice a user-configurable option. - -\subparagraph{Correct for non-linearity} - -If required, the object image (after bias correction) must be -corrected for the effects of non-linearity through a provided -polynomial fit to the pixel data values. The choice to apply the -correction must be set by the user. - -\subparagraph{Flat-field correction} - -The object image (after bias correction and non-linearity correction) -must be corrected for sensitivity variations as a function of -position, dividing by a flat-field image. The flat-field images must -be appropriately normalized (see section \ref{mkcal}). The -flat-fielded image must have a consistent photometric zero-point -across the chip, and across the full FPA, to within 0.2\%. - -\subparagraph{Sky \& Fringe subtraction} - -The flux contribution of the sky (from both continuum emission and the -line emission that causes fringing) must be subtracted from the -flat-fielded object image. The subtraction must remove background -(technically, foreground) variations which are not celestial but -generated in the atmosphere or by more localized scattering. This -background subtraction does not address the artifacts generated by -bright stars: bleeding columns, ghosts, or other localized reflection -effects. The background subtraction must remove the variations with -an accuracy such that the residual variations do not introduce, on -average, more than \tbd{0.2\%} photometric scatter or more than -\tbd{1\%} extremely deviant outlier stars (stars for which the -photometry is in error by more than 3\%). \tbd{what is the -requirement on galaxy photometry? morphology determinations?} -\tbd{What is allowed power-spectrum of background variations?} - -\subparagraph{Identify `cosmic rays'} - -Charged particles in the detector frequently cause features which do -not have the morphology of astronomical objects. In a well-sampled -image, these may be easily identified by the sharpness of the image. -In a near critically-sampled image, these `cosmic rays' may be -indistinguishable from stellar objects. The IPP must have the -capability of making the morphological identification of cosmic rays -if the imaging data is suitable. The identified cosmic rays must be -masked with a configurable growth factor (additional pixels beyond the -detected pixels in the feature). \tbd{The determination if the image -can be treated with morphological cosmic ray rejection must be made by -Phase~2.} - -\subparagraph{Find objects in the image} - -Objects on the flat-fielded object image must be found, and general -parameters, including the object centroid, instrumental magnitude, -local background level, and basic shape parameters ($\sigma_{\rm min}, -\sigma_{maj}$) measured. The detection threshold must be -configurable, and be a function of the average background flux or the -image noise map. Minimal object classification must be performed to -distinguish objects which are consistent with a single PSF, objects -which are inconsistent, and objects which are saturated. The -resulting collection of detected objects must be saved along with the -relevant image metadata (\ie filter, exposure time, etc). - -\subparagraph{Astrometry} - -Objects detected in Phase~2 must be matched with known astrometric -reference objects, using reference object coordinates which have been -adjusted for proper motion. The matched objects must be used to -improve the astrometric solutions for the individual OTAs. At this -stage, a user-defined collection of OTA astrometry parameters must be -fitted on the basis of the matched stars. The Cell astrometric -parameters must not be allowed to vary at this stage. The fit must be -robust, rejecting outlier matches (either stars with poorly determined -proper motion or spurious matches). The resulting astrometric -solution must be consistent across the OTA field to within \tbd{0.2 -arcsec}. - -\subparagraph{Postage Stamps} - -The IPP must have the capability of extracting regions surrounding a -specified subset of objects from the flattened images. These postage -stamp images must be saved for additional use by client science -pipelines. The identification of these objects must be on the basis -of a set of rules applied to the object magnitude and position. - -\paragraph{Phase 3 : exposure analysis} - -The Phase 3 analysis stage works with the results from a complete FPA -obtained during Phase 2 to improve the photometric and astrometric -calibrations. - -Phase 3 must use the objects detected in Phase 2, matched with an -appropriate reference catalog, to determine the image photometric zero -point and zero-point variations across the field. If zero-point -variations are significant \tbd{level TBD}, the zero-point variations -must be modeled with a chebychev polynomial correction of order 3 or -less. The complete FPA image must be categorized as photometric or -not \tbd{numerical scale?} on the basis of the zero-point consistency, -the transparency compared with recent long-term measurements in the -filter, and the external indicators of photometricity. - -Phase 3 must use the objects detected in Phase 2, matched with an -appropriate reference catalog, to determine improvements to the -astrometric solutions. The distortion model appropriate to this image -must be determined. The resulting astrometric accuracy must be -limited by the astrometric reference catalog \tbd{30 mas for USNO?} - -\paragraph{Phase 4 : image combination} - -Phase 4 is the image combination stage, in which multiple images of -the same portion of the sky are merged and confronted with the static -sky image. Phase 4 operates on the smallest data unit of the static -sky, the sky cell, along with the associated pixels from a collection -of images which have been processed through phases 1--3. For each sky -cell, the corresponding pixels are extracted from the exposures being -processed and mapped to the projection of the sky cell. The pixels -from the multiple input processed images are combined into a single, -cleaned image. This image is then confronted with the static sky cell -data to produce a difference image. Residual objects in the -difference image, above a threshold are detected and excised from the -original cleaned image. The remaining pixels are added to the -existing static sky image. Object detection must be performed on the -difference and cleaned images. \tbd{when is static sky object -detection \& classification performed?} Phase 4 naturally divides -into several stages, each of which are discussed in detail below. - -\subparagraph{Extract image pixels} - -For the given sky cell, the corresponding set of image pixels must be -determined and extracted from the input images. This process must use -the astrometric information for each OTA and Cell to determine the -exact overlaps. It must not miss any pixels, and it must read no more -than 20\% more pixels than necessary from the input images. - -\subparagraph{Transform pixel coordinates} - -Pixels which have been extracted from the input images must be mapped -to the corresponding pixels in the sky image. The tranformation must -be based on the measured astrometric solution for the input images -relative to the reference catalog used to generate the static sky -image. This warping must use a locally linear astrometric solution to -minimize computational effort. The output image must maintain be -photometric consistent with the input image to within 0.2\%. -\tbd{interpolation method?} - -\subparagraph{Flux matching} - -The multiple input images must have their object fluxes intercompared -using the stars measured in Phase 2 in order to determine the -appropriate photometry scaling factors needed to properly combine them -photometrically. - -\subparagraph{Image outlier pixel rejection} - -Pixels from the group of images which are inconsistent with the -ensemble of pixel values must be identified and flagged. The -resulting collection of pixels must be used to construct a single -output image, cleaned of the outliers. This outlier rejection must be -performed optionally since moving objects will be rejected in images -obtained over a wide range of times. - -\subparagraph{PSF matching} - -The multiple input images must have their PSF mutually matched to -allow for proper image subtraction. - -\subparagraph{Image Subtraction} - -The static sky image must be subtracted from the stacked, cleaned -image. All objects in the difference image must be detected and the -pixels belonging to variable sources flagged in the input image. -Object detection at this stage is the same as that used for Phase 2. - -\subparagraph{Cleaned Input Image} - -The flagged pixels must be excluded from the input images and a new, -cleaned image constructed. This image must have object detection -applied to it. \tbd{parameters} - -\subparagraph{Update static sky} - -The final, cleaned input image must be added to the static sky so that -an incrementally-deeper static sky image may be made. -\tbd{parameters, weight map} - -\subparagraph{Products} - -Phase 4 must produce the following data products at a minimum: -\begin{enumerate} -\item Subtracted image --- the combined image using each of the -telescopes, with the static sky subtracted; -\item New static sky image --- the combined image using each of the -telescopes, with the (old) static sky added; -\item Metadata about the quality of each of these images; and -\item A catalog of variable sources. -\item A catalog of sources from the combined image. -\end{enumerate} - -\subparagraph{Timing} - -It is required that the {\em total} processing for each exposure by -the Pan-STARRS system not take longer than $n \times T_{\rm min}$, -where $T_{\rm min}$ is the minimum time between exposures (30 sec), -and $n$ is a small positive number. Increasing $n$ results in a -proportionally higher expenditure on CPUs, hence it is strongly -desirable that $n \le 2$. - -Since we envision 4 OTAs (each 4k pixels, square) being processed by a -single CPU, we need Phase 4 to process 64 (input) Mpix in -approximately 30 sec (since Phase 4 is the most intensive, it should -receive the lion's share of the time budget), or 2 (input) Mpix per -second. - -\subparagraph{Accuracies} - -Transformations/mappings from detector to sky must preserve both -photometric and astrometric accuracies: -\begin{itemize} -\item Relative photometric accuracy better than \tbd{0.005 mag} -\item Absolute photometric accuracy better than \tbd{0.02 mag} -\item Relative astrometric accuracy better than \tbd{0.01 arcsec} -\item Absolute astrometric accuracy better than \tbd{0.2 arcsec} -\end{itemize} - -\subparagraph{Robustness} - -It is essential that the static sky image (which may have been -painstakingly accumulated over many months) not be corrupted by adding -in transient sources, or data that is of suspect quality (due, e.g., -to an error upstream in the processing). - -\paragraph{Calibration Stages} -\label{mkcal} - -The Calibration analysis stages may be performed on whatever -timescales are appropriate and necessary to maintain the quality and -relevance of the calibration images. We distinguish two major classes -of calibration images which require significantly different techniques -for their construction. We list the specific calibration images which -must be constructed in the calibration analysis stages. The -requirements for each of these stages are discussed in more detail -below. - -\paragraph{Basic Calibration Stages} - -The IPP must generate basic calibration images using the raw bias, -dark, and flat-field (dome or twilight) images obtained by the -telescope as the input. The analysis of these images requires -relatively simple stacking of the input set of images. Outlier -rejection, both of complete input images as well as pixels within the -input stack, must be performed. In addition, each type of image -requires an appropriate normalization which may depend on the data -levels in other detectors in the input set. Each of these calibration -stages must be able to determine from the input stack if the relevant -calibration image needs to be updated and perform an initial test to -see which input images are consistent and valid. - -\subparagraph{bias images} - -Bias images may be needed to correct for structure in the bias. The -IPP must have the capability of constructing a master bias image from -a stack of raw bias frames. The input bias images, representing -offsets from the overscan level, must have the overscan removed, -including 1D structure if needed. The bias construction must -incorporate outlier image and outlier pixel rejection. The statistic -used to determine pixel values must optionally be derived from the -sample mean, median, and mode, robust mean, median, and mode, and the -clipped mean and median. Residual images, in which the master bias is -applied to the input images must be constructed and their statistics -used to exclude any significant outlier input images. - -\subparagraph{dark images} - -Dark images may be needed to correct for structure in the dark -current. The IPP must have the capability of constructing a master -dark image from a stack of raw dark frames. The input dark images -must first be corrected for the bias using whatever method is -appropriate for the science images. The master dark frame must be -specified for a particular exposure time. As such, the input dark -frames must have a limited range of exposure times. The dark frame -construction must incorporate outlier image and outlier pixel -rejection. The statistic used to determine pixel values must -optionally be derived from the sample mean, median, and mode, robust -mean, median, and mode, and the clipped mean and median. Residual -images, in which the master dark image is applied to the input images -must be constructed and their statistics used to exclude any -significant outlier input images. \tbd{The dark frames must be -examined to determine the non-linearity of the measured dark current --- by what component?}. - -\subparagraph{flat-field images} - -Master flat-field images must be constructed from a collection of -input flat-field images. An appropriate set of input images must be -selected on the basis of their flux levels, time of observations, and -the observing conditions. The input flat-field images must be -processed (bias and dark corrected if needed) and the resulting images -stacked. The master flat-field construction must incorporate image -and pixel outlier rejection. The statistic used to determine pixel -values must optionally be derived from the sample mean, median, and -mode, robust mean, median, and mode, and the clipped mean and median. -Residual images, in which the master flat-field image is applied to -the input images must be constructed and their statistics used to -exclude any significant outlier input images. - -\paragraph{Other Calibration Stages} - -\subparagraph{mask images} - -Initial bad-pixel mask images must be generated on the basis of -comparison between raw flat-field images and a cleaned, stacked -master. The mask creation analysis stage must accept a collection of -flat-field images and identify pixels which are repeatedly -inconsistent from image to image. If too many pixels are -inconsistent, an error must be raised. - -\subparagraph{fringe frames} - -Fringe-correction frames must be generated to remove the fringe -pattern caused by thin-film interference in the top layers of CCDs, -particularly in the redder passbands. Fringe correction frames must -be constructed on the basis of observations of the night-sky in the -appropriate filters. The images must first be flattened to remove the -pixel-to-pixel sensitivity variations of the detector. The -combination of multiple input fringe frames may not be simply stacked -since the amplitude of the fringe pattern varies independently of -other variations in the image. The amplitude of the fringe frames -must be measured and the images scaled to normalize the fringe -amplitude to the range -1 to +1 before combining with one of the -standard combination statistics (mean, median, mode, etc). - -\subparagraph{low-k sky models} - -Large-scale background structure in images which is not caused by -thin-film interference must also be detected and corrected. Models of -this background structure may be the necessary input to the correction -proceedure. The IPP must have the capability of generating image -models of the large-scale structure patterns observed with the -telescope. \tbd{discuss principal components, SVD?} - -\subparagraph{Flat-field correction frame} - -Flat-field images, whether constructed from the dome, twilight, or -night-sky images, rarely will perfectly correct the detector response -in a consistent fashion across the full field of the camera. The IPP -must have the capability of generating flat-field photometric -correction frames on the basis of the measured photometry of objects -which are placed at a variety of locations on the detector in a -sequence of images. - -\subparagraph{Non-linearity correction frames} - -The IPP must have the capability of constructing non-linear correction -frames. These frames are constructed from exposures of a uniform -source with a range of exposure times. The non-linearity correction -frames provide polynomial correction coefficients as a function of -pixel to convert the observed pixel counts to the expected pixel count -from a linear detector. - -\paragraph{Reference Catalog Creation} - -For PS-1, one of the primary goals is the creation of photometric and astrometric -reference catalogs for the general community and for the future -Pan-STARRS requirements. The generation of these catalogs is -inherently a research project, and will require human control and -intervention. The IPP will be required to provide the data access, -manipulation and visualization tools needed to construct these -reference catalogs and to assess their quality. In this section, we -list the requirements of the tools needed for this effort. - -\paragraph{Astrometry Reference Creation} - -The existing astrometric reference catalogs are known to have -limitations at the level of \tbd{NN} milli-arcsec. The internal -accuracy of the Pan-STARRS dataset can potentially be much higher than -the external reference catalogs. The IPP must have the capability of -generating an astrometric reference on the basis of the observations -obtained by the PnA survey. The IPP must provide the analysis tools -needed to generate the master astometric reference catalog. Much of -the required functionality is covered by the PnA Database. - -The necessary ingredients for the construction of the PS-1 Astrometric -Reference Catalog are: the observed coordinates of stars and the -existing astrometric reference catalogs. A variety of reference -catalogs will be required, including: -\begin{itemize} -\item Hipparcos -\item Tycho2 -\item UCAC -\item YBx -\item USNO-Bx -\item 2MASS -\end{itemize} -These catalog must be available and accessible to the PnA Database. -It is necessary to have the tools to convert the reference catalog -object coordinates to all of the possible coordinate frame of -relevance in the telescope and camera system, including: -\begin{itemize} -\item Catalog to mean positions -\item Mean to apparent positions -\item Apparent positions + pointing to focal plane coordinates -\item focal plane to specific detector (OTA) -\item specific detector to detector cell -\end{itemize} - -In addition to the reference catalogs, it will be necessary to -determine and have available for the analysis system a variety of -approximate calibration data, including the telescope and camera -optical distortion models and the variation of the image PSF across -the camera field, as a function of color. - -The final ingredient in the astrometry solution is the observation of -stars with the PS-1 telescope. The object detections are produced by -several of the analysis stages discussed in the Science Analysis -section. The likely measurement of relevance to the astrometric -reference catalog is the object extraction for the individual, -detrended images (section~\ref{foo}). \tbd{is it necessary to have - multiple centroiding methods available?}. The detected objects must -be matched against the reference catalogs, and it must be possible to -determine fit coefficients as a function of simply position, or with -combinations of magnitude or color. The fitting method must include -robust outlier rejection. It is also necessary to have information -about the objects which are detected in the catalog, but not the -science image or vice-versa, as well as an assessment of the -centroiding errors for each object. It must be possible to plot the -fit residuals against a wide variety of parameters, including the -object positions, magnitudes, colors, etc, and to make subset -selections of the objects on the basis of these parameters. . - -An alternative measurement of the stellar positions is derived from -the guide stars, which are much brighter than the typical saturated -stars. It must be possible to compare the coordinates of the guide -stars with the coordinates of the other stars in the image. It must -also be possible to perform the various fitting steps for the guide -stars rather than for the normal image data. - -\paragraph{Photometry Reference Creation} - -The IPP must provide the analysis tools needed to generate a master -photometric reference catalog. The tools needed for generation of the -photometric reference catalogs are similar in essence to those used -for the astrometric reference catalog. It is necessary to confront -the observed objects against the existing reference catalogs to -determine the necessary calibrations. Again, much of the required -functionality is covered by the PnA Database. - -The necessary ingredients for the construction of the PS-1 Photometric -Reference Catalog are: the observed magnitudes of stars and the -existing photometric reference catalogs. A variety of reference -catalogs will be required, including: -\begin{itemize} -\item SDSS -\item CFHT-LS standards -\item Landolt -\item etc -\end{itemize} -These catalog must be available and accessible to the PnA Database. - -The final ingredient in the photometry solution is the observation of -stars with the PS-1 telescope. The object detections are produced by -several of the analysis stages discussed in the Science Analysis -section. The likely measurement of relevance to the photometric -reference catalog is the object extraction for the individual, -detrended images (section~\ref{foo}). It is necessary to have the -tools to convert between different photometric systems, including: -\begin{itemize} -\item instrumental to nominal detector magnitude -\item nominal detector magnitude to average filter system -\item average filter system to reference photometry system -\end{itemize} -These transformations are based on a set of measured coefficients for -the color and airmass dependency of the measurement. In addition to -these types of transformations, it is necessary to have the ability to -measure and apply relative photometry corrections. - -The detected objects must be matched against the reference catalogs, -and it must be possible to determine fit coefficients as a function of -airmass, magnitude, color and detector coordinates, or with -combinations of the above. The fitting method must include robust -outlier rejection. It is also necessary to perform exclusions on the -basis of object locations, instrumental magnitudes, observed and -reference errors, and in particular time of the observations. It must -be possible to plot the fit residuals against a wide variety of -parameters, including the object positions, magnitudes, colors, etc, -and to make subset selections of the objects on the basis of these -parameters. . - -An alternative measurement of the stellar positions is derived from -the guide stars, which are much brighter than the typical saturated -stars. It must be possible to relate the magnitudes of the guide -stars with the magnitudes of the other stars in the image. It must -also be possible to perform the above fitting steps for the guide -stars rather than for the normal image data. - -\subsubsection{Modules} - -In order to encapsulation functionality, the analysis stages are -constructed of a sequence of steps. The analysis stages consist of a -\tbd{python} script which executes a sequence of C-level functions. -The C-level functions called by the \tbd{python} script are called -{\em modules} and represent basic data analysis operations. - -The required set of Pan-STARRS modules and their functionality is -specfied in the document `Pan-STARRS Image Processing Pipeline Modules -Supplementary Design Requirements' (PSDC-430-xxx), and details of -specific apgorithms are specfied in the document `Pan-STARRS Image -Processing Pipeline Algorithm Design Document' (PSDC-430-006). - -\subsubsection{PanSTARRS IPP Library} - -In order to facilitate testing and development, and to encourage -flexibility, the IPP will be built in a layered fashion. The lowest -level functions will be written in C and collected together into a -Pan-STARRS library, \code{PSLib}. - -The Pan-STARRS Data Library will consist of C structures describing -the basic data types needed by the IPP and C functions which perform -the basic data manipulation operations. The library is organized into -four topics: System Utilities, Basic Data Collections, Data -Manipulation, and Astronomy-Specific Functions. - -The required functionality of the Pan-STARRS Data Library is specified -by the document `Pan-STARRS Image Processing Pipeline Library, -Supplementary Design Requirements' (PSDC-430-007), and details of -specified algorithms are specified in the document `Pan-STARRS Image -Processing Pipeline Algorithm Design Document' (the ADD; -PSDC-430-006). - -\subsubsection{Data Sources and Formats} - -\paragraph{Image Formats} - -FITS images - -\paragraph{Table Formats} - -FITS tables - -\paragraph{Other Data Formats} - -XML files - -\paragraph{External Catalogs} - -\begin{itemize} -\item Hipparcos -\item Tycho2 -\item HST-GSC -\item USNO-A -\item UCAC -\item 2Mass -\item USNO-Bx -\item YBx -\end{itemize} - -\paragraph{Analysis Reference Data} - -\begin{itemize} -\item Telescopes -\item Cameras -\item Detectors -\item Filters -\item software basic parameters -\end{itemize} - -\paragraph{Installation Reference Data} - -\begin{itemize} -\item computers -\end{itemize} - -\subsection{External Interfaces} - -\subsection{Internal Interfaces} - -\subsection{Internal Data Requirements} - -\subsection{Computer Hardware} - -\subsubsection{Overview} - -This section discusses the Pan-STARRS Image Processing Pipeline (IPP) -PS-1 hardware requirements. The hardware requirements addressed in -this section consist of: - -\begin{itemize} -\item Total Disk Volume -\item Total Processing Power -\item Sustained Switch Bandwidth -\item Sustained Node Network I/O -\item Sustained Disk I/O -\end{itemize} - -Even without the complete IPP design, it is possible to identify the -major drivers on the hardware requirements. The total disk volume -requirements are dominated by the need to store raw images for a -certain period, the need to store calibration images for a longer -period, and the need to store the static sky images. Of the various -analysis stages, Phase 2 and Phase 4 present the most significant -demands in terms of data I/O throughput on the network. Phase 2 and -Phase 4 also present the most significant CPU demands. In this -discusion, Phase 2 refers to the per-OTA image pre-processing in which -the instrumental signature is removed and a first pass object -detection is performed. Phase 4 refers to the multiple OTA -combination in which the pre-processed images are merged and combined, -in both addition and subtraction, with the static sky image, and up to -three object detection passes are performed. - -This document does not address the hardware requirements implied by -Phase 1 or 3, nor the load required by the calibration or reference -catalog creation stages. In the first instance, the operations are -only performed on the metadata and are extremely minimal both in terms -of data I/O and computation requirements. In the second case, the -processing is less time critical than the per-image processing and is -performed only infrequently (once per night to once per week, month or -year). \tbd{The software implementation for metadata storage (RDBMS, -FITS tables, etc) will have a very large impact and will be evaluated -along with the needed hardware at a later date.} - -We will address the various hardware requirements by referring to an -assumed data processing and data organization scenario. The -organization of the data and certain aspects of the data processing -scheme have very large implications for the hardware requirements. In -this analysis, we assume that data types are chosen to minimize the -data volume and that the data is organized to minimize the I/O -bandwidth needs, as defined below. We address the data requirements -of the single-telescope Pan-STARRS-1 scenario based on the Design -Reference Mission \tbd{REF}. - -\subsubsection{Data Organization} - -The IPP hardware system must provide both data storage and -computational resources. The IPP requires relativley large amounts of -data storage space, primarily for the image data. Image data is -organized in two categories. First, there is the per-OTA data -- data -associated with specific OTAs, including the raw images, the -calibration images, and temporary processed images at various stages. -Second, there is the data associated with the static sky imagery, -which is in turn organized into smaller sky-cell units. The first -assumption we make is that the hardware is organized into nodes which -provide both data storage and computational resources. The second -assumption we make is that the data storage nodes are divided into two -classes: those which deal with the per-OTA data and those that provide -the static sky storage. In addition, we assume that the computational -tasks related to Phase 2 take place on the per-OTA storage nodes and -the Phase 4 computation takes place on the static sky storage nodes. - -Figure~\ref{hardware} shows our basic concept for the hardware -organization for the IPP. This diagram shows the two types of compute -nodes: OTA-level processing and storage nodes (dominated by Phase 2) -and static sky processing and storage nodes (mostly Phase 4). Also -shown are two switches used in this configuration; although it is -currently possible to buy a single switch with sufficient number of -ports, this organization represents a minimal configuration for the -PS-1 IPP hardware. In such a case, the interswitch communication must -also meet the required throughput needs. We discuss the hardware -requirements in the assumption that such an organization will be -necessary. - -The way in which the images are distributed among the storage and -compute nodes will largely determine the I/O bandwidth requirements. -For data bandwidth requirements calculations, it is necessary to make -some assumptions about the data organization. We make the assumption -that the OTA data is optimally distributed to the OTA nodes such that -the OTA processing is always on a machine with local OTA data. This -implies that all OTA data from a specific OTA are targetted to a -specific machine. (see below for discussion of data duplication). - -A second factor which will have a significant impact on the I/O -requirements is the image storage format for the processed and -calibration images. We have two basic choices: 32 bit floating point -format or 16 bit integer format with appropriate scaling. In the -former case, additional dynamic range is retained, while in the latter -case, we reduce the data volume by a factor of 2. Since the science -requirements for PS-1 do not specify a need for dynamic range greater -than 16 bits, we assume all images are stored as 16 bit data. - -A third determining factor is the number of calibration images needed -by the processing system. Since the complete analysis is not yet -defined, this number is difficult to ascertain. However, we can make -a reasonable guess at the total number for scaling purposes. We -assume that each frame requires a total of 4 calibration frames on -average - -\begin{table}[b] -\begin{center} -\caption{Data Storage Requirements \label{storage}} -\begin{tabular}{lrrrr} -\hline -\hline -Raw data & 200 TB \\ -static sky & 256 TB \\ -calibration frames & 5 TB \\ -metadata db & 0.3 TB \\ -object db & 4 TB \\ -\hline -total & 116 TB \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Data Storage Requirements} - -The Pan-STARRS IPP data storage requirements may be divided into five -principal areas: raw image data, static sky image data, master -calibration images, the metadata database, and the object database. -We discuss each of these data items and their impact on the data -storage requirements for the IPP for PS-1. Table~\ref{storage} -summarizes the data storage requirements in the different scenarios. - -\paragraph{Raw Data Storage} - -There are two basic image types which will be acquired: night-time -science images and calibration images. The night-time science images -consist of 1Gpix per image, or 2GB in raw format. At nominal cadence, -the PS-1 telescope can obtain images at a sustained rate of 1 image -per 30 seconds for the entire night of 10 hours (36000 seconds). A -total of 100 calibration images per night would be a substantial -overestimate of the typical expectation. Combining these numbers, we -can expect to receive a total of 1300 images, or 2.6 TB of data per -night. The total data storage requirements for the raw data are -governed by the number of nights' worth of data we are required to -keep online. \tbd{for the first year, we are required to keep all -images from the PnA and IPV surveys. This amounts to a total of 200 -TB of data}. - -\paragraph{Static Sky Data Storage} - -The static sky is represented by images with 0.2 arcsec per pixel. -There will be one summed image and one weight image for each of the -\tbd{6} filters, each stored with 16 bits of resolution, for a total -of 24 bytes per sky pixel. At this resolution, there are 324 Mpix per -square degree, and we will observe a potential total area of 30,000 -square degrees. Allowing for 10\% overage for overlapping tiling, we -require a total of 10.7 Tpix to cover the sky once, or a total of -$\sim 256$ TB to maintain a single image of the static sky in all 6 -filters. - -\paragraph{Calibration Frame Storage} - -The possible required calibration frames consist of the bias, dark, -and mask images, along with one flat, one flat-correction, and -multiple sky/fringe library frames per filter. In fact, not all types -are needed at all stages. It is very likely that we will not require -bias or dark images, and mask images may be represented by a single -byte per pixel. Nonetheless, it is necessary for us to generate and -store all master calibration frames at least until we prove that they -are not needed. We assume a total of 21 calibration images are -necessary (one flat, fringe, and sky per filter, along with a bias, -dark, and mask). If we intend to keep all master calibration frames -for the project lifetime, and generate a new master on a weekly basis -(a reasonable time-scale), then we can expect to require a total of 5 -TB of calibration image by the end of the 2 years of PS-1. We note -that this is likely to be a drastic overestimate as we are unlikely to -need to regenerate all master calibration frames on a weekly -time-scale. - -\paragraph{Metadata Database Storage} - -The metadata data storage requirements are driven by the need to store -the data for the project lifetime. There are two types of metadata -generated at the summit: data associated with images and environmental -data. The environmental data consists of measurements on a regular -cadence, roughly 1 per minute, of a variety of parameters. We suggest -an expected of 1kB per entry, for a total of 1 GB over the two-year -term of PS-1. The additional systems, such as the DIMM, SkyProbe, NIR -Sky Camera, and the LRProbe will have higher data requirements, but -should be considered as separate, self-contained systems. Their data -products are distilled to a limited number of parameters per minute -which are included in the 1kB given above. Furthermore, items such as -guide-star history, if saved, will be saved with the image data and -represents only a small fraction of the total image data volume. Some -subset of the telescope diagnosic information may be a high volume -data product as well, but only retained by the telescope control -system for the purpose of diagnostic studies. Such data will be -excluded from this analysis. - -The image metadata consists of values associated with the FPA (1), the -OTAs (64), and the Cells (4096). Aside from the guide star history, -the total data requirements for each of these entries will be scaled -by the number of bytes required for the metadata from each data level. -Clearly, if the Cell entry is allowed to be large, it will dominate -the total Metadata data volume. We suggest an expected number of 64 -bytes per Cell, 256 B per OTA, and 1k per FPA, yielding a total -metadata volume per exposure of roughly 0.3 MB, completely dominated -by the Cell metadata. With the exposure rates above, we find a total -of metadata volume of 0.3 TB over the two-year term of PS-1. - -\paragraph{Object Database Storage} - -The hardware requirements for the IPP object database are rather -flexible: the total volume depends critically on the depth to which -the object detection analyses are performed (and thus the total number -of object detections) and the number of object parameters which are -measured. We can make very rough estimates that the total number of -detections over the 2 year lifetime of the project may be in the -vicinity of $10^{11}$. We can conservatively estimate the number of -bytes needed to represent each detection as 128 B, resulting in a -total data storage for the object detections of 12 TB. However, this -number depends strongly on the timescale for which the IPP is required -to maintain all object detections, and may potentially be -significantly reduced. - -\subsubsection{CPU Requirements} - -Phase 2 and Phase 4 dominate the processing requirement, primarily -because they must keep up with the image delivery rate of 1 per 30 -seconds. We have performed benchmarks of a demonstration version for -both the Phase 2 and Phase 4 analyses. - -For the Phase 2, a substantial fraction of the processing time is -consumed by the need to perform FFTs on the images in order to -convolve them with the guide-star kernel, and in the smoothing used -for the object detection process. Additional processing time is -needed by the object detection, deblending, and analysis. Experiments -with the FFTW package show that FFTs may be performed on Intel -processors at rates of approximately 0.25 GHz-sec / Mpix for data sets -of order 1 Megapixel. The FFTs required for the Phase 2 analysis are -performed on the 512$^2$ pixel cells, so these numbers may roughly be -scaled linearly to determine the total time required for OTA -processing. A single FFT on a full OTA, with 64 Cells, therefore -requires roughly 4 GHz-sec. For the full Phase 2 analysis, there are -roughly 4 single direction FFTs required excluding those associated -with object detection; thus the total processing time for these FFTs -is approximately 16 GHz-sec. The addtional analysis steps, excluding -object detection and characterization, account for a small fraction of -this compute time, which we estimate at 10\%. The object detection -stage depends somewhat on the depth to which the analysis is -performed, and the number of measurements made per object. Typical -analysis performed by the Sextractor routine, which performs a -substantial number of per-object analyses, requires 27 GHz-sec for a -full OTA, including the FFTs used for smoothing. We can therefore -assume a total of 50 GHz-sec per OTA for the Phase 2 processing. This -converts to a total of 12800 GHz-sec for a complete major frame. - -For Phase 4, the main computational tasks are combining the multiple -images, with cosmic-ray rejection, and performing the object detection -tasks. Nick Kaiser has done tests of the Phase 4 image combine and -rejection stages, and finds a total processing time of roughly 96 -GHz-sec for a full stack of 4 OTA images. If we add in an additional -34 GHz-sec for detailed object detection and image differencing, we -find a conservative estimage of 130 GHz-sec for a 4-image OTA stack, -equivalent to 7800 GHz-sec for a major frame. - -For PS-1, the typical time for a major frame is $4 \times 30$ seconds. -Some reduction in the load may be gained by reducing the complexity -and depth of analysis for PS-1. Depending on the details and depth of -the analysis, we may reduce the computational load by a factor of 2. - -\begin{table} -\begin{center} -\caption{Data I/O (MB per OTA or Sky-cell) \label{scenarios}} -\begin{tabular}{lrrrr} -\hline -\hline -{\em Phase 2 input} \\ -from summit & $2 \times 32$ MB \\ -input image & {\bf 32 MB} \\ -calibration & {\bf 4 $\times$ 32 MB} \\ -mask image & {\bf 8 MB} \\ -\hline -network I/O: & 64 MB \\ -disk I/O: & 176 MB \\ - & \\ -{\em Phase 2 output} \\ -output image & {\bf 32 MB} \\ -output mask & {\bf 8 MB} \\ -image to P4 & $1.5 \times 32$ MB \\ -mask to P4 & $1.5 \times 8$ MB \\ -\hline -network I/O: & 60 MB \\ -disk I/O: & 40 MB \\ - & \\ -{\em Phase 4} & \\ -input images & $1.5 \times 4 \times 32$ MB \\ -input masks & $1.5 \times 4 \times 8$ MB \\ -static sky & 32 MB \\ -static weight & 32 MB \\ -\hline -input: & 304 MB \\ -output: & 96 MB \\ -\hline -\multicolumn{5}{l}{\em Bold-faced entries are access to local-disk} \\ -\multicolumn{5}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\ -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Per-Node I/O Requirements} - -Data I/O per node is defined as the number of bytes per second passed -through the node's network adapter. The data throughput for each node -depends strongly on the how the data is organized and processed. In -this section, we identify the data which is passed between nodes for -the two stages of the science analysis process. Table~\ref{scenarios} -lists the per-node data I/O for the analysis stages. - -For PS-1, there are 120 seconds of compute time allowed for each of -the Phase 2 and Phase 4 analyses for the collection of four images -which makes up a cannonical major frame. We use the data I/O volumes -and some assumptions about expected network and disk bandwidth to -estimate the I/O and processing timeline for the four scenarios. From -this analysis, we can judge the total CPU requirements in terms of -GHz, not just GHz-sec. We have assumed that GigE network adapters are -capable of delivering data at 50MB/sec sustained and that a disk RAID -can deliver sustained 100 MB/sec reads and writes. These numbers are -conservative estimates based on recent tests discussed below. Using -these assumptions, Table~\ref{throughput} lists the time allocations -for the processing stages. - -\paragraph{Phase 2 Node I/O Requirements} - -In the assumed data distribution scenario, there is a single CPU -allocated to each OTA in the OTA farm and a single CPU for each Sky -cell process. In addition, all data for the specified OTA are stored -on local disks attached to the same computer as the CPU, with the -result that all Phase 2 I/O is made to a local disk. For each science -OTA image which is observed, each OTA node will read from the network -a total of 2 raw images (one for the original image, one for a backup -copy) and write an average of roughly 1.5 processed images and masks -to the Phase 4 machines for a total of 124 MB of network I/O. During -the processing stage, the OTA node will read from disk a total of 176 -MB (4 calibration frames at 32 MB each, one 16 MB mask, and one raw -science image at 32 MB) and write a total of 40 MB (one processed -image at 32 MB and one mask at 8 MB). Given the assumptions for the -network and disk bandwidths (50 MB/s and 100 MB/s respectively), the -data volumes imply a total I/O period of 4.6 seconds. In this -instance, the network I/O is presumed to be sequential with the disk -I/O. - -\paragraph{Phase 4 Node I/O Requirements} - -Although it is easy to arrange the OTA data in such a way that the -majority of I/O is performed locally, it is not as easy to arrange -this for the Static Sky data used by the Phase 4 analysis. We -therefore make the assumption that the Phase 4 analysis will require -all input OTA data to be loaded across the network, as well as all -Static Sky data. This is somewhat of an overestimate as some of the -Static Sky data will be processed by machines with the data stored -locally, and clever Static-Sky data organization schemes can enhance -this chance. - -In the Phase 4 analysis, the images from the 4 separate telescopes are -combined into a single image, confronted with the appropriate segment -of the static sky, with output difference image and updated static sky -image. If we restrict input access to the individual OTA cells, the -maximum read overhead is 50\% (need to read a 10x10 set of cells for -an 8x8 input image). If the processing is performed on Static Sky -segments equivalent in size to the OTAs, the total volume of input -data per node is 304 MB (192 MB of processed science image, 48 MB of -input mask, 32 MB of static sky image and 32 MB of static sky weight -map) while the output data is 96 MB (32 MB static sky, 32 MB weight -map, and 32 MB difference image). Thus, we require a total of 400 MB -network I/O, which implies an I/O period of 8 seconds. - -\begin{table} -\begin{center} -\caption{Data Throughput \label{throughput}} -\begin{tabular}{lrrrr} -\hline -\hline -Phase 2 per-node network I/O & 2.2 s \\ -Phase 2 per-node disk I/O (read) & 1.3 s \\ -Phase 2 per-node disk I/O (write) & 1.2 s \\ -Phase 2 CPU total & 25 s : 128 GHz \\ -Phase 4 per-node I/O & 8 s \\ -Phase 4 CPU total & 112 s : 70 GHz \\ -Phase 2 switch load & 264 MB/s \\ -Phase 4 switch load & 215 MB/s \\ -Phase 2 to Phase 4 switch load & 160 MB/s \\ -Summit to Phase 2 switch load & 70 MB/s \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Switch I/O Requirements} - -The switch I/O requirements are defined by the total number of bytes -per second serviced by the two switches in the system. - -The Phase 2 network I/O is 124 MB per OTA. With 64 OTAs per image, -and 30 seconds average between images, this implies a total of 264 -MB/s switch bandwidth. The Phase 4 network I/O is 400 MB per sky -cell. With 64 cells and 120 seconds between major frames, this is an -average switch bandwidth of 215 MB/s switch bandwidth. The total -switch-to-switch load is 304 MB per OTA, with an average timescale of -120 seconds. With 64 OTAs, this corresponds to 160 MB/s. The -summit-to-Phase 2 switch load is 70 MB/s. - -\begin{table} -\begin{center} -\caption{Hardware Throughput Tests \label{existing-hardware}} -\begin{tabular}{lrrrr} -\hline -\hline -Test & where \& when & model & result \\ -\hline -node I/O & CFHT 11/2002 & Intel 1000 Gigabit & 35 - 40 MB/s sustained \\ -node I/O & CFHT 2/2004 & Intel 1000 Gigabit & 65 - 70 MB/s sustained \\ -RAID write & CFHT 2/2004 & 3ware RAID cntl + IDE & 110 MB/s sustained \\ -Switch Load & VeriTest & Cisco & 3 GB/s (for 32 ports) \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Existing Hardware Throughput} - -We have collected a few representative tests of various pieces of -modern hardware to give a reference for the throughput capabilities. -A number of hardware configurations have been tested at CFHT for the -Elixir project, and we include here their recent reported hardware -RAID-5 I/O speeds and GigE card speeds. We also have included data -from VeriTest studies of Cisco switch throughput, commissioned by -Cisco for a 32 port GigE switch. These tests are summarized in -Table~\ref{existing-hardware}. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Test Verification} - -A testing regime must be implemented to demonstrate the working state -of the provided software. Certain tests as specified must be -performed by MHPCC, with code release contingent on success. Other -specified tests will be performed by IfA to verify the validity of the -implemented algorithms. The tests include: software configuration -tests, software integrity tests, basic unit tests, and detailed -functional analysis. - -\subsection{Software Configuration Tests} - -MHPCC must test the validity of the software configuration, -specifically to check that the code can be compiled on the specified -platforms and that the compilation produces no errors or warnings, -except as noted and allowed. - -\subsection{Software Integrity Tests} - -MHPCC must test the integrity of the software, specifically to check -that the code does not produce memory leaks, segmentation faults. - -\subsection{Basic Unit Tests} - -MHPCC must perform basic unit tests with sample input data and known -output results, including invalid input data to test error handling. -These tests must exercise the complete range of module options. - -\subsection{Detailed Functional Analysis} - -IfA must perform detailed tests with a wide range of input data and -compare the results with existing software system. - -\subsection{Test Verification Matrix} - -\subsubsection{Pan-STARRS IPP Library} - -See Appendix A \& B of the IPP Library SDR (PSDC-430-007) for the test -verification matricies for the Pan-STARRS IPP Library - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Appendices} - -\bibliographystyle{plain} -\bibliography{panstarrs} -\end{document} - -Requirements Trace Matrix - -active state \ref{req:active-state} -paused state \ref{req:paused-state} -interactive state \ref{req:interactive-state} - -system capabilities - -C for source code \ref{req:languages} -Python for scripts \ref{req:languages} - -SWIG interfaces -C APIs - -POSIX -Pan-STARRS Coding Standard - -Naming Conventions -