Index: /trunk/doc/design/ippSRS.tex =================================================================== --- /trunk/doc/design/ippSRS.tex (revision 1083) +++ /trunk/doc/design/ippSRS.tex (revision 1084) @@ -1,3 +1,3 @@ -%%% $Id: ippSRS.tex,v 1.5 2004-06-22 08:35:17 eugene Exp $ +%%% $Id: ippSRS.tex,v 1.6 2004-06-24 20:24:27 eugene Exp $ \documentclass[panstarrs]{panstarrs} @@ -14,5 +14,5 @@ % allow paragraphs to be listed in TOC for now -\setcounter{tocdepth}{4} +\setcounter{tocdepth}{3} \begin{document} @@ -28,4 +28,9 @@ DR.05 & 2003.05.24 & Incorporating comments from internal review \\ \hline \RevisionsEnd + +\TBDsStart +% section page TBR number Description +section & page & TBR & description \\ \hline +\TBDsEnd \tableofcontents @@ -99,10 +104,4 @@ \subsection{Science Requirements} \label{req:system-capabilities} - -\tbd{distinguish data products in commissioning, during AP survey, -after AP survey} - -\tbd{define astronomy concepts: cleaned, pre-processed, significant, -outlier} The IPP must perform the following tasks: @@ -171,6 +170,6 @@ variations in the atmosphere across the field, and the reference catalogs. The photometric scatter in photometric weather must be - better than \tbd{5 millimag} for relative photometry (relative to the - internal filter system) and \tbd{10 millimag} for absolute photometry + better than \tbr{5 millimag} for relative photometry (relative to the + internal filter system) and \tbr{10 millimag} for absolute photometry (relative to other filter systems such as the SDSS filters). @@ -218,10 +217,17 @@ \label{req:active-state} -In active state, the IPP must accept images and metadata from the -external sources (i.e., the summit) and automatically perform the -complete set of image processing tasks, including both calibration and -science image processing. The IPP must respond to requests for data -from client science pipelines. In the active state, the IPP must -respond to analysis priority requests issued by the IPP users. +In active state, the IPP must: + +\begin{enumerate} +\item Accept images and metadata from the external sources (i.e., the + summit) + +\item Automatically perform the complete set of image processing + tasks, including both calibration and science image processing. + +\item Respond to requests for data from client science pipelines. + +\item Respond to analysis priority requests issued by the IPP users. +\end{enumerate} \subsubsection{Paused State} @@ -234,7 +240,12 @@ \label{req:interactive-state} -In interactive state, the IPP must accept incoming data and metadata, -but must not automatically process the data. The IPP must respond to -user commands to initiate portions of the data analysis. +In interactive state, the IPP must: + +\begin{enumerate} +\item Accept incoming data and metadata from the external sources. +\item Not automatically process the data +\item Respond to user commands to initiate portions of the data + analysis. +\end{enumerate} \subsection{Software Coding Requirements} @@ -243,12 +254,12 @@ \label{req:languages} -Source code must be in C. All source code must be compiled with `gcc' -version v2.95 or higher, and the tested compiler version must be -defined for the delivered software product. - -Scripting language must be \tbd{Python, version TBD}. +\begin{enumerate} +\item Source code must be in C. +\item All source code must be compiled with `gcc' version v2.95 or higher. +\item The tested compiler version must be defined for the delivered software product. +\item Scripting language must be \tbd{Python}, version X.X. +\end{enumerate} \subsubsection{Interfaces} -We require the following types of interfaces: \begin{enumerate} \item Access to low-level Library functions must be provided via C @@ -278,82 +289,99 @@ \subsubsection{Naming Conventions} -Header files must have names starting \code{ps} or \code{p_ps} for -private interface definitions. The latter must appear in a +\begin{enumerate} +\item Header files must have names starting \code{ps} or \code{p_ps} +for private interface definitions. The latter must appear in a subdirectory \code{private} of whichever directory is being searched for the public header files. -Functions visible at global scope that are part of the public API must -have names beginning with \code{ps} and follow the naming conventions -in the coding standard. Functions visible at global scope but which -are not part of the public interface must have names beginning with -\code{p_ps}. Functions that are local to a file must \textit{not} -start with \code{ps} or \code{p_ps}. +\item Functions visible at global scope that are part of the public +API must have names beginning with \code{ps} and follow the naming +conventions in the coding standard. + +\item Functions visible at global scope but which are not part of the +public interface must have names beginning with \code{p_ps}. -Variables visible at global scope which are part of the public API -must have names beginning with \code{ps}, and follow the naming -conventions in the coding standard. Variables that are visible at -global scope but which are not part of the public interface must have -names beginning with \code{p_ps}. Variables that are local to a file -must \textit{not} start with \code{ps} (or \code{p_ps}). - -The names of all enumerated types and C-preprocessor symbols (but not -variables declared \code{const}) must start with \code{PS_}, in the -case of public symbols, or \code{P_PS_}, for private symbols. The -rest of the name must be uppercase with words separated by underscores -(\code{_}). An exception is the case of system utilities implemented -as macros, in which case the names must conform to the convention for -function names. - -When defining a function to convert from one type to another, the name -must be of the form \code{psOldToNew}, e.g.\hfil\break -\code{psEquatorialToEcliptic} (\emph{not} +\item Functions that are local to a file must \textit{not} start with +\code{ps} or \code{p_ps}. + +\item Variables visible at global scope which are part of the public +API must have names beginning with \code{ps}, and follow the naming +conventions in the coding standard. + +\item Variables that are visible at global scope but which are not +part of the public interface must have names beginning with +\code{p_ps}. + +\item Variables that are local to a file must \textit{not} start with +\code{ps} (or \code{p_ps}). + +\item The names of all enumerated types and C-preprocessor symbols +(but not variables declared \code{const}) must start with \code{PS_}, +in the case of public symbols, or \code{P_PS_}, for private symbols. +The rest of the name must be uppercase with words separated by +underscores (\code{_}). An exception is the case of system utilities +implemented as macros, in which case the names must conform to the +convention for function names. + +\item When defining a function to convert from one type to another, +the name must be of the form \code{psOldToNew}, +e.g.\code{psEquatorialToEcliptic} (\emph{not} \code{psEquatorial2Ecliptic}). +\end{enumerate} \subsubsection{C Programming Guidelines} -Functions that assign to a variable must list that argument -\textit{first}, following the pattern of \code{strcpy}; e.g. -\begin{verbatim} -void psVectorCopy(restrict psVector *out, const restrict psVector *in); -\end{verbatim} - -Type definitions should always be accompanied by prototypes for their -constructors and destructors, following these guidelines: - -\begin{itemize} -\item The constructor name should consist of the type name followed by -\code{Alloc}; e.g. a type \code{psImage} would be created by a -function \code{psImage *psImageAlloc();}. - -\item The type should be freed with a destructor named -\code{typeFree}, e.g. \code{void psImageFree(psImage *image);}. - -\item The constructor must never return \code{NULL}, and no code calling the -constructor should ever check the return value. - -\item The destructor must not return a value. - -\item The destructor must handle being passed \code{NULL} by simply -returning immediately. This must not be treated as an error -condition. - -\item Constructors and Destructors should use the memory reference +\begin{enumerate} +\item Functions that assign to a variable must list that argument + \textit{first}, following the pattern of \code{strcpy}. For + example: + \begin{verbatim} + void psVectorCopy(restrict psVector *out, const restrict psVector *in); + \end{verbatim} + +\item Type definitions should always be accompanied by prototypes for + their constructors and destructors, following these guidelines: + +\begin{enumerate} + \item The constructor name should consist of the type name followed + by \code{Alloc}; e.g. a type \code{psImage} would be created by a + function \code{psImage *psImageAlloc();}. + + \item The type should be freed with a destructor named + \code{typeFree}, e.g. \code{void psImageFree(psImage *image);}. + + \item The constructor must never return \code{NULL}, and no code + calling the constructor should ever check the return value. + + \item The destructor must not return a value. + + \item The destructor must handle being passed \code{NULL} by simply + returning immediately. This must not be treated as an error + condition. + + \item Constructors and Destructors should use the memory reference counter facilities of the PSLib memory management system. - -\end{itemize} +\end{enumerate} +\end{enumerate} \subsubsection{Commenting and Documentation} -Commenting of delivered C and Python code must follow the C and -Python coding standards and must provide tags for Doxygen -interpretation of the comments and program structures. - -Documentation for the IPP consists of source code documentation and -user documentation. Source code documentation must be generated with -Doxygen from the in-line comments and must be provided as HTML, -Latex, and man pages. User documentation includes the API usage for -the modules and library functions as well as user interface -description for the higher-level architectural systems. User -documentation must be delivered as PDF documents. +\begin{enumerate} +\item Commenting of delivered C code must follow the C coding + standards and provide tags for Doxygen interpretation of the + comments and program structures. + +\item Commenting of delivered Python code must follow the Python + coding standards. + +\item Source code documentation must be generated with Doxygen from + the in-line comments and must be provided as HTML, Latex, and man + pages. + +\item User documentation includes the API usage for the modules and + library functions as well as user interface description for the + higher-level architectural systems. User documentation must be + delivered as PDF documents. +\end{enumerate} \subsubsection{Version Control} @@ -409,5 +437,5 @@ The architectural components consist of: -\begin{enumerate} +\begin{itemize} \item {\bf Image Server:} This component is a large data store for all @@ -444,5 +472,5 @@ to assign the analysis tasks to the Controller. -\end{enumerate} +\end{itemize} The relationship between these software elements is shown in @@ -461,40 +489,44 @@ \subsubsection{Image Server} -The IPP Image Server must store images on a distributed collection of -computer disks. Individual instances of a file are only required to -be stored on a single machine (striping across computers is not a -requirement). - -The IPP Image Server must be capable of honoring requests to store an -image on a specific machine. If such a request cannot be honored (ie, -the machine is down), the IPP Image Server must select an appropriate -machine and notify the requesting agent of the new location. - -The IPP Image Server must store multiple copies of each image upon -request, the number of copies specified independently for each file by -the user. - -The IPP Image Server must maintain a record of all image copies -currently available in the repository. This record must include the -image name, location (which machine), the image size, and the state of -the image (available, locked, deleted). - -The IPP Image Server must lock images in the repository on request. -Both read (shared) and write (exclusive) locks must be provided. A -read lock must prevent write access to the file; a write lock must -prevent both read and write access. - -The IPP Image Server must return the image location (the computer on -which it resides) upon request. - -The IPP Image Server must provide a specified image upon request. - -The IPP Image Server must delete images in the repository on request. - -The IPP Image Server must accept images from the summit at the maximum -rate of 1 full-camera image every 30 seconds. The IPP Image Server -must therefore accept new images into the repository at a rate of 64 -raw OTAs in 30 seconds and a total input data volume rate of 75 -MB/sec. +\begin{enumerate} +\item The IPP Image Server must store images on a distributed + collection of computer disks. Individual instances of a file are + only required to be stored on a single machine (striping across + computers is not a requirement). + +\item The IPP Image Server must be capable of honoring requests to + store an image on a specific machine. + +\item If such a request cannot be honored (ie, the machine is down), + the IPP Image Server must select an appropriate machine and notify + the requesting agent of the new location. + +\item The IPP Image Server must store multiple copies of each image + upon request, the number of copies specified independently for each + file by the user. + +\item The IPP Image Server must maintain a record of all image copies + currently available in the repository. This record must include the + image name, location (which machine), the image size, and the state + of the image (available, locked, deleted). + +\item The IPP Image Server must lock images in the repository on + request. Both read (shared) and write (exclusive) locks must be + provided. A read lock must prevent write access to the file; a + write lock must prevent both read and write access. + +\item The IPP Image Server must return the image location (the + computer on which it resides) upon request. + +\item The IPP Image Server must provide a specified image upon request. + +\item The IPP Image Server must delete images in the repository on request. + +\item The IPP Image Server must accept images from the summit at the + maximum rate of 1 full-camera image every 30 seconds. The IPP Image + Server must therefore accept new images into the repository at a + rate of 64 raw OTAs in 30 seconds and a total input data volume rate + of 75 MB/sec. +\end{enumerate} \tbd{archive lifetime} @@ -532,75 +564,91 @@ \end{table} -The AP Database must accept and store individual detections and -collections of detections along with information about the image which -provided the detections. - -Detections must be saved as one of several detection classes (P2, P4S, -P4D, SS) and the AP Database must store the appropriate parameters, -listed in Table~\ref{APdetections}, for each class. - -The AP Database must identify the image which provided the detection, -or in the case of external references, an identifier specific to the -reference source. - -The AP Database must group detections into objects and measure average -parameters of those objects. - -The AP Database must store parallax and proper motion parameters for a -subset of the average objects. - -The AP Database must store image and filter calibration information -necessary to convert between instrumental magnitudes and calibrated -magnitudes in standard systems. - -The AP Database must perform at least the follow queries, with -constraints on the output based on at least time ranges, magnitude -limits, error limits: -\begin{enumerate} -\item given $(RA,DEC)$ and a Radius, return all objects and/or -detections in the region. - -\item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all objects and/or - detections in the region. - -\item given $(RA,DEC)$, return closest object. - -\item given object ID, return all detections - -\item given detection, return source image data. - -\item given detection, return object. - -\item given $(RA,DEC)$, return all images overlapping coordinate. - -\item given $(RA,DEC)$ and a Radius, return all images overlapping region. - -\item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping - region. - -\item given detection instrumental magnitude, return derived - magnitudes based on calibration information. - -\item given a collection of detections in a filter, determine the - object average magnitude in that filter. - -\item given a collection of objects and detections, determine the - individual image zero-points. - -\item given a region, return all possible combinations of the object - or detection magnitudes $(M_1 - M_2)$. - -\item given a list of $(RA,DEC)$ entries, return all nearest objects. - -\item given a filter, telescope, or detector, return all calibration - terms and history. - -\item given a detection, return all non-detections from images which - overlapped the detection coordinates. - -\end{enumerate} - -The AP Database must accept detection IDs of moving objects and label -the detections with the identified object. +\begin{enumerate} +\item The AP Database must accept and store individual detections and + collections of detections along with information about the image + which provided the detections. + +\item Detections must be saved as one of several detection classes + (P2, P4S, P4D, SS) and the AP Database must store the appropriate + parameters, listed in Table~\ref{APdetections}, for each class. + +\item The AP Database must identify the image which provided the + detection, or in the case of external references, an identifier + specific to the reference source. + +\item The AP Database must group detections into objects and measure + average parameters of those objects. + +\item The AP Database must store parallax and proper motion parameters + for a subset of the average objects. + +\item The AP Database must store image and filter calibration + information necessary to convert between instrumental magnitudes and + calibrated magnitudes in standard systems. + +\item The AP Database must perform at least the follow queries, with + constraints on the output based on at least time ranges, magnitude + limits, error limits: + + \begin{enumerate} + \item given $(RA,DEC)$ and a Radius, return all objects and/or + detections in the region. + + \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all objects and/or + detections in the region. + + \item given $(RA,DEC)$, return closest object. + + \item given object ID, return all detections + + \item given detection, return source image data. + + \item given detection, return object. + + \item given $(RA,DEC)$, return all images overlapping coordinate. + + \item given $(RA,DEC)$ and a Radius, return all images overlapping region. + + \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping + region. + + \item given detection instrumental magnitude, return derived + magnitudes based on calibration information. + + \item given a collection of detections in a filter, determine the + object average magnitude in that filter. + + \item given a collection of objects and detections, determine the + individual image zero-points. + + \item given a region, return all possible combinations of the object + or detection magnitudes $(M_1 - M_2)$. + + \item given a list of $(RA,DEC)$ entries, return all nearest objects. + + \item given a filter, telescope, or detector, return all calibration + terms and history. + + \item given a detection, return all non-detections from images which + overlapped the detection coordinates. + \end{enumerate} + +\item The AP Database must accept detection IDs of moving objects and + label the detections with the identified object. + +\item The AP Database must accept new detections at the rate generated + by the telescope from the Phase 2 and Phase 4 analysis. Except + within 10 degrees of the galactic plane, the AP Database must keep + up with the incoming rates. The expected rates are listed in + Table~\ref{APrates}, along with the total data volume required for + storage space over the PS-1 lifetime. + +\end{enumerate} + +\tbd{archive lifetime} + +\tbd{reliability} + +\tbd{backups} \begin{table} @@ -625,23 +673,9 @@ \end{table} -The AP Database must accept new detections at the rate generated by -the telescope from the Phase 2 and Phase 4 analysis. Except within 10 -degrees of the galactic plane, the AP Database must keep up with the -incoming rates. The expected rates are listed in Table~\ref{APrates}, -along with the total data volume required for storage space over the -PS-1 lifetime. - -\tbd{archive lifetime} - -\tbd{reliability} - -\tbd{backups} - \subsubsection{Metadata Database} \begin{table} \begin{center} -\caption{Metadata Classes\label{, and the while -the metadata}} +\caption{Metadata Classes\label{metadata}} \begin{tabular}{l} \hline @@ -673,128 +707,152 @@ stages -The Metadata Database must store and provide metadata for all raw -images, for processed images, for the calibration images (both raw and -master), for the extracted object lists. Metadata describing the -environmental conditions at the telescope must also be stored and -provided as needed. Table~\ref{metadata} lists the classes of -metadata which must be stored by the Metadata Database. - -If analysis results are exchanged between analysis stages via the -Metadata Database, it must provide access to the queried data on -timescales of $<2$ seconds to avoid slowing down the analysis systems. - -The Metadata Database must store the metadata for the lifetime of the -project. The Metadata Database must be capable of accepting a total -data volume after 2 years of operation of 128 GB. - -The Metadata Database must respond to simple queries which return the -data in the categories listed in Table~\ref{metadata} based on the -primary data key and with basic constraints of time ranges and other -simple conditional constraints. - -The Metadata must store descriptive information about the raw images -received from the summit and the current state of the data processing. -The Metadata must also store descriptive information for each of the -static sky images currently available. - -The IPP requires configuration information defining the organization -and configuration of the IPP itself. The Metadata database must store -the configuration information with restricted access so that only -specific people may change the information. Examples of configuration -data include the default parameters for the various analysis programs, -the description of the computing environment, and the process status -information, etc. The Metadata Database must restrict access to the -scientific parameters to a different group from the software and -hardware configuration parameters. +\begin{enumerate} +\item The Metadata Database must store and provide metadata for all + raw images, for processed images, for the calibration images (both + raw and master), for the extracted object lists. Metadata + describing the environmental conditions at the telescope must also + be stored and provided as needed. Table~\ref{metadata} lists the + classes of metadata which must be stored by the Metadata Database. + +\item If analysis results are exchanged between analysis stages via + the Metadata Database, it must provide access to the queried data on + timescales of $<2$ seconds to avoid slowing down the analysis + systems. + +\item The Metadata Database must store the metadata for the lifetime + of the project. + +\item The Metadata Database must be capable of accepting a total data + volume after 2 years of operation of 128 GB. + +\item The Metadata Database must respond to simple queries which + return the data in the categories listed in Table~\ref{metadata} + based on the primary data key and with basic constraints of time + ranges and other simple conditional constraints. + +\item The Metadata must store descriptive information about the raw + images received from the summit and the current state of the data + processing. + +\item The Metadata must also store descriptive information for each of + the static sky images currently available. + +\item The IPP requires configuration information defining the + organization and configuration of the IPP itself. The Metadata + database must store the configuration information with restricted + access so that only specific people may change the information. + Examples of configuration data include the default parameters for + the various analysis programs, the description of the computing + environment, and the process status information, etc. + +\item The Metadata Database must restrict access to the scientific + parameters to a different group from the software and hardware + configuration parameters. + +\item In the discussion of the Analysis Stages below, various steps + specify that the values are user-configurable parameters. These + parameters must be stored in and extracted from the Metadata + Database. +\end{enumerate} \subsubsection{Controller} - -The IPP Controller must manage tasks on a cluster of up to 128 -computers. - -On startup, the IPP Controller must attempt to establish communication -with all of its computers and set their state to be {\tt alive} or -{\tt dead} based on the success of the connection. - -The IPP Controller must detect computers which crash or stop -responding and set their state to {\tt dead}. - -The IPP Controller must attempt to re-establish communication with -{\tt dead} computers. - -The IPP Controller must accept tasks from external users and systems, -which may specify a desired CPU (node) and priority in addition to the -task command. - -The IPP Controller must attempt to run pending tasks on the desired -node, if available (not {\tt dead} or {\tt off}). If the node is -unavailable, the IPP Controller must attempt to run the task on -another node. If the node is available, the IPP Controller must -attempt to run a given task only if no higher-priority tasks are -available and no task is currently being executed. - -The IPP Controller must monitor the output from the task and write it -to an associated log destination. - -The IPP Controller must monitor the execution status of each task -currently executing on a node and perform the following actions: -\begin{enumerate} -\item identify the task as successful if it has a valid exit status. -\item identify the task as unsuccessful if it has an error exit - status. -\item identify the task as unattempted if the computer crashed. -\end{enumerate} - -The IPP Controller must accept and perform the following external -commands: -\begin{enumerate} -\item add a task to the pending task list. -\item delete a specific task from the pending task list. -\item return the current status of a specific task. -\item return a list of all pending and non-pending tasks. -\item set a specified computer state to {\tt off} or {\tt dead}. -\item restrict a specified CPU to a class of tasks. -\item halt execution of a specified task. -\item set the IPP Controller state to {\tt finish}, {\tt abort}, or - {\tt stop}. +\begin{enumerate} + +\item The IPP Controller must manage tasks on a cluster of up to 128 + computers. + +\item On startup, the IPP Controller must attempt to establish + communication with all of its computers and set their state to be + {\tt alive} or {\tt dead} based on the success of the connection. + +\item The IPP Controller must detect computers which crash or stop + responding and set their state to {\tt dead}. + +\item The IPP Controller must attempt to re-establish communication + with {\tt dead} computers. + +\item The IPP Controller must accept tasks from external users and + systems, which may specify a desired CPU (node) and priority in + addition to the task command. + +\item The IPP Controller must attempt to run pending tasks on the + desired node, if available (not {\tt dead} or {\tt off}). + +\item If the node is unavailable, the IPP Controller must attempt to + run the task on another node. + +\item If the node is available, the IPP Controller must attempt to run + a given task only if no higher-priority tasks are available and no + task is currently being executed. + +\item The IPP Controller must monitor the output from the task and + write it to an associated log destination. + +\item The IPP Controller must monitor the execution status of each + task currently executing on a node and perform the following + actions: + + \begin{enumerate} + \item identify the task as successful if it has a valid exit status. + \item identify the task as unsuccessful if it has an error exit + status. + \item identify the task as unattempted if the computer crashed. + \end{enumerate} + +\item The IPP Controller must accept and perform the following + external commands: + \begin{enumerate} + \item add a task to the pending task list. + \item delete a specific task from the pending task list. + \item return the current status of a specific task. + \item return a list of all pending and non-pending tasks. + \item set a specified computer state to {\tt off} or {\tt dead}. + \item restrict a specified CPU to a class of tasks. + \item halt execution of a specified task. + \item set the IPP Controller state to {\tt finish}, {\tt abort}, or + {\tt stop}. + \end{enumerate} \end{enumerate} \subsubsection{Scheduler} - -The IPP Scheduler initiates analysis tasks which it must send to the -IPP Controller. - -All analysis tasks sent by the IPP Scheduler must include a complete -UNIX command with necessary arguments, the priority of the task, and -optionally the desired processing node. - -The IPP Scheduler must refer to several input data sources to decide -what tasks to initiate. These data sources include the IPP Metadata -Database, the Summit Metadata Database, and User requests. - -The IPP Scheduler must query the Databases on a regular basis to check -for new input information. These queries must take place at least -once every \tbr{5 seconds}. - -The IPP Scheduler must accept new User input in real-time (within 0.1 -seconds of the request). - -The IPP Scheduler must construct new tasks on the basis of the inputs -and a task dependency table. - -When the IPP Scheduler is placed in the {\em paused state}, it must -only initiate User-requested tasks. - -When the IPP Scheduler is placed in the {\em interactive state}, it -must initiate User-requested tasks as well as data transfer tasks. - -When the IPP Scheduler is placed in the {\em automatic state}, it must -initiate the most appropriate task based on the inputs. - -The IPP Scheduler must receive the exit status of tasks from the IPP -Controller. - -The IPP Scheduler must send the exit status of the analysis tasks to -the appropriate destination as defined by the task dependency table. +\begin{enumerate} +\item The IPP Scheduler must send the analysis tasks which it + initiates to the IPP Controller. + +\item All analysis tasks sent by the IPP Scheduler must include a + complete UNIX command with necessary arguments, the priority of the + task, and optionally the desired processing node. + +\item The IPP Scheduler must refer to several input data sources to + decide what tasks to initiate. These data sources include the IPP + Metadata Database, the Summit Metadata Database, and User requests. + +\item The IPP Scheduler must query the Databases on a regular basis to + check for new input information. These queries must take place at + least once every \tbr{5 seconds}. + +\item The IPP Scheduler must accept new User input in real-time + (within 0.1 seconds of the request). + +\item The IPP Scheduler must construct new tasks on the basis of the + inputs and a task dependency table. + +\item When the IPP Scheduler is placed in the {\em paused state}, it + must only initiate User-requested tasks. + +\item When the IPP Scheduler is placed in the {\em interactive state}, + it must initiate User-requested tasks as well as data transfer + tasks. + +\item When the IPP Scheduler is placed in the {\em automatic state}, + it must initiate the most appropriate task based on the inputs. + +\item The IPP Scheduler must receive the exit status of tasks from the + IPP Controller. + +\item The IPP Scheduler must send the exit status of the analysis + tasks to the appropriate destination as defined by the task + dependency table. +\end{enumerate} \subsection{Analysis Stages} @@ -813,59 +871,72 @@ group. -The science image analysis stages must perform their analysis quickly -enough to keep up with the incoming data stream. The required -processing time is derived from the rate at which science images are -obtained by PS-1. At a minimum, the Science Image Analysis must keep -up with the average image rate over the course of 1 day. In order to -provide a sufficient buffer for variations in the processing speed, -the Science Image Analysis must be able to process all images from a -night within 12 hours. - -The maximum latency between the acquisition of an image and the -completion of the science image analysis is set by the science -requirements of the fast transient recovery programs. The science -image analysis must process images from these observing programs -within \tbr{5 min} of their arrival time in the IPP Image Server. - -The science image analysis stages must processes up to 1000 science -images per night. +\begin{enumerate} +\item The science image analysis stages must perform their analysis + quickly enough to keep up with the incoming data stream. The + required processing time is derived from the rate at which science + images are obtained by PS-1. + +\item At a minimum, the Science Image Analysis must keep up with the + average image rate over the course of 1 day. + +\item In order to provide a sufficient buffer for variations in the + processing speed, the Science Image Analysis must be able to process + all images from a night within 12 hours. + +\item The maximum latency between the acquisition of an image and the + completion of the science image analysis is set by the science + requirements of the fast transient recovery programs. The science + image analysis must process images from these observing programs + within \tbr{5 min} of their arrival time in the IPP Image Server. + +\item The science image analysis stages must processes up to 1000 + science images per night. + +\end{enumerate} \subsubsection{Phase 1 : image processing preparation} -The Phase 1 analysis stage must determine the astrometric solution of -the complete camera (FPA image) with an accuracy of \tbr{1 arcsec} -peak-to-peak deviation. - -The Phase 1 analysis stage must load the guide star pixel and -celestial coordinates from the \tbd{IPP Metadata Database}\comment{or -from the image header?}. - -If guide stars are not available, the Phase 1 analysis stage must -extract bright stars from the image. This extraction must be done in -less than \tbr{1 second}. The total number of stars and size of the -bright-star acquisition box must be a user-configurable parameter. - -In order for blind astrometry of an image to succeed, it is necessary -that approximate image coordinates be known. The Phase 1 analysis -must be able to succeed despite initial coordinate errors as large as -\tbr{20\arcsec}. - -The Phase 1 analysis stage must construct a table of the overlaps -between the science image to be processed and the static sky images. - -The overlaps must be overestimated by a small amount so that errors in -astrometry at Phase 1 will not cause any valid static sky / science -image pairs to be missed. The amount of overlap must be a -user-configurable parameter. - -Sky cells which do not have sufficient science image overlap \tbd{$< -5\%$} must be excluded from the overlap table. - -It is not unusual for an image to be obtained with invalid coordinates -or without any valid stars. For example, the telescope control system -may make an error and report the wrong time or coordinates. Or, the -image may be obtained in exceptionally poor conditions with no -detected stars. Phase 1 must return a descriptive error message in -these conditions. +\begin{enumerate} +\item The Phase 1 analysis stage must determine the astrometric + solution of the complete camera (FPA image) with an accuracy of + \tbr{1 arcsec} peak-to-peak deviation. + +\item The Phase 1 analysis stage must load the guide star pixel and + celestial coordinates from the \tbd{IPP Metadata + Database}\comment{or from the image header?}. + +\item If guide stars are not available, the Phase 1 analysis stage + must extract bright stars from the image. + +\item This extraction must be done in less than \tbr{1 second}. + +\item The total number of stars and size of the bright-star + acquisition box must be a user-configurable parameter. + +\item In order for blind astrometry of an image to succeed, it is + necessary that approximate image coordinates be known. The Phase 1 + analysis must be able to succeed despite initial coordinate errors + as large as \tbr{20\arcsec}. + +\item The Phase 1 analysis stage must construct a table of the + overlaps between the science image to be processed and the static + sky images. + +\item The overlaps must be overestimated by a small amount so that + errors in astrometry at Phase 1 will not cause any valid static sky + / science image pairs to be missed. + +\item The amount of overlap must be a user-configurable parameter. + +\item Sky cells which do not have sufficient science image overlap + \tbd{$< 5\%$} must be excluded from the overlap table. + +\item It is not unusual for an image to be obtained with invalid + coordinates or without any valid stars. For example, the telescope + control system may make an error and report the wrong time or + coordinates. Or, the image may be obtained in exceptionally poor + conditions with no detected stars. Phase 1 must return a + descriptive error message in these conditions. +\end{enumerate} \subsubsection{Phase 2 : image reduction} @@ -874,53 +945,61 @@ the detector are processed to remove instrumental signatures. -The Phase 2 analysis stage must consult the processing recipe to -define the necessary analysis steps performed by the Phase 2 stage. - -Phase 2 must perform the analysis steps only if required by the -processing recipe. The processing recipe must define the stages to be -executed with optional exposure time and background flux limits to -require or exclude select certain stages. - -In the discussion below, various steps specify that the values are -user-configurable parameters. These parameters must be stored in and -extracted from the Metadata Database. +\paragraph{Processing Recipe} +\begin{enumerate} +\item The Phase 2 analysis stage must consult the processing recipe to + define the necessary analysis steps performed by the Phase 2 stage. + +\item Phase 2 must perform the analysis steps only if required by the + processing recipe. + +\item The processing recipe must define the stages to be executed with + optional exposure time and background flux limits to require or + exclude select certain stages. +\end{enumerate} \paragraph{Detrend Image Convolutions} - -The Phase 2 analysis stage must determine the OT kernel from the IPP -Metadata Database\comment{or image header}. - -The Phase 2 analysis stage must convolve the flat-field and -high-spatial-frequency fringe images with the OT kernel. If no OT -kernel exists, this step must be silently skipped. +\begin{enumerate} + +\item The Phase 2 analysis stage must determine the OT kernel from the + IPP Metadata Database\comment{or image header}. + +\item The Phase 2 analysis stage must convolve the flat-field and + high-spatial-frequency fringe images with the OT kernel. + +\item If no OT kernel exists, this step must be silently skipped. +\end{enumerate} \paragraph{Flag bad and saturated pixels} - -The Phase 2 analysis must load the basic bad pixel map appropriate to -the detector of interest. - -The Phase 2 analysis must use the OT kernel to grow the traps in the +\begin{enumerate} + +\item The Phase 2 analysis must load the basic bad pixel map appropriate to +the detector of interest. + +\item The Phase 2 analysis must use the OT kernel to grow the traps in the raw bad pixel map. -The Phase 2 analysis must mask saturated pixels and a user-specified +\item The Phase 2 analysis must mask saturated pixels and a user-specified number of surrounding pixels. -Different bits must be set to identify different reasons for masking +\item Different bits must be set to identify different reasons for masking the pixels. +\end{enumerate} \paragraph{Bias correction via overscan subtraction} - -Phase 2 must perform bias subtraction on the image. - -Phase 2 must choose the bias subtraction method and analysis statistic +\begin{enumerate} + +\item Phase 2 must perform bias subtraction on the image. + +\item Phase 2 must choose the bias subtraction method and analysis statistic based on the user-configured parameters. -The bias correction must be measured from the image overscan region. - -The overscan region must be determined from the image +\item The bias correction must be measured from the image overscan region. + +\item The overscan region must be determined from the image header\comment{or Metadata DB}. -The bias subtraction must apply one of the following bias corrections, +\item The bias subtraction must apply one of the following bias corrections, depending on the user parameters: + \begin{enumerate} \item subtract a single constant from the image. @@ -934,169 +1013,197 @@ \end{enumerate} -The statistic used to calculate the overscan constant or the inputs to -the spline and polynomial fits must be derived from groups of pixels -on the basis of one of several possible statistics, as specified by -the user parameters. The choice of statistics must include the sample -and robust mean, median, and modes. - -In the case of a single constant, all of the overscan pixel values are -used in the calculation of this statistic. In the case of the 1D -functional representation, the input values to the fit must represent -the coordinate along the overscan, with the statistic derived from the -pixels in the perpendicular direction at each location. - -If specified in the user parameters, sigma-clipping must be performed -on the input data values. +\item The statistic used to calculate the overscan constant or the +inputs to the spline and polynomial fits must be derived from groups +of pixels on the basis of one of several possible statistics, as +specified by the user parameters. + +\item The choice of statistics must include the sample and robust +mean, median, and modes. + +\item In the case of a single constant, all of the overscan pixel +values are used in the calculation of this statistic. + +\item In the case of the 1D functional representation, the input +values to the fit must represent the coordinate along the overscan, +with the statistic derived from the pixels in the perpendicular +direction at each location. + +\item If specified in the user parameters, sigma-clipping must be +performed on the input data values. The bias subtraction must leave no residuals greater than \tbr{1 DN} peak-to-peak. +\end{enumerate} \paragraph{Trim object image} - -The Phase 2 analysis must trim the non-imaging pixels from the image. - -The definition of the imaging area must be determined from the +\begin{enumerate} + +\item The Phase 2 analysis must trim the non-imaging pixels from the +image. + +\item The definition of the imaging area must be determined from the Metadata Database\comment{or image header?}. -Phase 2 must trim pixel near the edges that have been compromised due -to OT operation. +\item Phase 2 must trim pixel near the edges that have been +compromised due to OT operation. +\end{enumerate} \paragraph{Correct for non-linearity} -If required, the science image must be corrected for the effects of -non-linearity. The correction must be a function of chip. +If required by the recipe, each chip must be independently corrected for the +effects of non-linearity. \paragraph{Flat-field correction} - -The Phase 2 analysis must divide the science image by the provided -flat-field image. - -The division must handle zero-valued pixels in the flat-field image -without raising floating point exceptions, setting the corresponding -bit value in the mask. - -The flat-field images must be appropriately normalized (see section -\ref{mkcal}). - -The flat-fielded image must have a consistent photometric zero-point -across the chip, and across the full FPA, to within 0.2\% with -peak-to-peak deviations of \tbr{0.5\%}. +\begin{enumerate} + +\item The Phase 2 analysis must divide the science image by the + provided flat-field image. + +\item The division must handle zero-valued pixels in the flat-field + image without raising floating point exceptions, setting the + corresponding bit value in the mask. + +\item The flat-field images must be appropriately normalized (see + section \ref{mkcal}). + +\item The flat-fielded image must have a consistent photometric + zero-point across the chip, and across the full FPA, to within 0.2\% + with peak-to-peak deviations of \tbr{0.5\%}. +\end{enumerate} \tbd{color of stars in flat-field correction?} \paragraph{Sky \& Fringe subtraction} - -The Phase 2 analysis must subtract the sky (and fringe where needed) -contributions from the images. - -The residual after the background subtraction must have an average -offset of 0 counts, excluding the signal from astronomical features. - -The background residuals must have peak-to-peak variations of less -than \tbr{1\%} of the input background amplitude. - -The background residuals must have a scatter of less than \tbr{1\%} of -the input background amplitude for apertures of less than -\tbr{10~arcsec}.\comment{derived from the need for systematic errors -of better than 0.5\% and known background ranges.} +\begin{enumerate} + +\item The Phase 2 analysis must subtract the sky (and fringe where + needed) contributions from the images. + +\item The residual after the background subtraction must have an + average offset of 0 counts, excluding the signal from astronomical + features. + +\item The background residuals must have peak-to-peak variations of + less than \tbr{1\%} of the input background amplitude. + +\item The background residuals must have a scatter of less than + \tbr{1\%} of the input background amplitude for apertures of less + than \tbr{10~arcsec}.\comment{derived from the need for systematic + errors of better than 0.5\% and known background ranges.} +\end{enumerate} \paragraph{Identify `cosmic rays'} - -The Phase 2 analysis must detect cosmic rays in single images which -are brighter than a user-configurable threshold. - -The Phase 2 analysis must mask detected cosmic rays with a unique -bit value in the mask. - -The Phase 2 analysis must extend the masked region by a -user-configurable growth factor. - -The Phase 2 analysis must perform the cosmic ray detection only if it -is required by the analysis recipe. +\begin{enumerate} + +\item The Phase 2 analysis must detect cosmic rays in single images + which are brighter than a user-configurable threshold. + +\item The Phase 2 analysis must mask detected cosmic rays with a + unique bit value in the mask. + +\item The Phase 2 analysis must extend the masked region by a + user-configurable growth factor. + +\item The Phase 2 analysis must perform the cosmic ray detection only + if it is required by the analysis recipe. +\end{enumerate} \paragraph{Find objects in the image} - -The Phase 2 analysis must perform object detection on the processed -images. - -The object detection must detect all objects above a user-configured -threshold. The threshold must be a positive value; negative values -must invoke an error. The detection threshold must optionally be a -function of the average background flux or the local noise level. - -The object detection must measure the following object parameters: -\begin{enumerate} -\item object centroid and position errors -\item an extended object position ($x_g, y_g$) -\item instrumental PSF magnitude and error -\item local background level and error -\item second moments ($\sigma_{\rm min}, \sigma_{maj}$) of the object - and their covariance matrix -\end{enumerate} - -Minimal object classification must be performed to distinguish objects -which are consistent with a single PSF, objects which are -inconsistently large, objects which are inconsistently small, and -objects which are saturated. - -The resulting collection of detected objects must be saved along with -the relevant image metadata (\ie filter, exposure time, etc). +\begin{enumerate} + +\item The Phase 2 analysis must perform object detection on the + processed images. + +\item The object detection process must detect all objects above a + user-configured threshold. + +\item The threshold must be a positive value; negative values must + invoke an error. + +\item The detection threshold must optionally be a function of the + average background flux or the local noise level. + +\item The object detection must measure the following object + parameters: + \begin{enumerate} + \item object centroid and position errors + \item an extended object position ($x_g, y_g$) + \item instrumental PSF magnitude and error + \item local background level and error + \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) of the object + and their covariance matrix + \end{enumerate} + +\item Minimal object classification must be performed to distinguish + objects which are consistent with a single PSF, objects which are + inconsistently large, objects which are inconsistently small, and + objects which are saturated. + +\item The resulting collection of detected objects must be saved along + with the relevant image metadata (\ie filter, exposure time, etc). +\end{enumerate} \paragraph{Astrometry} - -The Phase 2 analysis must match the detected objects with known -astrometric reference objects. - -The astrometric reference object coordinates must be adjusted for -proper motion. - -The reference and detected object coordinates must be fit to determine -astrometric parameters for the individual OTAs. - -The OTA astrometric parameters must include Chebychev polynomials of the -coordinates up to 3rd order. - -The fitted number of polynomial orders must be a user-configured -parameter. - -The Cell astrometric parameters must not be allowed to vary in the -fit. - -The fit must be robust, rejecting outlier matches (either stars with -poorly determined proper motion or spurious matches). - -The resulting astrometric solution must be consistent across the OTA -field to within \tbr{300 milli-arcsec}. +\begin{enumerate} + +\item The Phase 2 analysis must match the detected objects with known + astrometric reference objects. + +\item The astrometric reference object coordinates must be adjusted + for proper motion. + +\item The reference and detected object coordinates must be fit to + determine astrometric parameters for the individual OTAs. + +\item The OTA astrometric parameters must include Chebychev + polynomials of the coordinates up to 3rd order. + +\item The fitted number of polynomial orders must be a user-configured + parameter. + +\item The Cell astrometric parameters must not be allowed to vary in + the fit. + +\item The fit must be robust, rejecting outlier matches (either stars + with poorly determined proper motion or spurious matches). + +\item The resulting astrometric solution must be consistent across the + OTA field to within \tbr{300 milli-arcsec}. +\end{enumerate} \paragraph{Postage Stamps} - -The Phase 2 analysis must extract subrasters (`postage stamps') -surrounding a user-specified list of coordinates from the flattened -images. - -The postage stamp images must be saved in the IPP Image Server. - +\begin{enumerate} + +\item The Phase 2 analysis must extract subrasters (`postage stamps') + surrounding a user-specified list of coordinates from the flattened + images. + +\item The postage stamp images must be saved in the IPP Image Server. +\end{enumerate} + \subsubsection{Phase 3 : exposure analysis} - -The Phase 3 analysis must use the objects detected in Phase 2, matched -with a user-specified reference photometry catalog, to determine the -image photometric zero point and zero-point variations across the -field. - -If zero-point variations are significant \tbd{level TBD}, the -zero-point variations must be modeled with a Chebychev polynomial -correction of order 3 or less. - -The photometric nature of the FPA image must be categorized -\tbd{numerical scale?} on the basis of the zero-point consistency, the -transparency compared with recent long-term measurements in the -filter, and the external indicators of photometricity. - -The Phase 3 analysis must use the objects detected in Phase 2, matched -with an appropriate astrometric reference catalog, to improve the -distortion model used for the image. - -The resulting astrometric accuracy must be limited by the astrometric -reference catalog, ie, 250 mas for USNO-B1.0. +\begin{enumerate} + +\item The Phase 3 analysis must use the objects detected in Phase 2, + matched with a user-specified reference photometry catalog, to + determine the image photometric zero point and zero-point variations + across the field. + +\item If zero-point variations are significant \tbd{level TBD}, the + zero-point variations must be modeled with a Chebychev polynomial + correction of order 3 or less. + +\item The photometric nature of the FPA image must be categorized + \tbd{numerical scale?} on the basis of the zero-point consistency, + the transparency compared with recent long-term measurements in the + filter, and the external indicators of photometricity. + +\item The Phase 3 analysis must use the objects detected in Phase 2, + matched with an appropriate astrometric reference catalog, to + improve the distortion model used for the image. + +\item The resulting astrometric accuracy must be limited by the + astrometric reference catalog, ie, 250 mas for USNO-B1.0. +\end{enumerate} \subsubsection{Phase 4 : image combination} @@ -1108,32 +1215,36 @@ \paragraph{Extract image pixels} - -The Phase 4 analysis must determine the corresponding set of image -pixels for a given sky cell. - -The corresponding image pixels must be extracted from the input -images, using the astrometric information for each OTA and Cell to -determine the exact overlaps. - -The Phase 4 analysis must not miss any pixels in this match, and it -must read no more than 20\% more pixels than necessary from the input -images. - -The Phase 4 analysis must skip any sky cells with fewer than 5\% of -their pixels overlapping the input images. +\begin{enumerate} + +\item The Phase 4 analysis must determine the corresponding set of + image pixels for a given sky cell. + +\item The corresponding image pixels must be extracted from the input + images, using the astrometric information for each OTA and Cell to + determine the exact overlaps. + +\item The Phase 4 analysis must not miss any pixels in this match, and + it must read no more than 20\% more pixels than necessary from the + input images. + +\item The Phase 4 analysis must skip any sky cells with fewer than 5\% + of their pixels overlapping the input images. +\end{enumerate} \paragraph{Transform pixel coordinates} - -Pixels which have been extracted from the input images must be mapped -to the corresponding pixels in the sky image. - -The transformation must be based on the measured astrometric solution -for the input images relative to the reference catalog used to -generate the static sky image. - -This warping must use a locally-linear astrometric solution. - -The output image must maintain photometric consistency with the input -image to within 0.2\%. +\begin{enumerate} + +\item Pixels which have been extracted from the input images must be + mapped to the corresponding pixels in the sky image. + +\item The transformation must be based on the measured astrometric + solution for the input images relative to the reference catalog used + to generate the static sky image. + +\item This warping must use a locally-linear astrometric solution. + +\item The output image must maintain photometric consistency with the + input image to within 0.2\%. +\end{enumerate} \tbd{interpolation? does interpolation method choice risk losing flux?} @@ -1147,11 +1258,12 @@ \paragraph{Image outlier pixel rejection} - -When multiple images are combined, the group of input pixels which -contribute to an output pixel must be examined and pixels from the -group of images which are inconsistent with the ensemble \tbd{how -much?} must be identified and flagged. - -This outlier rejection must be performed optionally. +\begin{enumerate} + +\item When multiple images are combined, the group of input pixels + which contribute to an output pixel must be examined and pixels from + the group of images which are inconsistent with the ensemble + \tbd{how much?} must be identified and flagged. + +\item This outlier rejection must be performed optionally. \tbd{for moving objects and images which are not simultaneous, do we @@ -1160,4 +1272,5 @@ \tbd{use the spatial information? fit a 2-D Nth order polynomial to the collection of pixels and then look for outliers} +\end{enumerate} \paragraph{Initial cleaned image} @@ -1178,70 +1291,86 @@ \paragraph{Find objects in the image} - -The Phase 4 analysis must perform object detection on the difference -images. - -All objects in the difference image must be detected and the pixels -belonging to variable sources flagged in the input image. - -The object detection must detect all objects above a user-configured -threshold. Both positive and negative objects must be detected; the -specified threshold must define the absolute value of the detection -thresholds. The detection threshold must optionally be a function of -the average background flux or the local noise level. - -The object detection must measure the following object parameters: -\begin{enumerate} -\item object centroid and position errors -\item instrumental PSF magnitude and error -\item local background level and error -\item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$ -\item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix -\end{enumerate} - -Minimal object classification must be performed to distinguish objects -which are consistent with a single PSF, objects which are -inconsistent, and objects which are saturated. - -The resulting collection of detected objects must be saved along with -the relevant image metadata (\ie filter, exposure time, etc). +\begin{enumerate} + +\item The Phase 4 analysis must perform object detection on the + difference images. + +\item All objects in the difference image must be detected and the + pixels belonging to variable sources flagged in the input image. + +\item The object detection must detect all objects above a + user-configured threshold. + +\item Both positive and negative objects must be detected: the + specified threshold must define the absolute value of the detection + thresholds. + +\item The detection threshold must optionally be a function of the + average background flux or the local noise level. + +\item The object detection must measure the following object parameters: + \begin{enumerate} + \item object centroid and position errors + \item instrumental PSF magnitude and error + \item local background level and error + \item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$ + \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix + \end{enumerate} + +\item Minimal object classification must be performed to distinguish + objects which are consistent with a single PSF, objects which are + inconsistent, and objects which are saturated. + +\item The resulting collection of detected objects must be saved along + with the relevant image metadata (\ie filter, exposure time, etc). +\end{enumerate} \paragraph{Cleaned Input Image} - -The pixels flagged as being from the difference image sources must be -masked in the input images. - -A new, cleaned image must be constructed from the masked input images. +\begin{enumerate} + +\item The pixels flagged as being from the difference image sources + must be masked in the input images. + +\item A new, cleaned image must be constructed from the masked input + images. + +\end{enumerate} \tbd{how to handle variable stars?} \paragraph{Find objects in the image} - -The Phase 4 analysis must perform object detection on the cleaned, -summed image. - -The object detection must detect all objects above a user-configured -threshold. The threshold must be a positive value; negative values -must invoke an error. The detection threshold optionally must be a -function of the average background flux or the local noise level. - -The object detection must measure the following object parameters: -\begin{enumerate} -\item object centroid and position errors -\item an extended object position ($x_g, y_g$) -\item instrumental PSF magnitude and error -\item local background level and error -\item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their - covariance matrix -\item the Petrosian radius, magnitude, axis ratio, and angle -\item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$. -\end{enumerate} - -Minimal object classification must be performed to distinguish objects -which are consistent with a single PSF, objects which are -inconsistent, and objects which are saturated. - -The resulting collection of detected objects must be saved along with -the relevant image metadata (\ie filter, exposure time, etc). +\begin{enumerate} + +\item The Phase 4 analysis must perform object detection on the + cleaned, summed image. + +\item The object detection must detect all objects above a + user-configured threshold. + +\item The threshold must be a positive value; negative values must + invoke an error. + +\item The detection threshold optionally must be a function of the + average background flux or the local noise level. + +\item The object detection must measure the following object parameters: + \begin{enumerate} + \item object centroid and position errors + \item an extended object position ($x_g, y_g$) + \item instrumental PSF magnitude and error + \item local background level and error + \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their + covariance matrix + \item the Petrosian radius, magnitude, axis ratio, and angle + \item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$. + \end{enumerate} + +\item Minimal object classification must be performed to distinguish + objects which are consistent with a single PSF, objects which are + inconsistent, and objects which are saturated. + +\item The resulting collection of detected objects must be saved along + with the relevant image metadata (\ie filter, exposure time, etc). +\end{enumerate} \paragraph{Image Processing Q/A} @@ -1281,5 +1410,5 @@ \label{mkcal} -tbd{Requirements on the speed of processing the calibration images.} +\tbd{Requirements on the speed of processing the calibration images.} The Calibration analysis stages must construct the various types of @@ -1288,127 +1417,141 @@ \paragraph{bias images} - -The \code{bias} calibration stage must construct a master bias image -from a collection of raw bias images. - -The \code{bias} calibration stage must correct the input images based -on the overscan region. - -The \code{bias} calibration stage must combine the input images using -the statistic specified by the user, selected from one of the -following: sample mean, median, and mode, robust mean, median, and -mode, and the clipped mean and median. - -The \code{bias} calibration stage must construct residual images, in -which the master bias is applied to the input images. - -Outlier residual images, those for which the residual bias and -variance in the bias image are excessive ($> 1DN$), must be excluded -from the input image stack the the bias image reconstructed. +\begin{enumerate} + +\item The \code{bias} calibration stage must construct a master bias + image from a collection of raw bias images. + +\item The \code{bias} calibration stage must correct the input images + based on the overscan region. + +\item The \code{bias} calibration stage must combine the input images + using the statistic specified by the user, selected from one of the + following: sample mean, median, and mode, robust mean, median, and + mode, and the clipped mean and median. + +\item The \code{bias} calibration stage must construct residual + images, in which the master bias is applied to the input images. + +\item Outlier residual images, those for which the residual bias and + variance in the bias image are excessive ($> 1DN$), must be excluded + from the input image stack the the bias image reconstructed. +\end{enumerate} \paragraph{dark images} - -The \code{dark} calibration stage must construct a master dark image -from a collection of raw dark images. - -The \code{dark} calibration stage must raise an error if the input -images have exposure time which deviate by more than \tbr{2\%}. - -The \code{dark} calibration stage must correct the input dark images -for the bias. - -The \code{dark} calibration stage must combine the input images using -the statistic specified by the user, selected from one of the -following: sample mean, median, and mode, robust mean, median, and -mode, and the clipped mean and median. - -The \code{dark} calibration stage must construct residual images, in -which the master dark is applied to the input images. - -Outlier residual images, those for which the residual level and -variance are excessive ($> 1DN$), must be excluded from the input -image stack the the dark image reconstructed. +\begin{enumerate} + +\item The \code{dark} calibration stage must construct a master dark + image from a collection of raw dark images. + +\item The \code{dark} calibration stage must raise an error if the + input images have exposure time which deviate by more than + \tbr{2\%}. + +\item The \code{dark} calibration stage must correct the input dark + images for the bias. + +\item The \code{dark} calibration stage must combine the input images + using the statistic specified by the user, selected from one of the + following: sample mean, median, and mode, robust mean, median, and + mode, and the clipped mean and median. + +\item The \code{dark} calibration stage must construct residual + images, in which the master dark is applied to the input images. + +\item Outlier residual images, those for which the residual level and + variance are excessive ($> 1DN$), must be excluded from the input + image stack the the dark image reconstructed. +\end{enumerate} \paragraph{flat-field images} - -The \code{flat-field} calibration stage must construct a master -flat-field image from a collection of raw flat-field images. - -The \code{flat-field} calibration stage must accept a group of images -from one of the following flat-field sources: dome, twilight, -night-sky. - -The \code{flat-field} calibration stage must raise an error if the -input images in a single stack used more than one of the above -flat-field sources or multiple filters. - -The \code{flat-field} calibration stage must correct the input -flat-field images for the bias and dark. - -The \code{flat-field} calibration stage must combine the input images -using the statistic specified by the user, selected from one of the -following: sample mean, median, and mode, robust mean, median, and -mode, and the clipped mean and median. - -The \code{flat-field} calibration stage must construct residual -images, in which the master flat-field is applied to the input images. - -Outlier residual images, those for which the residual level and -variance are excessive ($> 0.1$\%, or 1.02 times the Poisson limit of -the flat-field image), must be excluded from the input image stack the -the flat-field image reconstructed. +\begin{enumerate} + +\item The \code{flat-field} calibration stage must construct a master + flat-field image from a collection of raw flat-field images. + +\item The \code{flat-field} calibration stage must accept a group of + images from one of the following flat-field sources: dome, twilight, + night-sky. + +\item The \code{flat-field} calibration stage must raise an error if + the input images in a single stack used more than one of the above + flat-field sources or multiple filters. + +\item The \code{flat-field} calibration stage must correct the input + flat-field images for the bias and dark. + +\item The \code{flat-field} calibration stage must combine the input + images using the statistic specified by the user, selected from one + of the following: sample mean, median, and mode, robust mean, + median, and mode, and the clipped mean and median. + +\item The \code{flat-field} calibration stage must construct residual + images, in which the master flat-field is applied to the input + images. + +\item Outlier residual images, those for which the residual level and + variance are excessive ($> 0.1$\%, or 1.02 times the Poisson limit + of the flat-field image), must be excluded from the input image + stack the the flat-field image reconstructed. +\end{enumerate} \paragraph{mask images} - -The \code{mask} calibration stage must construct a bad-pixel mask from -a stack of raw flat-field images and a master flat-field image. - -The \code{mask} calibration stage must mask the pixels which are -inconsistent in the input flats by more than \tbr{1\%}, given -sufficient signal-to-noise in the input flats. - -The \code{mask} calibration stage must mask the pixels which are -consistently low or high in the input flats by more than a factor of -\tbr{3} beyond the typical pixel. - -The \code{mask} calibration stage must mask the pixels identified in a -table of bad pixels generated externally to the calibration stage. - -The \code{mask} calibration stage must use multiple bit values to -identify the different types of masked pixels. - -The \code{mask} calibration stage must raise an error if the input -images generate too many bad pixels in the mask. +\begin{enumerate} + +\item The \code{mask} calibration stage must construct a bad-pixel + mask from a stack of raw flat-field images and a master flat-field + image. + +\item The \code{mask} calibration stage must mask the pixels which are + inconsistent in the input flats by more than \tbr{1\%}, given + sufficient signal-to-noise in the input flats. + +\item The \code{mask} calibration stage must mask the pixels which are + consistently low or high in the input flats by more than a factor of + \tbr{3} beyond the typical pixel. + +\item The \code{mask} calibration stage must mask the pixels + identified in a table of bad pixels generated externally to the + calibration stage. + +\item The \code{mask} calibration stage must use multiple bit values + to identify the different types of masked pixels. + +\item The \code{mask} calibration stage must raise an error if the + input images generate too many bad pixels in the mask. +\end{enumerate} \paragraph{fringe frames} - -The \code{fringe} calibration stage must construct a master fringe +\begin{enumerate} + +\item The \code{fringe} calibration stage must construct a master fringe frame from a stack of raw night-time sky images or from a stack of dome fringe frames. -The \code{fringe} calibration stage must raise an error if the input +\item The \code{fringe} calibration stage must raise an error if the input stack consists is images generated with more than one type of fringe source or with multiple filters. -The \code{fringe} calibration stage must flatten the input images +\item The \code{fringe} calibration stage must flatten the input images to remove the pixel-to-pixel sensitivity variations of the detector. -The \code{fringe} calibration stage must measure the fringe amplitude +\item The \code{fringe} calibration stage must measure the fringe amplitude on the input fringe images. -The \code{fringe} calibration stage must scale the input fringe images +\item The \code{fringe} calibration stage must scale the input fringe images based on the fringe amplitude. -The \code{fringe} calibration stage must combine the scaled input +\item The \code{fringe} calibration stage must combine the scaled input images using the statistic specified by the user, selected from one of the following: sample mean, median, and mode, robust mean, median, and mode, and the clipped mean and median. -The \code{fringe} calibration stage must construct residual images, in +\item The \code{fringe} calibration stage must construct residual images, in which the master fringe image is applied to the input images, along with all necessary preceding calibration images. -The \code{fringe} calibration stage must measure the residual fringe +\item The \code{fringe} calibration stage must measure the residual fringe amplitude on the residual images. +\end{enumerate} \paragraph{low-spatial-frequency sky models} @@ -1420,37 +1563,41 @@ \paragraph{Flat-field correction frame} - -The \code{flat-field correction} calibration stage must construct a +\begin{enumerate} + +\item The \code{flat-field correction} calibration stage must construct a flat-field correction image from dithered observations of a stellar field. -The \code{flat-field correction} calibration stage must construct a +\item The \code{flat-field correction} calibration stage must construct a flat-field correction image for each filter and camera independently. -The \code{flat-field correction} calibration stage must construct a +\item The \code{flat-field correction} calibration stage must construct a correction image which makes the photometry of multiple observations of the same stellar source consistent at different locations in the focal plane. -The \code{flat-field correction} calibration stage must construct +\item The \code{flat-field correction} calibration stage must construct corrected flat-field images using the measured correction. -The \code{flat-field correction} calibration stage must determine the +\item The \code{flat-field correction} calibration stage must determine the consistency of the corrected flat-field images using the dithered stellar field observations flattened with the corrected flat-field -image.. +image. +\end{enumerate} \paragraph{Non-linearity correction frames} - -The \code{non-linear correction} calibration stage must construct a +\begin{enumerate} + +\item The \code{non-linear correction} calibration stage must construct a non-linear correction from a collection of images of a constant source with varying exposure times. -The \code{non-linear correction} calibration stage must construct a +\item The \code{non-linear correction} calibration stage must construct a non-linear correction which linearizes the detector fluxes $<0.5\%$. -The \code{non-linear correction} calibration stage must determine the +\item The \code{non-linear correction} calibration stage must determine the saturation regime, in which the non-linear correction is no longer consistent to $<0.5\%$. +\end{enumerate} \subsubsection{Reference Catalog Creation} @@ -1460,10 +1607,10 @@ future Pan-STARRS calibration. The generation of these catalogs is inherently a research project, and will require human control and -intervention. The IPP is required to provide the data access, -manipulation and visualization tools needed to construct these -reference catalogs and to assess their quality. In this section, we -list the requirements of the tools needed for this effort. - -\subsubsection{Astrometry Reference Creation} +intervention. The IPP must provide the data access, manipulation and +visualization tools needed to construct these reference catalogs and +to assess their quality. In this section, we list the requirements of +the tools needed for this effort. + +\paragraph{Astrometry Reference Creation} \begin{table} @@ -1487,18 +1634,20 @@ \end{table} -The IPP must have the capability of generating an astrometric -reference on the basis of the observations obtained by the AP survey. -The IPP must provide the analysis tools needed to generate the master -astrometric reference catalog. Much of the required functionality is -covered by the AP Database. - -The Astrometry Reference creation tools must return the match between +The IPP will generate an astrometric reference on the basis of the +observations obtained by the AP survey. The IPP must provide the +analysis tools needed to generate the master astrometric reference +catalog. Much of the required functionality is covered by the AP +Database. The specific requirements for the Astrometric Reference +creation are listed below: + +\begin{enumerate} +\item The Astrometry Reference creation tools must return the match between stars observed with Pan-STARRS and any of several astrometric reference catalogs listed in Table~\ref{AstroRefs}. -The tools must convert the reference catalog object coordinates to all +\item The tools must convert the reference catalog object coordinates to all of the coordinate frames of relevance in the telescope and camera system: -\begin{itemize} +\begin{enumerate} \item Catalog to mean positions \item Mean to apparent positions @@ -1507,40 +1656,41 @@ \item focal plane to specific detector (OTA) \item specific detector to detector cell -\end{itemize} - -The tools must provide the necessary calibration data: the telescope +\end{enumerate} + +\item The tools must provide the necessary calibration data: the telescope and camera optical distortion models and the variation of the image PSF across the camera field, as a function of color. -The tools must fit the observed stellar coordinates to the astrometric +\item The tools must fit the observed stellar coordinates to the astrometric reference catalog coordinates to determine improved astrometric solutions for both the stars and the detectors. -The tools must determine improved telescope optical distortion models +\item The tools must determine improved telescope optical distortion models based on the astrometric solutions. -The tools must optionally determine the fit coefficients as a function +\item The tools must optionally determine the fit coefficients as a function of position along, or with combinations of magnitude or color. -The fitting method must include robust outlier rejection. - -The tools must identify objects which are detected in the catalog, but +\item The fitting method must include robust outlier rejection. + +\item The tools must identify objects which are detected in the catalog, but not the science image or vice-versa. -The tools must determine average centroiding errors for each object. - -The tools must plot the fit residuals against a wide variety of +\item The tools must determine average centroiding errors for each object. + +\item The tools must plot the fit residuals against a wide variety of parameters: the object positions, magnitudes, colors, etc. -The tools must allow the fit to exclude subsets of objects from the +\item The tools must allow the fit to exclude subsets of objects from the fits on the basis of these parameters. . -The tools must provide coordinates of the guide stars in the same frame +\item The tools must provide coordinates of the guide stars in the same frame of reference as the normal image data. -The tools must perform the various fitting steps for the guide stars +\item The tools must perform the various fitting steps for the guide stars rather than for the normal image data. - -\subsubsection{Photometry Reference Creation} +\end{enumerate} + +\paragraph{Photometry Reference Creation} \begin{table} @@ -1561,44 +1711,47 @@ \end{table} -The IPP must have the capability of generating a photometric reference -on the basis of the observations obtained by the AP survey. The IPP -must provide the analysis tools needed to generate a master -photometric reference catalog. Much of the required functionality is -covered by the AP Database. - -The Photometry Reference creation tools must return the match between +The IPP will generate a photometric reference catalog on the basis of +the observations obtained by the AP survey. The IPP must provide the +analysis tools needed to generate a master photometric reference +catalog. Much of the required functionality is covered by the AP +Database. The specific requirements for the Photometric Reference +creation are listed below: + +\begin{enumerate} +\item The Photometry Reference creation tools must return the match between stars observed with Pan-STARRS and any of several photometric reference catalogs listed in Table~\ref{PhotoRefs}. -The tools must convert between different photometric systems, including: -\begin{itemize} +\item The tools must convert between different photometric systems, including: +\begin{enumerate} \item instrumental to nominal detector magnitude \item nominal detector magnitude to average filter system \item average filter system to reference photometry system -\end{itemize} - -These transformations must account for color and airmass terms. - -The tools must measure and apply relative photometry corrections +\end{enumerate} + +\item These transformations must account for color and airmass terms. + +\item The tools must measure and apply relative photometry corrections between images. -The tools must determine photometric transformation fit coefficients +\item The tools must determine photometric transformation fit coefficients as a function of airmass, magnitude, color and detector coordinates, or with combinations of the above. -The fitting method must include robust outlier rejection. - -The tools must reject specific objects from the fit on the basis of +\item The fitting method must include robust outlier rejection. + +\item The tools must reject specific objects from the fit on the basis of object locations, instrumental magnitudes, observed and reference errors, and in particular time of the observations. -The tools must plot the fit residuals against a wide variety of +\item The tools must plot the fit residuals against a wide variety of parameters, including the object positions, magnitudes, colors, etc. -The tools must provide photometry from the guide stars in the same +\item The tools must provide photometry from the guide stars in the same system as observations of stars from the normal imaging data. -The tools must perform the above fitting steps for the guide stars +\item The tools must perform the above fitting steps for the guide stars rather than for the normal image data. +\end{enumerate} \subsection{Modules} @@ -1640,8 +1793,12 @@ \subsubsection{Image Formats} -Certain IPP programs must be able to read and write standard FITS images. - -Certain IPP programs must be able to read and write files in modified -FITS format with Pan-STARRS definitions for non-square pixel arrays. +\begin{enumerate} +\item Certain IPP programs must be able to read and write standard + FITS images. + +\item Certain IPP programs must be able to read and write files in + modified FITS format with Pan-STARRS definitions for non-square + pixel arrays. +\end{enumerate} \subsubsection{Table Formats} @@ -1658,5 +1815,5 @@ externally provided reference catalogs: -\begin{itemize} +\begin{enumerate} \item Hipparcos \item Tycho2 @@ -1667,5 +1824,5 @@ \item USNO-Bx \item YBx -\end{itemize} +\end{enumerate} \subsubsection{Analysis Reference Data} @@ -1673,5 +1830,5 @@ The IPP must store reference data describing the following entities: -\begin{itemize} +\begin{enumerate} \item Telescopes \item Cameras @@ -1680,5 +1837,5 @@ \item software basic parameters \item computer configuration -\end{itemize} +\end{enumerate} \subsection{External Interfaces} @@ -1709,5 +1866,5 @@ hardware requirements addressed in this section consist of: -\begin{itemize} +\begin{enumerate} \item Total Disk Volume \item Total Processing Power @@ -1715,5 +1872,5 @@ \item Sustained Node Network I/O \item Sustained Disk I/O -\end{itemize} +\end{enumerate} The report, `The Pan-STARRS Image Processing Pipeline Computational @@ -1735,5 +1892,5 @@ calibration frames & 1.8 TB \\ metadata db & 0.2 TB \\ -AP db & 24 TB \\ +AP db & 24 TB \\ \hline total & 461 TB \\ @@ -1751,25 +1908,27 @@ types of data. -The IPP must store all raw images from the first year from the AP and -IVP surveys. This corresponds to 175,000 images, or 175 TB, assuming -1 GB per image and compression. The IPP will require space for 200 TB -of raw imagery to store the data from these two survey components -along with raw calibration, test, and other raw images not in the AP -and IVP surveys. - -The IPP must store a single copy of the complete static sky in all -four filters. With the assumed image sampling of 0.2 arcsec per -pixel, this corresponds to 9.7 Tpix per filter, or a total of 235 TB -for the 6 filters, with 2 bytes for the noise map and 2 bytes for the -image map. - -The IPP must also store other, smaller collections of data. The other -components contribute only a small fraction of the data storage -requirement. The metadata is a fraction of a terabyte, while the -calibration frames (all master detrend frames) represent at most a few -terabytes. The AP object and detection data make up a total of 24 -terabytes (see Table~\ref{APrates}). - -The IPP must have storage capacity for a total of 461 TB of data. +\begin{enumerate} +\item The IPP must store all raw images from the first year from the + AP and IVP surveys. This corresponds to 175,000 images, or 175 TB, + assuming 1 GB per image and compression. The IPP will require space + for 200 TB of raw imagery to store the data from these two survey + components along with raw calibration, test, and other raw images + not in the AP and IVP surveys. + +\item The IPP must store a single copy of the complete static sky in + all four filters. With the assumed image sampling of 0.2 arcsec per + pixel, this corresponds to 9.7 Tpix per filter, or a total of 235 TB + for the 6 filters, with 2 bytes for the noise map and 2 bytes for + the image map. + +\item The IPP must also store other, smaller collections of data. The + other components contribute only a small fraction of the data + storage requirement. The metadata is a fraction of a terabyte, + while the calibration frames (all master detrend frames) represent + at most a few terabytes. The AP object and detection data make up a + total of 24 terabytes (see Table~\ref{APrates}). + +\item The IPP must have storage capacity for a total of 461 TB of data. +\end{enumerate} \subsubsection{CPU Requirements} @@ -1788,5 +1947,6 @@ be addressed independently. -The IPP must perform the Phase 2 analysis within an average time of 45 +\begin{enumerate} +\item The IPP must perform the Phase 2 analysis within an average time of 45 seconds per single Gigapixel camera image. The Phase 2 analysis has been measured to require 3200 GHz-sec on a x86/32 bit machine, @@ -1794,8 +1954,9 @@ are devoted to I/O. -The IPP must perform the Phase 4 analysis on a set of 4 input frames +\item The IPP must perform the Phase 4 analysis on a set of 4 input frames within an average time of 180 seconds. The Phase 4 analysis has been measured to require a total of 7800 GHz-sec on an x86/32 bit machine for a major frame of 4 input Gigapixel camera images. +\end{enumerate} \subsubsection{Network I/O Requirements} @@ -1808,5 +1969,5 @@ and updated static sky images, the total I/O for a 180 second major-frame period is: -\begin{itemize} +\begin{enumerate} \item 8 GB from summit to Phase 2 (4 images @ 2 GB each) \item 18 GB from Phase 2 to Phase 4 (3 bytes per pixel for image + @@ -1815,5 +1976,5 @@ input image pixel, 4 bytes per pixel). \item 9 GB from Phase 4 to Static Sky -\end{itemize} +\end{enumerate} for a grand total of 44 GB over 180 seconds, or 244 MB/second, of which 26 GB are processed by the Phase 2 nodes and 36 are processed by @@ -1826,5 +1987,5 @@ read from and written to disk. For each major frame processed, the total I/O to and from disk for Phase 2 is: -\begin{itemize} +\begin{enumerate} \item 8 GB raw image from summit to Phase 2 nodes (4 images @ 2 GB each) \item 8 GB raw image from Phase 2 disk to memory @@ -1833,12 +1994,12 @@ + 1 byte mask). \item 18 GB processed image from Phase 2 disk to Phase 4 -\end{itemize} +\end{enumerate} for a grand total of 86 GB I/O for Phase 2. Equivalently, for each major frame processed, the total I/O to and from disk for Phase 4 is: -\begin{itemize} +\begin{enumerate} \item 18 GB processed image from Phase 2 disk to Phase 4 \item 9 GB static image from Phase 4 disk to memory \item 9 GB static image from memory to Phase 4 disk -\end{itemize} +\end{enumerate} for a total of 36 GB I/O for Phase 4. @@ -1876,8 +2037,10 @@ \subsection{Software Integrity Tests} - -MHPCC must test that the code does not produce memory leaks. - -MHPCC must test that the code does not produce segmentation faults. +\begin{enumerate} + +\item MHPCC must test that the code does not produce memory leaks. + +\item MHPCC must test that the code does not produce segmentation faults. +\end{enumerate} \subsection{Basic Unit Tests}