Index: /trunk/doc/design/Makefile =================================================================== --- /trunk/doc/design/Makefile (revision 1398) +++ /trunk/doc/design/Makefile (revision 1399) @@ -1,6 +1,10 @@ -# $Id: Makefile,v 1.6 2004-06-25 22:06:34 eugene Exp $ +# $Id: Makefile,v 1.7 2004-08-06 19:06:01 eugene Exp $ PDFLATEX = pdflatex PSLATEX = latex + +srs : + trace.pl ippSRS.tex ippSRSout.tex ippSRStrace.tex + ltx ippSRSout all : ippSCD.pdf ippSRS.pdf ippSDRS.pdf Index: /trunk/doc/design/ippSDRS.tex =================================================================== --- /trunk/doc/design/ippSDRS.tex (revision 1398) +++ /trunk/doc/design/ippSDRS.tex (revision 1399) @@ -1,3 +1,3 @@ -%%% $Id: ippSDRS.tex,v 1.3 2004-06-25 03:05:31 eugene Exp $ +%%% $Id: ippSDRS.tex,v 1.4 2004-08-06 19:06:01 eugene Exp $ \documentclass[panstarrs]{panstarrs} @@ -2029,4 +2029,10 @@ \subparagraph{Combine Images} +\tbd{for moving objects and images which are not simultaneous, do we + identify the moving objects?} + +\tbd{use the spatial information? fit a 2-D Nth order polynomial to + the collection of pixels and then look for outliers} + The first module for Phase 4 is to combine the images from each telescope, rejecting artifacts such as cosmic rays and low altitude @@ -2086,4 +2092,6 @@ \subparagraph{Transient Identification} +\tbd{what about different stellar colors?} + This module identifies variable/moving sources. The inputs are: \begin{enumerate} @@ -2134,4 +2142,6 @@ \subparagraph{Add to Static Sky} + +\tbd{how to handle variable stars?} This module adds the combined sky cell image into the static sky, so Index: /trunk/doc/design/ippSRS.tex =================================================================== --- /trunk/doc/design/ippSRS.tex (revision 1398) +++ /trunk/doc/design/ippSRS.tex (revision 1399) @@ -1,4 +1,4 @@ -%%% $Id: ippSRS.tex,v 1.6 2004-06-24 20:24:27 eugene Exp $ -\documentclass[panstarrs]{panstarrs} + %%% $Id: ippSRS.tex,v 1.7 2004-08-06 19:06:01 eugene Exp $ +\documentclass[panstarrs,spec]{panstarrs} % basic document variables @@ -7,4 +7,5 @@ \shorttitle{IPP SRS} \author{Eugene Magnier, Paul A. Price, Josh Hoblitt} +\audience{Pan-STARRS PMO} \group{Pan-STARRS Algorithm Group} \project{Pan-STARRS Image Processing Pipeline} @@ -12,4 +13,14 @@ \version{DR} \docnumber{PSDC-430-005} + +\newcommand\FRM[2]{\parbox[t]{#1}{\raggedright #2}} +\newcommand\FRA{100pt} +\newcommand\FRB{260pt} +\newcommand\FRC{40pt} +\newcommand\FRD{80pt} +\newcommand\FRN[1]{\FRM{50pt}{#1}} +\newcommand\FRS[1]{\FRM{190pt}{#1}} +\newcommand\VER[2]{\\ {\scriptsize QUALIFICATION METHOD: #1, TRACE: #2}} +\newcommand\TASK{\\ {\scriptsize TASK}} % allow paragraphs to be listed in TOC for now @@ -30,6 +41,8 @@ \TBDsStart -% section page TBR number Description -section & page & TBR & description \\ \hline +Section & Page & Number & Description \\ \hline + & & & choice of scripting language \\ \hline + & & & coding standards for scripting language \\ \hline + & & & low-spatial frequency sky model details \\ \hline \TBDsEnd @@ -47,14 +60,58 @@ \subsection{Identification} -This document establishes the software requirements for the Pan-STARRS -Image Processing Pipeline (IPP) as applied to Pan-STARRS 1 (PS-1), the -initial demonstration telescope to be constructed on Haleakala by Jan -2006. +This document is the Software Requirements Specification (SRS) for the +Panoramic Survey Telescope and Rapid Response System (Pan-STARRS) +Image Processing Pipeline (IPP) for the prototype telescope PS-1, and +is a System-level controlled specification/design description document +in the official Pan-STARRS engineering specification tree. \subsection{System Overview} -\tbd{description of the Pan-STARRS System and PS-1.} +The Institute for Astronomy at the University of Hawaii is developing +a large optical synoptic survey telescope system, the Panoramic Survey +Telescope and Rapid Response System (Pan-STARRS). The science goals, +priorities, top-level concept of operations with associated +operational requirements, and system performance drivers with +associated system performance requirements are described in the +Pan-STARRS Science Goals Statement (SGS). As described in this +document, The system conceptual design for Pan-STARRS utilizes an +array of four 1.8m telescopes each with a 7 degree$^2$ field of view, +giving the system an \'etendue larger than all existing survey +instruments combined (defined as the product of the collecting area +$A$ multiplied by the field-of-view solid angle $\Omega$). Each +telescope will be equipped with a 1 billion pixel CCD camera with low +noise and rapid read-out, and the data will be reduced in near real +time to produce both cumulative static sky and difference images from +which transient, moving, and variable objects can be +detected. Pan-STARRS will be able to survey up to $\approx 6,000$ +degree$^{2}$ per night to a detection limit of approximately 24$^{th}$ +magnitude. This unique combination of sensitivity and sky coverage +will open up many new possibilities in time domain astronomy including +a major goal of surveying the Potentially Hazardous Object (PHO) +population down to a diameter of $\approx 300$ meters. In addition, +the Pan-STARRS data will be used to investigate a broad range of +astronomical problems of extreme current interest concerning the Solar +System, the Galaxy, and the Cosmos at large. A prototype single +telescope system, PS-1, is being developed as a preliminary step +before construction of the complete four telescope system. + +\begin{tabular}{ll} +Project sponsor:& AFRL, United States Air Force \\ +Acquirer: & University of Hawaii Institute for Astronomy \\ +User: & Astronomical community \\ +Developer: & University of Hawaii Institute for Astronomy, participating \\ + & institutions, and associated subcontractors +\end{tabular} \subsection{Document Overview} + +The Pan-STARRS IPP Software Requirements Specification contains the +complete system requirements of the Pan-STARRS PS-1 IPP in order to +achieve the top-level performance and operational requirements +specified by the SCD. The requirements flow begun in the SGS and +continued in the SCD is further developed in this SRS to provide +additional derived system and subsystem requirements. + +\subsection{Requirements Definitions} The Pan-STARRS document naming scheme is PSDC-NNN-MMM-VV, where the VV @@ -63,23 +120,18 @@ that series is implied. -Open issues (TBDs) in this document are marked \tbd{in bold, red with -surrounding square brackets}. - -Quantities which should be reviewed (TBRs) are marked \tbr{in bold, -blue with surrounding square brackets}. - -\subsubsection{Requirements Definitions} - -\paragraph{``Must''} When used in this specification, the word -``must'' refers to an explicit requirement of a system component or -the complete system. In this document, the use of the word ``must'' -replaces, and is equivalent to, use of the word ``shall'' found in -many requirements documents. - -\paragraph{``Should''} When used in this specification, the word +Open issues (TBDs) in this document are marked \tbd{in bold red}. + +Quantities which should be reviewed (TBRs) are marked \tbr{in bold +blue}. + +\subsubsection{``Shall''} When used in this specification, the word +``shall'' refers to an explicit requirement of a system component or +the complete system. + +\subsubsection{``Should''} When used in this specification, the word ``should'' refers to a desired characteristic of a system component or the complete system. -\paragraph{``Will''} When used in this specification, the word +\subsubsection{``Will''} When used in this specification, the word ``will'' provides information about a characteristic of a related system component or a complete related system. @@ -102,150 +154,142 @@ \section{Requirements} -\subsection{Science Requirements} +\subsection{Top-Level Requirements} \label{req:system-capabilities} -The IPP must perform the following tasks: - -\begin{enumerate} - -\item Accept raw images from the summit at a sustained rate of 1 - exposure (2~GB) per 30 seconds. - -\item Accept metadata from the summit at a sustained rate of \tbr{1 MB - per second}. - -\item Produce master calibration images from the raw calibration - images. The master calibration images must not introduce systematic - uncertainties in the photometry greater than \tbr{0.2\%}. - -\item Pre-process the science images with the master calibration - images. - -\item Merge multiple pre-processed science images -- from multiple - telescopes or from sequential, dithered exposures -- into stacked - images with corresponding signal-to-noise maps. Pixels from the - input images which are outliers for the ensemble of corresponding - pixels must be excised. - -\item Subtract a static sky image from the stacked images to produce - an image of only the transient objects. - -\item Excise transients and outliers which exceed a user-configurable - threshold in the subtracted image from the pre-processed science - images. - -\item Merge the cleaned images into the static sky image, and update - the corresponding exposure (S/N) maps. - -\item Detect and measure parameters of objects on the four types of - images: pre-processed images, the stacked image, the difference - image, and the static sky image. - -\item Determine astrometry of the detected objects relative to an - astrometric reference. For the Commissioning phase of PS-1, the - astrometric calibration will be limited by the determination of the - optical model of the focal plane, and may be as poor as \tbr{750 - mas}. For the AP reference construction phase of PS-1, after the - optical model has been measured, the astrometry solution must be - limited by the reference catalog in use, and will be in the vicinity - of \tbr{75 mas (UCAC) - 250 mas (USNO B1.0)}. After the construction - of the AP astrometric reference catalog, the accuracy will be limited - by atmospheric variations, and must be no worse than \tbr{50 mas}, - with a goal of \tbr{10 mas}. - -\item Determine photometry of the detected objects, both within an - internal photometric system and in terms of appropriate external - photometric reference systems. For the Commissioning phase, the - accuracy of the photometric calibration will be limited by the - quantity and quality of the standard star observations, and the - consistency of the flat-field images across the camera; the scatter - must be less than \tbr{25 millimags}. During the AP reference - construction phase of PS-1, after the flat-field correction has been - measured, the photometric accuracy will be limited by the standard - star observations, the zero-point determinations, and in the case of - calibration to the external standard, the color corrections. The - photometric accuracy in this stage must be better than \tbr{10 - millimags}. After the construction of the AP Reference Catalog, the - photometric accuracy will be limited by knowledge of the flat-field, - variations in the atmosphere across the field, and the reference - catalogs. The photometric scatter in photometric weather must be - better than \tbr{5 millimag} for relative photometry (relative to the - internal filter system) and \tbr{10 millimag} for absolute photometry - (relative to other filter systems such as the SDSS filters). - -\item Produce a high-quality astrometric reference catalog from the - extracted objects within 6 months of the end of the AP Survey. The - astrometric reference must have an absolute accuracy of \tbr{30 mas} - and a local relative accuracy of \tbr{10 mas}. Proper motions of - detected non-solar-system objects must be determined with an - accuracy of \tbr{20 mas / year} for unsaturated, bright stars. - -\item Produce a high-quality photometric reference catalog from the - extracted point-source objects within 6 months of the end of the AP - Survey. The photometric reference must have an consistency across - the sky of \tbr{5 millimag} and an absolute calibration to the - external system (defined by \tbr{SDSS} and the CFHT Legacy Survey - Standards) with an accuracy of \tbr{10 millimag}. - -\item Publish the static sky images to the Pan-STARRS published static - sky server on a time-scale of \tbr{1 month}. - -\item Publish the detected objects to the Pan-STARRS published object - database on a time-scale of \tbr{1 week}. - -\item Provide access to external Pan-STARRS clients to the detected - objects on time-scales of \tbr{10 minute} after the image is - obtained.\comment{this is derived from the top-level science - requirement.} - -\item Store the raw images for a period of time which depends on the - survey source of the data. In PS-1, the AP and IVP Survey data must - be stored for the lifetime of the project. Other raw data must be - stored for \tbr{1 month}. - -\item Store the detected objects for a period of time, depending on - the type of detection. Transients from the P4$\Delta$ images may be - excised after \tbr{6 months}. - +The Pan-STARRS System Concept Definition (SCD) specifies the derived +top-level requirements for the IPP, which we reproduce here (with +numbering consistent with this document): + +\begin{enumerate} +\item Produce reduced science images for each full camera exposure + which are photometrically consistent across the field to within 1\%.\VER{ANALYSIS}{SCD:3.2.2.5} + \label{TLR:1} + +\item Produce reduced science images for each full camera exposure + which are photometrically calibrated to within 1\%.\VER{ANALYSIS}{SCD:3.2.2.5} + \label{TLR:2} + +\item Produce reduced science images for each full camera exposure + which are astrometrically calibrated to 100 milliarcseconds to an + absolute reference.\VER{ANALYSIS}{SCD:3.2.2.6} + \label{TLR:3} + +\item Produce reduced science images for each full camera exposure + which are astrometrically consistent to 30 + milliarcseconds.\VER{ANALYSIS}{SCD:3.2.2.7} + \label{TLR:4} + +\item Produce reduced science images for each full camera exposure + which have foreground emission subtracted with no more than 1\% + variation in the non-astronomical background.\VER{ANALYSIS}{SCD:3.5.12} + \label{TLR:5} + +\item Merge all $g$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} + \label{TLR:6} + +\item Merge all $r$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} + \label{TLR:7} + +\item Merge all $i$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} + \label{TLR:8} + +\item Merge all $z$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} + \label{TLR:9} + +\item Merge all $y$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} + \label{TLR:10} + +\item Merge all $w$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} + \label{TLR:11} + +\item Detect and classify objects on the individual processed science images.\VER{TEST}{SCD:3.2.2.16} + \label{TLR:12} + +\item Detect and classify objects on the stacked groups of science images.\VER{TEST}{SCD:3.2.2.16} + \label{TLR:13} + +\item Detect and classify objects on the static sky image.\VER{TEST}{SCD:3.2.2.16} + \label{TLR:14} + +\item Detect all significant transients in the individual science + images relative to the static sky image.\VER{TEST}{SCD:3.2.2.16} + \label{TLR:15} + +\item Degrade the stacked image by no more than \tbr{10 milliarcseconds}.\VER{ANALYSIS}{SCD:3.5.2} + \label{TLR:16} + +\item Perform the processing of science images to the level of + transient detection and static sky inclusion at a rate such that + exposures taken at a cadence of \tbr{40} seconds do not accumulate + in the processing buffer.\VER{TEST}{SCD:3.2.2.3} + \label{TLR:17} + +\item Limit the false alarm rate (FAR) to less than \tbr{5\%} for + transient detections $> 5\sigma$ sent to the preferred client science + pipelines.\footnote{note difference with PS-4: 1\%} + \VER{ANALYSIS}{SCD:3.2.2.13} + \label{TLR:18} + +\item Publish the static sky images to the Pan-STARRS Published + Science Products Subsystem (PSPS) once per \tbr{6 + months}.\VER{TEST}{SCD:3.2.2.18} + \label{TLR:19} + +\item Publish the detected objects to the Pan-STARRS Published Science + Products Subsystem (PSPS) once per month.\VER{TEST}{SCD:3.2.2.18} + \label{TLR:20} + +\item Send the IPP metadata and received OTIS metadata to the + Pan-STARRS Published Science Products Subsystem (PSPS) weekly.\VER{TEST}{SCD:3.2.2.18} + \label{TLR:21} + +\item Provide access to preferred Pan-STARRS science clients to the + detected transient objects within \tbr{5 minutes}.\VER{TEST}{SCD:3.5.10} + \label{TLR:22} + +\item Provide sufficent storage volume for \tbr{1 year} of raw images + from the AP and IVP Surveys.\footnote{note difference with PS-4: 1 + month of raw images} \VER{INSPECT}{allocated} + \label{TLR:23} + +\item Provide sufficient storage volume for all detections from the + AP, IVP, and MVP Surveys.\footnote{note difference with PS-4: 1 + year of detections}\VER{INSPECT}{allocated} + \label{TLR:24} + +\item Provide sufficient storage volume for \tbr{2 year} of + metadata.\footnote{note difference with PS-4: 10 + years of metadata}\VER{INSPECT}{allocated} + \label{TLR:25} \end{enumerate} \subsection{Required States} -The IPP must have 3 states: active, paused, and interactive. - -\subsubsection{Active State} -\label{req:active-state} - -In active state, the IPP must: - -\begin{enumerate} -\item Accept images and metadata from the external sources (i.e., the - summit) - -\item Automatically perform the complete set of image processing - tasks, including both calibration and science image processing. - -\item Respond to requests for data from client science pipelines. - -\item Respond to analysis priority requests issued by the IPP users. -\end{enumerate} - -\subsubsection{Paused State} -\label{req:paused-state} - -In paused state, the IPP must refuse incoming data and metadata and -data requests from the client science pipelines. - -\subsubsection{Interactive State} -\label{req:interactive-state} - -In interactive state, the IPP must: - -\begin{enumerate} -\item Accept incoming data and metadata from the external sources. -\item Not automatically process the data -\item Respond to user commands to initiate portions of the data - analysis. -\end{enumerate} +The IPP has 3 operating states: active, paused, and interactive. In active state, the IPP: + +\begin{itemize} +\item Accepts images and metadata from the external sources (i.e., the summit) + +\item Automatically performs the complete set of image processing + tasks, including both calibration and science image + processing. + +\item Responds to requests for data from client science pipelines, +possibly pre-registered classes of data requests. + +\item Responds to analysis priority requests issued by the IPP operators. +\end{itemize} + +In paused state, the IPP refuses incoming data and metadata and data +requests from the client science pipelines. + +The interactive state is intermediate between these two. In +interactive state, the IPP: + +\begin{itemize} +\item Accepts incoming data and metadata from the external sources. +\item Does {\em not} automatically process the data. +\item Responds to user commands to perform portions of the data analysis. +\end{itemize} \subsection{Software Coding Requirements} @@ -255,18 +299,16 @@ \begin{enumerate} -\item Source code must be in C. -\item All source code must be compiled with `gcc' version v2.95 or higher. -\item The tested compiler version must be defined for the delivered software product. -\item Scripting language must be \tbd{Python}, version X.X. +\item Source code shall be in C. \VER{INSPECT}{allocated} +\item All source code shall be tested with `gcc' version v2.95 or higher. \VER{INSPECT}{allocated} +\item The tested compiler version shall be defined for the delivered software product. \VER{INSPECT}{allocated} +\item Scripting language shall be Perl. \VER{INSPECT}{allocated} \end{enumerate} \subsubsection{Interfaces} \begin{enumerate} -\item Access to low-level Library functions must be provided via C -APIs consisting of the function calls and the defined data structures -and other data types. -\item Access to high-level functions must be provided via C APIs as -well as SWIG interfaces, where specified. -\item Access to processing jobs must be available via the UNIX shell. +\item Access to low-level Library functions shall be provided via C APIs consisting of the function calls and the defined data structures and other data types. \VER{INSPECT}{allocated} +\item Access to high-level functions shall be provided via C APIs. \VER{INSPECT}{allocated} +\item Access to specified C functions in higher level languages shall employ SWIG. \VER{INSPECT}{allocated} +\item Access to processing jobs shall be available via the UNIX shell. \VER{INSPECT}{allocated} \end{enumerate} @@ -274,15 +316,9 @@ \begin{enumerate} -\item The C code must comply with ANSI Standard C99. -\item Because the delivered code is required to run on UNIX machines, -the delivered code must be in compliance with the language-independent -UNIX operating system standard POSIX (Open Group Based Specifications -Issue 6, IEEE Std 1003.1, 2004). -\item Source code files must use the UNIX line-break -convention (line-feed only). -\item C coding style must adhere to the standard defined in the -document 'Pan-STARRS C-coding standard' (PSDC-430-004). -\item \tbd{Python} coding must follow the standard defined in the -document \tbd{TBD}. +\item The C code shall comply with ANSI Standard C99. \VER{INSPECT}{allocated} +\item Because the delivered code is required to run on UNIX machines, the delivered code shall be in compliance with the language-independent UNIX operating system standard POSIX (Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2004).\VER{INSPECT}{allocated} +\item Source code files shall use the UNIX line-break convention (line-feed only). \VER{INSPECT}{allocated} +\item C coding style shall adhere to the standard defined in the document `Pan-STARRS C-coding standard' (PSDC-430-004). \VER{INSPECT}{allocated} +\item Perl coding shall follow the standard defined in the document `Pan-STARRS Perl-coding standard' \tbd{(PSDC-430-0XX)}.\VER{INSPECT}{allocated} \end{enumerate} @@ -290,42 +326,42 @@ \begin{enumerate} -\item Header files must have names starting \code{ps} or \code{p_ps} -for private interface definitions. The latter must appear in a +\item Header files shall have names starting \code{ps} or \code{p_ps} +for private interface definitions. The latter shall appear in a subdirectory \code{private} of whichever directory is being searched -for the public header files. +for the public header files.\VER{INSPECT}{allocated} \item Functions visible at global scope that are part of the public -API must have names beginning with \code{ps} and follow the naming -conventions in the coding standard. +API shall have names beginning with \code{ps} and follow the naming +conventions in the coding standard. \VER{INSPECT}{allocated} \item Functions visible at global scope but which are not part of the -public interface must have names beginning with \code{p_ps}. +public interface shall have names beginning with \code{p_ps}.\VER{INSPECT}{allocated} -\item Functions that are local to a file must \textit{not} start with -\code{ps} or \code{p_ps}. +\item Functions that are local to a file shall \textit{not} start with +\code{ps} or \code{p_ps}.\VER{INSPECT}{allocated} \item Variables visible at global scope which are part of the public -API must have names beginning with \code{ps}, and follow the naming -conventions in the coding standard. +API shall have names beginning with \code{ps}, and follow the naming +conventions in the coding standard. \VER{INSPECT}{allocated} \item Variables that are visible at global scope but which are not -part of the public interface must have names beginning with -\code{p_ps}. - -\item Variables that are local to a file must \textit{not} start with -\code{ps} (or \code{p_ps}). +part of the public interface shall have names beginning with +\code{p_ps}.\VER{INSPECT}{allocated} + +\item Variables that are local to a file shall \textit{not} start with +\code{ps} (or \code{p_ps}).\VER{INSPECT}{allocated} \item The names of all enumerated types and C-preprocessor symbols -(but not variables declared \code{const}) must start with \code{PS_}, +(but not variables declared \code{const}) shall start with \code{PS_}, in the case of public symbols, or \code{P_PS_}, for private symbols. -The rest of the name must be uppercase with words separated by +The rest of the name shall be uppercase with words separated by underscores (\code{_}). An exception is the case of system utilities -implemented as macros, in which case the names must conform to the -convention for function names. +implemented as macros, in which case the names shall conform to the +convention for function names.\VER{INSPECT}{allocated} \item When defining a function to convert from one type to another, -the name must be of the form \code{psOldToNew}, +the name shall be of the form \code{psOldToNew}, e.g.\code{psEquatorialToEcliptic} (\emph{not} -\code{psEquatorial2Ecliptic}). +\code{psEquatorial2Ecliptic}).\VER{INSPECT}{allocated} \end{enumerate} @@ -333,34 +369,22 @@ \begin{enumerate} -\item Functions that assign to a variable must list that argument - \textit{first}, following the pattern of \code{strcpy}. For - example: - \begin{verbatim} - void psVectorCopy(restrict psVector *out, const restrict psVector *in); - \end{verbatim} - -\item Type definitions should always be accompanied by prototypes for - their constructors and destructors, following these guidelines: - -\begin{enumerate} - \item The constructor name should consist of the type name followed - by \code{Alloc}; e.g. a type \code{psImage} would be created by a - function \code{psImage *psImageAlloc();}. - - \item The type should be freed with a destructor named - \code{typeFree}, e.g. \code{void psImageFree(psImage *image);}. - - \item The constructor must never return \code{NULL}, and no code - calling the constructor should ever check the return value. - - \item The destructor must not return a value. - - \item The destructor must handle being passed \code{NULL} by simply - returning immediately. This must not be treated as an error - condition. - - \item Constructors and Destructors should use the memory reference - counter facilities of the PSLib memory management system. -\end{enumerate} +\item Functions that assign to a variable shall list that argument +\textit{first}, following the pattern of \code{strcpy}. \VER{INSPECT}{allocated} + +Type definitions should always be accompanied by prototypes for their +constructors. Corresponding destructors are private functions +registered with the PSLib memory management system. + +\item The constructor name shall consist of the type name followed by +\code{Alloc}; e.g. a type \code{psImage} would be created by a +function \code{psImage *psImageAlloc();}.\VER{INSPECT}{allocated} + +\item The constructor shall never return \code{NULL}, so code calling +the constructor should not check the return value.\VER{INSPECT}{allocated} + +\item The destructor shall not return a value.\VER{INSPECT}{allocated} + +\item Constructors and Destructors shall use the memory reference + counter facilities of the PSLib memory management system.\VER{INSPECT}{allocated} \end{enumerate} @@ -368,36 +392,36 @@ \begin{enumerate} -\item Commenting of delivered C code must follow the C coding +\item Commenting of delivered C code shall follow the C coding standards and provide tags for Doxygen interpretation of the - comments and program structures. - -\item Commenting of delivered Python code must follow the Python - coding standards. - -\item Source code documentation must be generated with Doxygen from - the in-line comments and must be provided as HTML, Latex, and man - pages. + comments and program structures.\VER{INSPECT}{allocated} + +\item Commenting of delivered Perl code shall follow the Perl + coding standards.\VER{INSPECT}{allocated} + +\item Source code documentation shall be generated with Doxygen from + the in-line comments and shall be provided as HTML, Latex, and man + pages. \VER{INSPECT}{allocated} \item User documentation includes the API usage for the modules and library functions as well as user interface description for the - higher-level architectural systems. User documentation must be - delivered as PDF documents. + higher-level architectural systems. User documentation shall be + delivered as PDF documents.\VER{INSPECT}{allocated} \end{enumerate} \subsubsection{Version Control} -Source code version control must be implemented with CVS. +Source code version control shall be implemented with CVS. \VER{INSPECT}{allocated} \subsubsection{CSCI Deliverable} -All final source code generated for the IPP must be delivered via CVS, -including the test code. CVS revision history must be included and -made available via CVS. +All final source code generated for the IPP shall be delivered via CVS, +including the test code. CVS revision history shall be included and +made available via CVS.\VER{INSPECT}{allocated} \subsubsection{Platform architectures and operating systems} -Makefiles must be provided with appropriate flags set so that all -code compiles without warnings under 'gcc -Wall' for the following -platform architectures and operating systems: +Makefiles shall be provided with appropriate flags set so that all +code compiles without warnings under `gcc -Wall' for the following +platform architectures and operating systems:\VER{INSPECT}{allocated} \begin{itemize} @@ -410,5 +434,5 @@ such as those caused by lex-generated code. -Although the code must compile successfully under both listed +Although the code shall compile successfully under both listed operating systems, unit testing should only be performed for the x86/Linux combination. @@ -416,15 +440,34 @@ \subsubsection{Timing measurements} -Timing requirements specified in this document must be achieved on the -deployed Pan-STARRS analysis computers. +Timing requirements specified in this document shall be achieved on the +deployed Pan-STARRS analysis computers.\VER{TEST}{allocated} \subsubsection{Software Configuration} -\tbd{deferred} +\paragraph{Version Management} + +The IPP software configuration management system shall ensure that +validated versions of both internal and external software are used +when the software is compiled.\VER{TEST}{allocated} + +\paragraph{Optional Modes} + +The IPP software configuration management system shall provide +optionally selected software version sets under compilation +conditions. For example, compilation of the software for test +purposes with a non-standard FFT tool shall be an +option.\VER{TEST}{allocated} \subsection{Architectural Components} +\begin{figure} +\begin{center} +\resizebox{6in}{!}{\includegraphics{pics/IPPoverview.ps}} +\caption{ \label{overview} IPP System Overview} +\end{center} +\end{figure} + As discussed in the Pan-STARRS System Concept Definition -(PSDC-xxx-xxx), the IPP is organized into a number of clearly-defined +(PSDC-250-002), the IPP is organized into a number of clearly-defined software elements. The SCD provides a detailed description of the roles and responsibilities of these subsystems. In brief, the IPP @@ -439,27 +482,27 @@ \begin{itemize} -\item {\bf Image Server:} This component is a large data store for all +\item {\bf IPP Image Server:} This component is a large data store for all images used by the IPP, including the raw images from the telescope, the master calibration images, the reference static-sky images, and any temporary image data products produced by the IPP. The Image - Server is required to meet all of the image storage needs identified - in the top-level requirements above. The Image Server may also store - large data files which do not contain imaging data. The Image Server - must accept the incoming data and store it until it is no longer - needed by other portions of the IPP. + Server may also store large data files which do not contain imaging + data. The Image Server accepts the incoming data and stores it until + it is no longer needed by other portions of the IPP. \item {\bf Astrometry \& Photometry Database (AP):} This component is - required to store and manipulate astronomical objects detected in - images processed by the IPP, including individual measurements of - objects on the images, the summary information about those objects, - and reference object data. - -\item {\bf Metadata Database:} This component is required to store the - all other data which are neither image files nor astronomical object + used to store and manipulate astronomical objects detected in images + processed by the IPP, including individual measurements of objects on + the images, the summary information about those objects, and + reference object data. It includes descriptive information about the + images, filter, cameras, telescopes, and other aspects of the system + needed to interpret the object data. + +\item {\bf IPP Metadata Database:} This component is used to store all + other data which are neither image files nor astronomical object data. The Metadata Database is the authoritative source for all metadata data, including metadata which may be duplicated elsewhere, such as in the headers of images in the image database. -\item {\bf Controller:} In order to perform the analysis stages +\item {\bf IPP Controller:} In order to perform the analysis stages required by the IPP, it is necessary to use distributed computing processes on a large number of computers. The Controller is required @@ -467,5 +510,5 @@ machines. -\item {\bf Scheduler:} This component is a decision-making mechanism +\item {\bf IPP Scheduler:} This component is a decision-making mechanism required to guide the operation of the IPP: to evaluate the currently available collection of data, to identify the necessary analysis, and @@ -476,12 +519,6 @@ The relationship between these software elements is shown in Figure~\ref{overview}. This figure also shows the interactions -between the IPP and other Pan-STARRS systems. - -\begin{figure} -\begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/overview}} -\caption{ \label{overview} IPP System Overview} -\end{center} -\end{figure} +between the IPP and other Pan-STARRS systems. The following sections +identify requirements of these five software elements. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -490,51 +527,57 @@ \begin{enumerate} -\item The IPP Image Server must store images on a distributed +\item The IPP Image Server shall accept raw images from the summit at + a sustained rate of 1 exposure (2~GB) per \tbr{40 + seconds}. \VER{TEST}{TLR:17, TLR:23} + +\item The IPP Image Server shall store images on a distributed collection of computer disks. Individual instances of a file are only required to be stored on a single machine (striping across - computers is not a requirement). - -\item The IPP Image Server must be capable of honoring requests to - store an image on a specific machine. + computers is not a requirement).\VER{TEST}{TLR:17, TLR:23} + +\item The IPP Image Server shall attempt to store an image on a + specific machine if requested by the user.\VER{TEST}{TLR:17, TLR:23} \item If such a request cannot be honored (ie, the machine is down), - the IPP Image Server must select an appropriate machine and notify - the requesting agent of the new location. - -\item The IPP Image Server must store multiple copies of each image + the IPP Image Server shall select an appropriate machine and notify + the requesting agent of the new location.\VER{TEST}{TLR:17, TLR:23} + +\item The IPP Image Server shall store multiple copies of each image upon request, the number of copies specified independently for each - file by the user. - -\item The IPP Image Server must maintain a record of all image copies - currently available in the repository. This record must include the - image name, location (which machine), the image size, and the state - of the image (available, locked, deleted). - -\item The IPP Image Server must lock images in the repository on - request. Both read (shared) and write (exclusive) locks must be - provided. A read lock must prevent write access to the file; a - write lock must prevent both read and write access. - -\item The IPP Image Server must return the image location (the - computer on which it resides) upon request. - -\item The IPP Image Server must provide a specified image upon request. - -\item The IPP Image Server must delete images in the repository on request. - -\item The IPP Image Server must accept images from the summit at the - maximum rate of 1 full-camera image every 30 seconds. The IPP Image - Server must therefore accept new images into the repository at a - rate of 64 raw OTAs in 30 seconds and a total input data volume rate - of 75 MB/sec. -\end{enumerate} - -\tbd{archive lifetime} - -\tbd{reliability} - -\tbd{backups} + file by the user.\VER{TEST}{TLR:17, TLR:23} + +\item The IPP Image Server shall maintain a record of all image copies + currently available in the repository. This record shall include at + least the image name, location (which machine), the image size, and + the state of the image (available, locked, deleted).\VER{INSPECT}{TLR:17, TLR:23} + +\item The IPP Image Server shall lock images in the repository on + request. Both read (shared) and write (exclusive) locks shall be + provided. A read lock shall prevent write access to the file; a + write lock shall prevent both read and write access. \tbr{Access + prevention may be advisory rather than enforced.} \VER{TEST}{TLR:17, TLR:23} + +\item The IPP Image Server shall return the image location (the + computer or computers on which it resides) upon request.\VER{TEST}{TLR:17, TLR:23} + +\item The IPP Image Server shall provide a specified image upon request.\VER{TEST}{TLR:17, TLR:23} + +\item The IPP Image Server shall delete images in the repository on request.\VER{TEST}{TLR:17, TLR:23} + +\end{enumerate} \subsubsection{AP Database} + +The purpose of the AP Database is: +\begin{itemize} +\item to enable the photometric calibration of images +\item to enable the astrometric calibration of images +\item to enable the construction of flat-field correction frames +\item to enable the construction of a photometric calibration catalog +\item to enable the construction of an astrometric calibration catalog +\item to monitor the system photometry calibration parameters +\item to monitor the system astrometry calibration parameters +\item to perform the identification of single-occurance transients +\end{itemize} \begin{table} @@ -544,19 +587,15 @@ \hline \hline -Object Parameter & P2 & P4S & P4D & SS \\ +Object Parameter & P2 & P4$\Sigma$ & P4$\Delta$ & SS \\ \hline -PSF x,y, M, $\sigma_{\rm M}$ & + & + & + & + \\ -$\sigma_x$, $\sigma_y$, covar. & + & + & + & + \\ -exp. spaced aps., Poisson noise, variance & - & - & - & + \\ -streak L, $\phi$, $\sigma_L$, $\sigma_\phi$ & - & - & + & + \\ -$x_g$, $y_g$, flag & + & + & - & + \\ -local sky data & + & + & + & + \\ -Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ -S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ -W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ -star/gal sep, star/streak sep. & - & + & + & + \\ -\hline -deVeucaleur R, M, AB, $\phi$ & - & + & - & + \\ -exponential R, M, AB, $\phi$ & - & + & - & + \\ +PSF x,y, covar, $\alpha,\delta$ & + & + & + & + \\ +PSF mag, $\sigma_{\rm mag}$ & + & + & + & + \\ +star/gal sep & + & + & + & + \\ +$\sigma_x$, $\sigma_y$, $\theta$ & + & + & + & + \\ +local sky data & + & + & + & + \\ +Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ +S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ +W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ +exp. spaced aps., Poisson noise, variance & - & - & - & + \\ \hline \end{tabular} @@ -565,27 +604,28 @@ \begin{enumerate} -\item The AP Database must accept and store individual detections and +\item The AP Database shall accept and store individual detections and collections of detections along with information about the image - which provided the detections. - -\item Detections must be saved as one of several detection classes - (P2, P4S, P4D, SS) and the AP Database must store the appropriate - parameters, listed in Table~\ref{APdetections}, for each class. - -\item The AP Database must identify the image which provided the + which provided the detections.\VER{TEST}{TLR:2, TLR:3, TLR:22, TLR:24} + +\item Detections shall be saved as one of several detection classes + (P2, P4$\Sigma$, P4$\Delta$, SS) and the AP Database shall store the + appropriate parameters, listed in Table~\ref{APdetections}, for each + class.\VER{TEST}{TLR:2, TLR:3, TLR:22, TLR:24} + +\item The AP Database shall identify the image which provided the detection, or in the case of external references, an identifier - specific to the reference source. - -\item The AP Database must group detections into objects and measure - average parameters of those objects. - -\item The AP Database must store parallax and proper motion parameters - for a subset of the average objects. - -\item The AP Database must store image and filter calibration + specific to the reference source.\VER{TEST}{TLR:2, TLR:3} + +\item The AP Database shall group detections into objects and measure + average parameters of those objects.\VER{ANALYSIS}{TLR:2, TLR:3, TLR:22} + +\item The AP Database shall store parallax and proper motion parameters + for a subset of the average objects.\VER{TEST}{TLR:2, TLR:3, TLR:22} + +\item The AP Database shall store image and filter calibration information necessary to convert between instrumental magnitudes and - calibrated magnitudes in standard systems. - -\item The AP Database must perform at least the follow queries, with + calibrated magnitudes in standard systems.\VER{INSPECT}{TLR:3} + +\item The AP Database shall perform at least the follow queries, with constraints on the output based on at least time ranges, magnitude limits, error limits: @@ -593,79 +633,79 @@ \begin{enumerate} \item given $(RA,DEC)$ and a Radius, return all objects and/or - detections in the region. + detections in the region.\VER{TEST}{TLR:2, TLR:3} \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all objects and/or - detections in the region. - - \item given $(RA,DEC)$, return closest object. - - \item given object ID, return all detections - - \item given detection, return source image data. - - \item given detection, return object. - - \item given $(RA,DEC)$, return all images overlapping coordinate. - - \item given $(RA,DEC)$ and a Radius, return all images overlapping region. - - \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping - region. + detections in the region.\VER{TEST}{TLR:2, TLR:3} + + \item given $(RA,DEC)$, return closest object.\VER{ANALYSIS}{TLR:2, TLR:3, TLR:22} + + \item given object ID, return all detections\VER{TEST}{TLR:2, TLR:3} + + \item given detection, return source image data.\VER{TEST}{TLR:2, TLR:3} + + \item given detection, return object.\VER{TEST}{TLR:2, TLR:3, TLR:22} + + \item given $(RA,DEC)$, return all images overlapping coordinate.\VER{ANALYSIS}{TLR:2, TLR:3} + + \item given $(RA,DEC)$ and a Radius, return all images overlapping region.\VER{ANALYSIS}{TLR:2, TLR:3} + + \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping region.\VER{ANALYSIS}{TLR:2, TLR:3} \item given detection instrumental magnitude, return derived - magnitudes based on calibration information. + magnitudes based on calibration information.\VER{TEST}{TLR:2, TLR:3} \item given a collection of detections in a filter, determine the - object average magnitude in that filter. + object average magnitude in that filter.\VER{ANALYSIS}{TLR:2, TLR:3} \item given a collection of objects and detections, determine the - individual image zero-points. + individual image zero-points.\VER{ANALYSIS}{TLR:2, TLR:3} \item given a region, return all possible combinations of the object - or detection magnitudes $(M_1 - M_2)$. - - \item given a list of $(RA,DEC)$ entries, return all nearest objects. + or detection magnitudes $(M_1 - M_2)$.\VER{TEST}{TLR:2, TLR:3} + + \item given a list of $(RA,DEC)$ entries, return all nearest objects.\VER{ANALYSIS}{TLR:2, TLR:3} \item given a filter, telescope, or detector, return all calibration - terms and history. + terms and history.\VER{TEST}{TLR:2, TLR:3} \item given a detection, return all non-detections from images which - overlapped the detection coordinates. + overlapped the detection coordinates.\VER{ANALYSIS}{TLR:2, TLR:3, TLR:22} \end{enumerate} -\item The AP Database must accept detection IDs of moving objects and - label the detections with the identified object. - -\item The AP Database must accept new detections at the rate generated - by the telescope from the Phase 2 and Phase 4 analysis. Except - within 10 degrees of the galactic plane, the AP Database must keep - up with the incoming rates. The expected rates are listed in - Table~\ref{APrates}, along with the total data volume required for - storage space over the PS-1 lifetime. - -\end{enumerate} - -\tbd{archive lifetime} - -\tbd{reliability} - -\tbd{backups} +\item The AP Database shall accept detection IDs of moving objects and + label the detections with the identified object.\VER{TEST}{TLR:2, TLR:3, TLR:22} + +\item The AP Database shall accept new detections at the rate + generated by the telescope from the Phase 2 and Phase 4 analysis. + \tbr{Except within 10 degrees of the galactic plane, the AP Database + shall keep up with the incoming rates.} The expected rates are + listed in Table~\ref{APrates}, along with the total data volume + required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2, TLR:3, TLR:22} + +\item The AP Database shall provide access to external Pan-STARRS + clients to the detected objects within \tbr{5 minute} after the + image is obtained.\VER{TEST}{TLR:22} +\label{IPP:DeReq:29c} +\end{enumerate} \begin{table} \begin{center} -\caption{AP Data Volume Requirements\label{APrates}} -\begin{tabular}{lrrrr} +\caption{AP Data Volume and Throughput Requirements\label{APrates}} +\begin{tabular}{lrrr} \hline \hline -Quantity & P2 & P4$\Sigma$ & P4$\Delta$ & SS \\ +Quantity & P2 & P4$\Sigma$ & P4$\Delta$ \\ \hline -detection limit & $20 \sigma$ & $5 \sigma$ & $3 \sigma$ & \\ -depth (r') & 20.8 & 23.0 & ? & \\ -stars deg$^{-2}$ ($|b|>10$) & $1 \times 10^5$ & $4 \times 10^5$ & $2 \times 10^5$ & \\ -stars FPA$^{-1}$ ($|b|>10$) & $7 \times 10^5$ & $2.8 \times 10^6$ & $1.4 \times 10^6$ & \\ -stars sec$^{-1}$ ($|b|>10$) & $2.3 \times 10^4$ & $2.3 \times 10^4$ & $1.2 \times 10^4$ & \\ -bytes star$^{-1}$ & 64 & 100 & 64 & \\ -MB sec$^{-1}$ & 1.4 & 2.2 & 0.7 & \\ -PS-1 total TB & 8 & 12 & 4 & \\ +detection limit & $20 \sigma$ & $5 \sigma$ & $3 \sigma$ \\ +depth (r') & 21.8 & 24.0 & 24.5 \\ +bytes star$^{-1}$ & 64 & 100 & 64 \\ +stars deg$^{-2}$ ($|b|>10$) & $2.0 \times 10^5$ & $8.0 \times 10^5$ & $2.0 \times 10^5$ \\ +stars FPA$^{-1}$ ($|b|>10$) & $1.4 \times 10^6$ & $5.6 \times 10^6$ & $1.4 \times 10^6$ \\ +stars sec$^{-1}$ ($|b|>10$) & $3.5 \times 10^4$ & $3.5 \times 10^4$ & $8.8 \times 10^3$ \\ +MB sec$^{-1}$ & 2.3 & 3.5 & 0.6 \\ +AP total TB & 7.7 & - & - \\ +IVP total TB & 13 & 20 & 3 \\ +MOPS total TB & 4 & 6 & 1 \\ +PS-1 total TB & 25 & 26 & 4 \\ \hline \end{tabular} @@ -679,5 +719,4 @@ \caption{Metadata Classes\label{metadata}} \begin{tabular}{l} -\hline \hline \hline @@ -700,58 +739,49 @@ \end{table} -The IPP requires a Metadata Database to store and provide access to -metadata of various types and from various sources. Metadata in the -context of the IPP corresponds to all data which is not included in -the two data stores discussed above (Images and Detection/Objects). -Metadata is generated at the telescope and during the various analysis -stages - -\begin{enumerate} -\item The Metadata Database must store and provide metadata for all - raw images, for processed images, for the calibration images (both - raw and master), for the extracted object lists. Metadata - describing the environmental conditions at the telescope must also - be stored and provided as needed. Table~\ref{metadata} lists the - classes of metadata which must be stored by the Metadata Database. - -\item If analysis results are exchanged between analysis stages via - the Metadata Database, it must provide access to the queried data on - timescales of $<2$ seconds to avoid slowing down the analysis - systems. - -\item The Metadata Database must store the metadata for the lifetime - of the project. - -\item The Metadata Database must be capable of accepting a total data - volume after 2 years of operation of 128 GB. - -\item The Metadata Database must respond to simple queries which +\begin{enumerate} +\item The IPP Metadata Database shall accept metadata from the summit + at a sustained rate of \tbr{1 MB per second}.\VER{TEST}{TLR:17, TLR:21, TLR:25} + +\item The Metadata Database shall store the classes of data listed in + Table~\ref{metadata}. Thus, the Metadata Database shall store and + provide metadata for all raw images, for processed images, for the + calibration images (both raw and master), for the extracted object + lists. Metadata describing the environmental conditions at the + telescope shall also be stored and provided as needed. + Database.\VER{INSPECT}{TLR:21, TLR:25} + +\item The Metadata Database queries shall have a latency of $< 0.1$ seconds.\VER{TEST}{TLR:17} + +\item The Metadata Database shall be capable of at least 100 queries per second.\VER{TEST}{TLR:17} + +\item The Metadata Database shall be capable of accepting a total data + volume after 2 years of operation of 280 GB. \VER{INSPECT}{TLR:25} + +\item The Metadata Database shall respond to simple queries which return the data in the categories listed in Table~\ref{metadata} based on the primary data key and with basic constraints of time - ranges and other simple conditional constraints. - -\item The Metadata must store descriptive information about the raw + ranges and other simple conditional constraints.\VER{TEST}{TLR:17} + +\item The Metadata shall store descriptive information about the raw images received from the summit and the current state of the data - processing. - -\item The Metadata must also store descriptive information for each of - the static sky images currently available. - -\item The IPP requires configuration information defining the - organization and configuration of the IPP itself. The Metadata - database must store the configuration information with restricted - access so that only specific people may change the information. - Examples of configuration data include the default parameters for - the various analysis programs, the description of the computing - environment, and the process status information, etc. - -\item The Metadata Database must restrict access to the scientific - parameters to a different group from the software and hardware - configuration parameters. - -\item In the discussion of the Analysis Stages below, various steps - specify that the values are user-configurable parameters. These - parameters must be stored in and extracted from the Metadata - Database. + processing.\TASK + +\item The Metadata shall also store descriptive information for each of + the static sky images currently available.\TASK + +\item Software configuration parameters shall be stored in and + extracted from the Metadata Database.\TASK + +\item The Metadata database shall store the configuration information + with restricted access so that only specific people may change the + information.\VER{TEST}{allocated} + +\item User-configurable software parameters shall be stored in and + extracted from the Metadata Database.\TASK + +\item The Metadata Database shall restrict write access of the + scientific parameters to a different group from the software and + hardware configuration parameters.\VER{TEST}{allocated} + \end{enumerate} @@ -759,104 +789,124 @@ \begin{enumerate} -\item The IPP Controller must manage tasks on a cluster of up to 128 - computers. - -\item On startup, the IPP Controller must attempt to establish +\item The IPP Controller shall manage tasks on a cluster of up to 128 + computers.\VER{TEST}{TLR:17} + +\item On startup, the IPP Controller shall attempt to establish communication with all of its computers and set their state to be - {\tt alive} or {\tt dead} based on the success of the connection. - -\item The IPP Controller must detect computers which crash or stop - responding and set their state to {\tt dead}. - -\item The IPP Controller must attempt to re-establish communication - with {\tt dead} computers. - -\item The IPP Controller must accept tasks from external users and + {\tt alive} or {\tt dead} based on the success of the connection.\VER{TEST}{TLR:17} + +\item The IPP Controller shall detect computers which crash or stop + responding and set their state to {\tt dead}.\VER{TEST}{TLR:17} + +\item The IPP Controller shall attempt to re-establish communication + with {\tt dead} computers.\VER{TEST}{TLR:17} + +\item The IPP Controller shall accept tasks from external users and systems, which may specify a desired CPU (node) and priority in - addition to the task command. - -\item The IPP Controller must attempt to run pending tasks on the - desired node, if available (not {\tt dead} or {\tt off}). - -\item If the node is unavailable, the IPP Controller must attempt to - run the task on another node. - -\item If the node is available, the IPP Controller must attempt to run + addition to the task command.\VER{TEST}{TLR:17} + +\item The IPP Controller shall attempt to run pending tasks on the + desired node, if available (not {\tt dead} or {\tt off}).\VER{TEST}{TLR:17} + +\item If the node is unavailable, the IPP Controller shall attempt to + run the task on another node.\VER{TEST}{TLR:17} + +\item If the node is available, the IPP Controller shall attempt to run a given task only if no higher-priority tasks are available and no - task is currently being executed. - -\item The IPP Controller must monitor the output from the task and - write it to an associated log destination. - -\item The IPP Controller must monitor the execution status of each + task is currently being executed.\VER{TEST}{TLR:17} + +\item The IPP Controller shall monitor the output from the task and + write it to an associated log destination.\VER{TEST}{TLR:17} + +\item The IPP Controller shall monitor the execution status of each task currently executing on a node and perform the following actions: \begin{enumerate} - \item identify the task as successful if it has a valid exit status. - \item identify the task as unsuccessful if it has an error exit - status. - \item identify the task as unattempted if the computer crashed. + \item identify the task as successful if it has a valid exit status.\VER{TEST}{TLR:17} + \item identify the task as unsuccessful if it has an error exit status.\VER{TEST}{TLR:17} + \item identify the task as unattempted if the computer crashed.\VER{TEST}{TLR:17} \end{enumerate} -\item The IPP Controller must accept and perform the following +\item The IPP Controller shall accept and perform the following external commands: \begin{enumerate} - \item add a task to the pending task list. - \item delete a specific task from the pending task list. - \item return the current status of a specific task. - \item return a list of all pending and non-pending tasks. - \item set a specified computer state to {\tt off} or {\tt dead}. - \item restrict a specified CPU to a class of tasks. - \item halt execution of a specified task. - \item set the IPP Controller state to {\tt finish}, {\tt abort}, or - {\tt stop}. + \item add a task to the pending task list.\VER{TEST}{TLR:17} + \item delete a specific task from the pending task list.\VER{TEST}{TLR:17} + \item return the current status of a specific task.\VER{TEST}{TLR:17} + \item return a list of all pending and non-pending tasks.\VER{TEST}{TLR:17} + \item set a specified computer state to {\tt off} or {\tt dead}.\VER{TEST}{TLR:17} + \item restrict a specified CPU to a class of tasks.\VER{TEST}{TLR:17} + \item halt execution of a specified task.\VER{TEST}{TLR:17} + \item set the IPP Controller state to {\tt finish}, {\tt abort}, or {\tt stop}.\VER{TEST}{TLR:17} \end{enumerate} + +\item The IPP Controller shall limit command latency to \tbr{$< 0.1$} seconds.\VER{TEST}{TLR:17} + +\item The IPP Controller shall be capable of performing up to \tbr{10 tasks per second}.\VER{TEST}{TLR:17} + +\item The IPP Controller shall be capable of buffering up to a total of \tbr{64 MB} of messages.\VER{TEST}{TLR:17} + +\item The IPP Controller shall be capable of executing up to \tbr{6 million tasks per month}.\VER{TEST}{TLR:17} + +\item The IPP Controller shall be capable of interacting with up to \tbr{256} client processes.\VER{TEST}{TLR:17} + +\item The IPP Controller shall be capable of accepting up to 2 non-client (external) requests per second.\VER{TEST}{TLR:17} \end{enumerate} \subsubsection{Scheduler} \begin{enumerate} -\item The IPP Scheduler must send the analysis tasks which it - initiates to the IPP Controller. - -\item All analysis tasks sent by the IPP Scheduler must include a +\item The IPP Scheduler shall send the analysis tasks which it + initiates to the IPP Controller.\VER{TEST}{TLR:17} + +\item All analysis tasks sent by the IPP Scheduler shall include a complete UNIX command with necessary arguments, the priority of the - task, and optionally the desired processing node. - -\item The IPP Scheduler must refer to several input data sources to - decide what tasks to initiate. These data sources include the IPP - Metadata Database, the Summit Metadata Database, and User requests. - -\item The IPP Scheduler must query the Databases on a regular basis to - check for new input information. These queries must take place at - least once every \tbr{5 seconds}. - -\item The IPP Scheduler must accept new User input in real-time - (within 0.1 seconds of the request). - -\item The IPP Scheduler must construct new tasks on the basis of the - inputs and a task dependency table. + task, and optionally the desired processing node.\VER{INSPECT}{TLR:17} + +\item The IPP Scheduler shall query the Databases on a regular basis + to check for new input information. These queries shall take place + at least once every \tbr{1 seconds}.\VER{TEST}{TLR:17} + +\item The IPP Scheduler shall accept new User input in real-time: +within 0.1 seconds of the request.\VER{TEST}{TLR:17} \item When the IPP Scheduler is placed in the {\em paused state}, it - must only initiate User-requested tasks. + shall only initiate User-requested tasks.\VER{TEST}{TLR:17} \item When the IPP Scheduler is placed in the {\em interactive state}, - it must initiate User-requested tasks as well as data transfer - tasks. + it shall initiate User-requested tasks as well as data transfer + tasks.\VER{TEST}{TLR:17} \item When the IPP Scheduler is placed in the {\em automatic state}, - it must initiate the most appropriate task based on the inputs. - -\item The IPP Scheduler must receive the exit status of tasks from the - IPP Controller. - -\item The IPP Scheduler must send the exit status of the analysis + it shall initiate the most appropriate task based on the inputs and + dependency rules.\VER{TEST}{TLR:17} + +\item The IPP Scheduler shall send the exit status of the analysis tasks to the appropriate destination as defined by the task - dependency table. + dependency table.\VER{TEST}{TLR:17} + +\item The IPP Scheduler shall publish the static sky images to the + Pan-STARRS PSPS on a time-scale of \tbr{6 month}.\VER{TEST}{TLR:19} + +\item The IPP Scheduler shall publish the detected objects to the + Pan-STARRS PSPS on a time-scale of \tbr{1 month}.\VER{TEST}{TLR:20} + +\item The IPP Scheduler shall publish the IPP and OTIS metadata to the + Pan-STARRS PSPS on a time-scale of \tbr{1 week}.\VER{TEST}{TLR:21} + +\item The IPP Scheduler shall send the detected single-occurance + transient objects to the MOPS subsystem within 5 minutes of the + image exposure time.\VER{TEST}{TLR:22} + +\item The IPP Scheduler shall send the metadata appropriate to the + images from which single-occurance transient objects were detected + to the MOPS subsystem within 5 minutes of the image exposure + time.\VER{TEST}{TLR:22} + \end{enumerate} \subsection{Analysis Stages} -We now consider the requirements of the analysis tasks which must be +We now consider the requirements of the analysis tasks which shall be performed by the IPP. These tasks represent the core of the required IPP functionality; the architectural components discussed above can be @@ -868,28 +918,80 @@ The Science Image analysis stages together represent the basic data analysis required by the IPP. There are several requirements which -must be met by the collection of science image analysis stages as a +shall be met by the collection of science image analysis stages as a group. \begin{enumerate} -\item The science image analysis stages must perform their analysis - quickly enough to keep up with the incoming data stream. The - required processing time is derived from the rate at which science - images are obtained by PS-1. - -\item At a minimum, the Science Image Analysis must keep up with the - average image rate over the course of 1 day. - -\item In order to provide a sufficient buffer for variations in the - processing speed, the Science Image Analysis must be able to process - all images from a night within 12 hours. +\item The IPP Science Analysis shall pre-process the science images + with the master calibration images at a sustained rate of 1 exposure + (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17} + +\item The IPP Science Analysis shall merge multiple pre-processed + science images into stacked images with corresponding signal-to-noise + maps at a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17} + +\item The IPP Science Analysis shall excise pixels from the input + images which are outliers for the ensemble of corresponding pixels + with an efficiency of $> 99$\%.\VER{ANALYSIS}{TLR:18} + +\item The IPP Science Analysis shall merge the cleaned images into the + static sky image, and update the corresponding exposure (S/N) maps, + at a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17} + +\item The IPP Science Analysis shall detect and measure parameters of +objects on the pre-processed science images.\VER{TEST}{TLR:12} + +\item The IPP Science Analysis shall detect and measure parameters of +objects on the stacked science images.\VER{TEST}{TLR:13} + +\item The IPP Science Analysis shall detect and measure parameters of +objects on the static sky images.\VER{TEST}{TLR:14} + +\item The IPP Science Analysis shall detect and measure parameters of +objects on the difference images.\VER{TEST}{TLR:15} + +\item The IPP Science Analysis shall determine astrometry of the + detected objects relative to an external astrometric reference with + an accuracy of \tbr{750 mas} (for bright objects) in the + Commissioning phase of the telescope.\VER{TEST}{TLR:4, TLR:3} + +\item The IPP Science Analysis shall determine astrometry of the + detected objects relative to an external astrometric reference with + an accuracy of \tbr{250 mas} (for bright objects) during the + construction of the Pan-STARRS Astrometric Reference Catalog.\VER{ANALYSIS}{TLR:4, TLR:3} + +\item The IPP Science Analysis shall determine astrometry of the + detected objects relative to the Pan-STARRS Astrometric Reference + with an accuracy of \tbr{100 mas} (for bright objects) during normal + operations.\VER{ANALYSIS}{TLR:4, TLR:3} + +\item The IPP Science Analysis shall determine photometry of the + detected objects within an internal photometric system with scatter + less than \tbr{25 millimags} (for bright objects) during the + Commissioning phase of the telescope in photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2} + +\item The IPP Science Analysis shall determine photometry of the + detected objects within an internal photometric system with scatter + less than \tbr{10 millimags} (for bright objects) during the + construction of the Pan-STARRS Photometric Reference Catalog in + photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2} + +\item The IPP Science Analysis shall determine photometry of the + detected objects within an internal photometric system with scatter + less than \tbr{5 millimags} (for bright objects) during normal + operations in photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2} + +\item The IPP Science Analysis shall determine photometry of the + detected objects in an external photometric system with scatter less + than \tbr{10 millimags} (for bright objects) during normal operations + in photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2} \item The maximum latency between the acquisition of an image and the completion of the science image analysis is set by the science requirements of the fast transient recovery programs. The science - image analysis must process images from these observing programs - within \tbr{5 min} of their arrival time in the IPP Image Server. - -\item The science image analysis stages must processes up to 1000 - science images per night. + image analysis shall process images to detection transients within + \tbr{5 min} of their acquisition.\VER{TEST}{TLR:22} + +\item The science image analysis stages shall processes up to 1000 + science images per night.\VER{TEST}{TLR:17} \end{enumerate} @@ -898,37 +1000,40 @@ \begin{enumerate} -\item The Phase 1 analysis stage must determine the astrometric +\item the Phase 1 analysis shall execute within 2 seconds for a + complete FPA image.\VER{TEST}{TLR:17} + +\item The Phase 1 analysis stage shall determine the astrometric solution of the complete camera (FPA image) with an accuracy of - \tbr{1 arcsec} peak-to-peak deviation. - -\item The Phase 1 analysis stage must load the guide star pixel and - celestial coordinates from the \tbd{IPP Metadata - Database}\comment{or from the image header?}. + \tbr{1 arcsec} peak-to-peak deviation.\VER{TEST}{TLR:3} + +\item The Phase 1 analysis stage shall load the guide star pixel and + celestial coordinates.\TASK \item If guide stars are not available, the Phase 1 analysis stage - must extract bright stars from the image. - -\item This extraction must be done in less than \tbr{1 second}. + shall extract bright stars from the image.\TASK + +\item This extraction shall be done in less than \tbr{1 second}.\VER{TEST}{TLR:17} \item The total number of stars and size of the bright-star - acquisition box must be a user-configurable parameter. + acquisition box shall be a user-configurable parameter in the range + 20 - 250.\TASK \item In order for blind astrometry of an image to succeed, it is necessary that approximate image coordinates be known. The Phase 1 - analysis must be able to succeed despite initial coordinate errors - as large as \tbr{20\arcsec}. - -\item The Phase 1 analysis stage must construct a table of the + analysis shall be able to succeed despite initial coordinate errors + as large as \tbr{20\arcsec}.\VER{TEST}{TLR:3} + +\item The Phase 1 analysis stage shall construct a table of the overlaps between the science image to be processed and the static - sky images. - -\item The overlaps must be overestimated by a small amount so that + sky images.\TASK + +\item The overlaps shall be overestimated by a small amount so that errors in astrometry at Phase 1 will not cause any valid static sky - / science image pairs to be missed. - -\item The amount of overlap must be a user-configurable parameter. + / science image pairs to be missed.\TASK + +\item The amount of overlap shall be a user-configurable parameter.\VER{TEST}{TLR:6, TLR:11} \item Sky cells which do not have sufficient science image overlap - \tbd{$< 5\%$} must be excluded from the overlap table. + \tbr{$< 5\%$} shall be excluded from the overlap table.\VER{TEST}{TLR:6, TLR:11} \item It is not unusual for an image to be obtained with invalid @@ -936,6 +1041,6 @@ control system may make an error and report the wrong time or coordinates. Or, the image may be obtained in exceptionally poor - conditions with no detected stars. Phase 1 must return a - descriptive error message in these conditions. + conditions with no detected stars. Phase 1 shall return a + descriptive error message in these conditions.\TASK \end{enumerate} @@ -945,15 +1050,20 @@ the detector are processed to remove instrumental signatures. +\paragraph{Timing} +The complete Phase~2 analysis shall be performed in $< 38$ seconds for +up to 4 complete FPA images at one time. \VER{TEST}{TLR:17} + \paragraph{Processing Recipe} \begin{enumerate} -\item The Phase 2 analysis stage must consult the processing recipe to - define the necessary analysis steps performed by the Phase 2 stage. - -\item Phase 2 must perform the analysis steps only if required by the - processing recipe. - -\item The processing recipe must define the stages to be executed with +\item The Phase 2 analysis stage shall consult the processing recipe + to define the necessary analysis steps performed by the Phase 2 + stage.\TASK + +\item Phase 2 shall perform the analysis steps only if required by the + processing recipe.\TASK + +\item The processing recipe shall define the stages to be executed with optional exposure time and background flux limits to require or - exclude select certain stages. + exclude select certain stages.\TASK \end{enumerate} @@ -961,11 +1071,11 @@ \begin{enumerate} -\item The Phase 2 analysis stage must determine the OT kernel from the - IPP Metadata Database\comment{or image header}. - -\item The Phase 2 analysis stage must convolve the flat-field and - high-spatial-frequency fringe images with the OT kernel. - -\item If no OT kernel exists, this step must be silently skipped. +\item The Phase 2 analysis stage shall convolve the flat-field and + high-spatial-frequency fringe images with the OT kernel.\VER{TEST}{TLR:1} + +\item The Phase 2 analysis stage shall determine the OT kernel from the + IPP Metadata Database.\TASK + +\item If no OT kernel exists, this step shall be silently skipped.\TASK \end{enumerate} @@ -973,15 +1083,17 @@ \begin{enumerate} -\item The Phase 2 analysis must load the basic bad pixel map appropriate to -the detector of interest. - -\item The Phase 2 analysis must use the OT kernel to grow the traps in the -raw bad pixel map. - -\item The Phase 2 analysis must mask saturated pixels and a user-specified -number of surrounding pixels. - -\item Different bits must be set to identify different reasons for masking -the pixels. +\item The Phase 2 analysis shall load the basic bad pixel map appropriate to +the detector of interest.\VER{TEST}{TLR:18} + +\item The Phase 2 analysis shall use the OT kernel to grow the traps in the +raw bad pixel map. \VER{TEST}{TLR:18} + +\item The Phase 2 analysis shall mask saturated pixels and a user-specified +number of surrounding pixels.\VER{TEST}{TLR:18} + +\item The Phase 2 analysis shall mask ghosts of bright stars.\VER{TEST}{TLR:18} + +\item Different bits shall be set to identify different reasons for masking +the pixels.\VER{TEST}{TLR:21} \end{enumerate} @@ -989,49 +1101,48 @@ \begin{enumerate} -\item Phase 2 must perform bias subtraction on the image. - -\item Phase 2 must choose the bias subtraction method and analysis statistic -based on the user-configured parameters. - -\item The bias correction must be measured from the image overscan region. - -\item The overscan region must be determined from the image -header\comment{or Metadata DB}. - -\item The bias subtraction must apply one of the following bias corrections, -depending on the user parameters: - -\begin{enumerate} -\item subtract a single constant from the image. - -\item subtract a 1-D bias which varies along the overscan. The function to be used must include +\item Phase 2 shall perform bias subtraction on the image.\VER{TEST}{TLR:1} + +\item Phase 2 shall choose the bias subtraction method and analysis statistic +based on the user-configured parameters.\TASK + +\item The bias correction shall be measured from the image overscan region.\TASK + +\item The overscan region shall be determined from the Metadata DB.\TASK + +\item The bias subtraction shall be capable of using one of following +bias corrections, depending on the user parameters: + +\begin{enumerate} +\item subtract a single constant from the image. \VER{TEST}{TLR:1} + +\item subtract a 1-D bias which varies along the overscan. The function to be used shall include a spline or a Chebychev polynomial derived from the data values along -the overscan, as specified by the user parameters. +the overscan, as specified by the user parameters. \VER{TEST}{TLR:1} \item correct the overscan {\em and} subtract a 2-D bias image which - has been overscan corrected using one of the two methods above. + has been overscan corrected using one of the two methods above.\VER{TEST}{TLR:1} \end{enumerate} \item The statistic used to calculate the overscan constant or the -inputs to the spline and polynomial fits must be derived from groups +inputs to the spline and polynomial fits shall be derived from groups of pixels on the basis of one of several possible statistics, as -specified by the user parameters. - -\item The choice of statistics must include the sample and robust -mean, median, and modes. +specified by the user parameters.\VER{TEST}{TLR:1} + +\item The choice of statistics shall include the sample and robust +mean, median, and modes.\VER{TEST}{TLR:1} \item In the case of a single constant, all of the overscan pixel -values are used in the calculation of this statistic. +values are used in the calculation of this statistic.\VER{TEST}{TLR:1} \item In the case of the 1D functional representation, the input -values to the fit must represent the coordinate along the overscan, +values to the fit shall represent the coordinate along the overscan, with the statistic derived from the pixels in the perpendicular -direction at each location. - -\item If specified in the user parameters, sigma-clipping must be -performed on the input data values. - -The bias subtraction must leave no residuals greater than \tbr{1 DN} -peak-to-peak. +direction at each location.\VER{TEST}{TLR:1} + +\item If specified in the user parameters, sigma-clipping shall be +performed on the input data values.\VER{TEST}{TLR:1} + +\item The bias subtraction shall leave no residuals greater than \tbr{1 DN} +peak-to-peak.\VER{TEST}{TLR:1} \end{enumerate} @@ -1039,56 +1150,53 @@ \begin{enumerate} -\item The Phase 2 analysis must trim the non-imaging pixels from the -image. - -\item The definition of the imaging area must be determined from the -Metadata Database\comment{or image header?}. - -\item Phase 2 must trim pixel near the edges that have been -compromised due to OT operation. +\item The Phase 2 analysis shall trim the non-imaging pixels from the +image.\TASK + +\item The definition of the imaging area shall be determined from the +Metadata Database.\TASK + +\item Phase 2 shall trim pixel near the edges that have been +compromised due to OT operation.\VER{TEST}{TLR:1} \end{enumerate} \paragraph{Correct for non-linearity} -If required by the recipe, each chip must be independently corrected for the -effects of non-linearity. +If required by the recipe, each chip shall be independently corrected for the +effects of non-linearity.\VER{TEST}{TLR:1} \paragraph{Flat-field correction} \begin{enumerate} -\item The Phase 2 analysis must divide the science image by the - provided flat-field image. - -\item The division must handle zero-valued pixels in the flat-field +\item The Phase 2 analysis shall divide the science image by the + provided flat-field image.\VER{TEST}{TLR:1} + +\item The division shall handle zero-valued pixels in the flat-field image without raising floating point exceptions, setting the - corresponding bit value in the mask. - -\item The flat-field images must be appropriately normalized (see - section \ref{mkcal}). - -\item The flat-fielded image must have a consistent photometric + corresponding bit value in the mask.\VER{TEST}{TLR:1} + +\item The flat-field images shall be appropriately normalized (see + section \ref{mkcal}).\VER{TEST}{TLR:1} + +\item The flat-fielded image shall have a consistent photometric zero-point across the chip, and across the full FPA, to within 0.2\% - with peak-to-peak deviations of \tbr{0.5\%}. -\end{enumerate} - -\tbd{color of stars in flat-field correction?} + with peak-to-peak deviations of \tbr{0.5\%}.\VER{TEST}{TLR:1} +\end{enumerate} \paragraph{Sky \& Fringe subtraction} \begin{enumerate} -\item The Phase 2 analysis must subtract the sky (and fringe where - needed) contributions from the images. - -\item The residual after the background subtraction must have an +\item The Phase 2 analysis shall subtract the sky (and fringe where + needed) contributions from the images.\VER{TEST}{TLR:1, TLR:5} + +\item The residual after the background subtraction shall have an average offset of 0 counts, excluding the signal from astronomical - features. - -\item The background residuals must have peak-to-peak variations of - less than \tbr{1\%} of the input background amplitude. - -\item The background residuals must have a scatter of less than + features.\VER{TEST}{TLR:5} + +\item The background residuals shall have peak-to-peak variations of + less than \tbr{1\%} of the input background amplitude.\VER{TEST}{TLR:5} + +\item The background residuals shall have a scatter of less than \tbr{1\%} of the input background amplitude for apertures of less - than \tbr{10~arcsec}.\comment{derived from the need for systematic - errors of better than 0.5\% and known background ranges.} + than \tbr{10~arcsec}.\VER{TEST}{TLR:1} \end{enumerate} @@ -1096,15 +1204,16 @@ \begin{enumerate} -\item The Phase 2 analysis must detect cosmic rays in single images - which are brighter than a user-configurable threshold. - -\item The Phase 2 analysis must mask detected cosmic rays with a - unique bit value in the mask. - -\item The Phase 2 analysis must extend the masked region by a - user-configurable growth factor. - -\item The Phase 2 analysis must perform the cosmic ray detection only - if it is required by the analysis recipe. +\item The Phase 2 analysis shall detect cosmic rays with flux $> + 5\sigma$ by morphology in single images with an efficiency of $> 95$\%. + \VER{TEST}{TLR:18} + +\item The Phase 2 analysis shall mask detected cosmic rays with a + unique bit value in the mask.\TASK + +\item The Phase 2 analysis shall extend the masked region by a + user-configurable growth factor.\TASK + +\item The Phase 2 analysis shall perform the cosmic ray detection only + if it is required by the analysis recipe.\TASK \end{enumerate} @@ -1112,34 +1221,34 @@ \begin{enumerate} -\item The Phase 2 analysis must perform object detection on the - processed images. - -\item The object detection process must detect all objects above a - user-configured threshold. - -\item The threshold must be a positive value; negative values must - invoke an error. - -\item The detection threshold must optionally be a function of the - average background flux or the local noise level. - -\item The object detection must measure the following object +\item The Phase 2 analysis shall perform object detection on the + processed images.\VER{TEST}{TLR:12} + +\item The object detection process shall detect all objects above a + user-configured threshold.\TASK + +\item The threshold shall be a positive value; negative values shall + invoke an error.\TASK + +\item The detection threshold shall optionally be a function of the + average background flux or the local noise level.\TASK + +\item The object detection shall measure the following object parameters: \begin{enumerate} - \item object centroid and position errors - \item an extended object position ($x_g, y_g$) - \item instrumental PSF magnitude and error - \item local background level and error + \item object centroid and position errors\VER{TEST}{TLR:12} + \item an extended object position ($x_g, y_g$)\VER{TEST}{TLR:12} + \item instrumental PSF magnitude and error\VER{TEST}{TLR:12} + \item local background level and error\VER{TEST}{TLR:12} \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) of the object - and their covariance matrix + and their covariance matrix\VER{TEST}{TLR:12} \end{enumerate} -\item Minimal object classification must be performed to distinguish +\item Minimal object classification shall be performed to distinguish objects which are consistent with a single PSF, objects which are inconsistently large, objects which are inconsistently small, and - objects which are saturated. - -\item The resulting collection of detected objects must be saved along - with the relevant image metadata (\ie filter, exposure time, etc). + objects which are saturated.\VER{TEST}{TLR:12} + +\item The resulting collection of detected objects shall be saved along + with the relevant image metadata (\ie filter, exposure time, etc).\VER{TEST}{TLR:20} \end{enumerate} @@ -1147,27 +1256,27 @@ \begin{enumerate} -\item The Phase 2 analysis must match the detected objects with known - astrometric reference objects. - -\item The astrometric reference object coordinates must be adjusted - for proper motion. - -\item The reference and detected object coordinates must be fit to - determine astrometric parameters for the individual OTAs. - -\item The OTA astrometric parameters must include Chebychev - polynomials of the coordinates up to 3rd order. - -\item The fitted number of polynomial orders must be a user-configured - parameter. - -\item The Cell astrometric parameters must not be allowed to vary in - the fit. - -\item The fit must be robust, rejecting outlier matches (either stars - with poorly determined proper motion or spurious matches). - -\item The resulting astrometric solution must be consistent across the - OTA field to within \tbr{300 milli-arcsec}. +\item The Phase 2 analysis shall match the detected objects with known + astrometric reference objects.\VER{TEST}{TLR:3} + +\item The astrometric reference object coordinates shall be adjusted + for proper motion.\VER{TEST}{TLR:3} + +\item The reference and detected object coordinates shall be fit to + determine astrometric parameters for the individual OTAs.\VER{TEST}{TLR:3} + +\item The OTA astrometric parameters shall include polynomials of the +coordinates up to 3rd order.\VER{TEST}{TLR:3} + +\item The fitted number of polynomial orders shall be a user-configured + parameter.\TASK + +\item The Cell astrometric parameters shall not be allowed to vary in + the fit.\VER{}{} + +\item The fit shall be robust, rejecting outlier matches (either stars + with poorly determined proper motion or spurious matches).\VER{TEST}{TLR:3} + +\item The resulting astrometric solution shall be consistent across the + OTA field to within \tbr{100 milli-arcsec}.\VER{TEST}{TLR:4} \end{enumerate} @@ -1175,9 +1284,9 @@ \begin{enumerate} -\item The Phase 2 analysis must extract subrasters (`postage stamps') +\item The Phase 2 analysis shall extract subrasters (`postage stamps') surrounding a user-specified list of coordinates from the flattened - images. - -\item The postage stamp images must be saved in the IPP Image Server. + images.\VER{TEST}{TLR:12} + +\item The postage stamp images shall be saved in the IPP Image Server.\VER{TEST}{TLR:12} \end{enumerate} @@ -1185,24 +1294,36 @@ \begin{enumerate} -\item The Phase 3 analysis must use the objects detected in Phase 2, +\item The Phase 3 analysis shall use the objects detected in Phase 2, matched with a user-specified reference photometry catalog, to determine the image photometric zero point and zero-point variations - across the field. - -\item If zero-point variations are significant \tbd{level TBD}, the - zero-point variations must be modeled with a Chebychev polynomial - correction of order 3 or less. - -\item The photometric nature of the FPA image must be categorized - \tbd{numerical scale?} on the basis of the zero-point consistency, - the transparency compared with recent long-term measurements in the - filter, and the external indicators of photometricity. - -\item The Phase 3 analysis must use the objects detected in Phase 2, + across the field.\VER{TEST}{??} + +\item If zero-point variations are significant (\tbr{$> 0.01$ mag + peak-to-peak}), the zero-point variations shall be modeled with a + polynomial correction of order 3 or less.\VER{TEST}{TLR:1} + +\item The photometric nature of the FPA image shall be categorized on + the basis of the zero-point consistency, the transparency compared + with recent long-term measurements in the filter, and the external + indicators of photometricity.\VER{TEST}{TLR:2} + +\item The Phase 3 analysis shall use the objects detected in Phase 2, matched with an appropriate astrometric reference catalog, to - improve the distortion model used for the image. - -\item The resulting astrometric accuracy must be limited by the - astrometric reference catalog, ie, 250 mas for USNO-B1.0. + improve the distortion model used for the image.\VER{TEST}{TLR:3} + +\item The resulting astrometric accuracy shall be consistent across +the field to 30 mas.\VER{TEST}{TLR:4} + +\item The resulting astrometric accuracy shall be limited by the + astrometric reference catalog, (eg, 100 - 250 mas for + USNO-B1.0).\VER{TEST}{TLR:3} + +\item The Phase 3 analysis shall modify the background correction of +Phase 2 based on the full-field statistics to achieve an accuracy of 1\% +of the background.\VER{TEST}{TLR:5} + +\item The complete Phase~3 analysis shall be performed in $< 2$ +seconds for up to 4 complete FPA images at one time. \VER{TEST}{TLR:17} + \end{enumerate} @@ -1217,17 +1338,17 @@ \begin{enumerate} -\item The Phase 4 analysis must determine the corresponding set of - image pixels for a given sky cell. - -\item The corresponding image pixels must be extracted from the input +\item The Phase 4 analysis shall determine the corresponding set of + image pixels for a given sky cell.\TASK + +\item The corresponding image pixels shall be extracted from the input images, using the astrometric information for each OTA and Cell to - determine the exact overlaps. - -\item The Phase 4 analysis must not miss any pixels in this match, and - it must read no more than 20\% more pixels than necessary from the - input images. - -\item The Phase 4 analysis must skip any sky cells with fewer than 5\% - of their pixels overlapping the input images. + determine the exact overlaps.\TASK + +\item The Phase 4 analysis shall not miss any pixels in this match, and + it shall read no more than 20\% more pixels than necessary from the + input images.\VER{TEST}{TLR:17} + +\item The Phase 4 analysis shall skip any sky cells with fewer than 5\% + of their pixels overlapping the input images.\VER{TEST}{TLR:17} \end{enumerate} @@ -1235,25 +1356,21 @@ \begin{enumerate} -\item Pixels which have been extracted from the input images must be - mapped to the corresponding pixels in the sky image. - -\item The transformation must be based on the measured astrometric +\item Pixels which have been extracted from the input images shall be + mapped to the corresponding pixels in the sky image.\TASK + +\item The transformation shall be based on the measured astrometric solution for the input images relative to the reference catalog used - to generate the static sky image. - -\item This warping must use a locally-linear astrometric solution. + to generate the static sky image.\VER{TEST}{TLR:3} + +\item This warping shall use a locally-linear astrometric solution.\VER{TEST}{TLR:17} -\item The output image must maintain photometric consistency with the - input image to within 0.2\%. -\end{enumerate} - -\tbd{interpolation? does interpolation method choice risk losing flux?} +\item The output image shall maintain photometric consistency with the + input image to within 0.2\%.\VER{TEST}{TLR:1} +\end{enumerate} \paragraph{Flux matching} -The Phase 4 analysis must determine appropriate photometry scaling -factors needed to combine the images photometrically. - -\tbd{is flux matched automatically by calibration?} +The Phase 4 analysis shall determine appropriate photometry scaling +factors needed to combine the images photometrically.\TASK \paragraph{Image outlier pixel rejection} @@ -1261,67 +1378,61 @@ \item When multiple images are combined, the group of input pixels - which contribute to an output pixel must be examined and pixels from - the group of images which are inconsistent with the ensemble - \tbd{how much?} must be identified and flagged. - -\item This outlier rejection must be performed optionally. - -\tbd{for moving objects and images which are not simultaneous, do we - identify the moving objects?} - -\tbd{use the spatial information? fit a 2-D Nth order polynomial to - the collection of pixels and then look for outliers} + which contribute to an output pixel shall be examined and pixels from + the group of images which are inconsistent with the ensemble (by an + amount defined by the user-configurable parameters) shall be + identified and flagged.\VER{TEST}{TLR:18} + +\item This outlier rejection shall be performed optionally.\TASK + \end{enumerate} \paragraph{Initial cleaned image} -The resulting collection of pixels must be used to construct a single -output image, cleaned of the outliers. +The resulting collection of pixels shall be used to construct a single +output image, cleaned of the outliers.\VER{TEST}{TLR:18} \paragraph{PSF matching} -The cleaned, combined image must be PSF matched with the static sky image. +The cleaned, combined image shall be PSF matched with the static sky image.\VER{TEST}{TLR:15} \paragraph{Image Subtraction} -The static sky image must be subtracted from the stacked, cleaned -image. - -\tbd{what about different stellar colors?} +The static sky image shall be subtracted from the stacked, cleaned +image. \VER{TEST}{TLR:15} \paragraph{Find objects in the image} \begin{enumerate} -\item The Phase 4 analysis must perform object detection on the - difference images. - -\item All objects in the difference image must be detected and the - pixels belonging to variable sources flagged in the input image. - -\item The object detection must detect all objects above a - user-configured threshold. - -\item Both positive and negative objects must be detected: the - specified threshold must define the absolute value of the detection - thresholds. - -\item The detection threshold must optionally be a function of the - average background flux or the local noise level. - -\item The object detection must measure the following object parameters: +\item The Phase 4 analysis shall perform object detection on the + difference images.\VER{TEST}{TLR:15} + +\item All objects in the difference image shall be detected and the + pixels belonging to variable sources flagged in the input image.\VER{TEST}{TLR:15} + +\item The object detection shall detect all objects above a + user-configured threshold.\VER{TEST}{TLR:15} + +\item Both positive and negative objects shall be detected: the + specified threshold shall define the absolute value of the detection + thresholds.\VER{TEST}{TLR:15} + +\item The detection threshold shall optionally be a function of the + average background flux or the local noise level.\VER{TEST}{TLR:15} + +\item The object detection shall measure the following object parameters: \begin{enumerate} - \item object centroid and position errors - \item instrumental PSF magnitude and error - \item local background level and error - \item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$ - \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix + \item object centroid and position errors\VER{TEST}{TLR:15} + \item instrumental PSF magnitude and error\VER{TEST}{TLR:15} + \item local background level and error\VER{TEST}{TLR:15} + \item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$\VER{TEST}{TLR:15} + \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix\VER{TEST}{TLR:15} \end{enumerate} -\item Minimal object classification must be performed to distinguish +\item Minimal object classification shall be performed to distinguish objects which are consistent with a single PSF, objects which are - inconsistent, and objects which are saturated. - -\item The resulting collection of detected objects must be saved along - with the relevant image metadata (\ie filter, exposure time, etc). + inconsistent, and objects which are saturated.\VER{TEST}{TLR:15, TLR:18} + +\item The resulting collection of detected objects shall be saved along + with the relevant image metadata (\ie filter, exposure time, etc).\VER{TEST}{TLR:22} \end{enumerate} @@ -1330,110 +1441,107 @@ \item The pixels flagged as being from the difference image sources - must be masked in the input images. - -\item A new, cleaned image must be constructed from the masked input - images. - -\end{enumerate} - -\tbd{how to handle variable stars?} + shall be masked in the input images.\VER{TEST}{TLR:6, TLR:11} + +\item A new, cleaned image shall be constructed from the masked input + images.\VER{TEST}{TLR:6, TLR:11} + +\end{enumerate} \paragraph{Find objects in the image} \begin{enumerate} -\item The Phase 4 analysis must perform object detection on the - cleaned, summed image. - -\item The object detection must detect all objects above a - user-configured threshold. - -\item The threshold must be a positive value; negative values must - invoke an error. - -\item The detection threshold optionally must be a function of the - average background flux or the local noise level. - -\item The object detection must measure the following object parameters: +\item The Phase 4 analysis shall perform object detection on the + cleaned, summed image.\VER{TEST}{TLR:13} + +\item The object detection shall detect all objects above a + user-configured threshold.\VER{TEST}{TLR:13} + +\item The threshold shall be a positive value; negative values shall + invoke an error.\VER{TEST}{TLR:13} + +\item The detection threshold optionally shall be a function of the + average background flux or the local noise level.\VER{TEST}{TLR:13} + +\item The object detection shall measure the following object parameters: \begin{enumerate} - \item object centroid and position errors - \item an extended object position ($x_g, y_g$) - \item instrumental PSF magnitude and error - \item local background level and error + \item object centroid and position errors\VER{TEST}{TLR:13} + \item an extended object position ($x_g, y_g$)\VER{TEST}{TLR:13} + \item instrumental PSF magnitude and error\VER{TEST}{TLR:13} + \item local background level and error\VER{TEST}{TLR:13} \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their - covariance matrix - \item the Petrosian radius, magnitude, axis ratio, and angle - \item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$. + covariance matrix\VER{TEST}{TLR:13} + \item the Petrosian radius, magnitude, axis ratio, and angle\VER{TEST}{TLR:13} + \item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$.\VER{TEST}{TLR:13} \end{enumerate} -\item Minimal object classification must be performed to distinguish +\item Minimal object classification shall be performed to distinguish objects which are consistent with a single PSF, objects which are - inconsistent, and objects which are saturated. - -\item The resulting collection of detected objects must be saved along - with the relevant image metadata (\ie filter, exposure time, etc). + inconsistent, and objects which are saturated.\VER{TEST}{TLR:13} + +\item The resulting collection of detected objects shall be saved along + with the relevant image metadata (\ie filter, exposure time, etc).\VER{TEST}{TLR:20} \end{enumerate} \paragraph{Image Processing Q/A} -Before the image is added to the static sky, it must pass Q/A tests. - -\tbd{how do we specify auotmatic Q/A tests? astrometry, photometry} +Before the image is added to the static sky, it shall pass Q/A tests: + +\begin{enumerate} +\item the measured photometry scatter for the image shall be less than + \tbr{1\%}.\VER{TEST}{TLR:1} + +\item the measured astrometry scatter for the image shall be less than + \tbr{30 mas}.\VER{TEST}{TLR:3} + +\end{enumerate} \paragraph{Update static sky} -The final, cleaned input image must be added to the static sky so that -an incrementally-deeper static sky image may be made. - -\tbd{parameters, weight map} - -\paragraph{Timing} - -It is required that the {\em total} processing for each exposure by -the Pan-STARRS system not take longer than the time between a complete -set of exposures. For PS-1, the primary mode of operation will use -four exposures to form a complete set (major frame), with 30 second -exposures times and 2 second readout times. Thus, the complete Phase -4 analysis must be performed on average within 120 seconds, assuming a -separate collection of computers are dedicated to the Phase 2 -analysis. - -\paragraph{Robustness} - -It is essential that the static sky image (which may have been -painstakingly accumulated over many months) not be corrupted by adding -in transient sources, or data that is of suspect quality (due, e.g., -to an error upstream in the processing). - -\tbd{what are the corresponding requirements?} +The final, cleaned input image shall be added to the static sky so that +an incrementally-deeper static sky image may be made.\VER{TEST}{TLR:6, TLR:11} + +\paragraph{Timing} +The complete Phase~4 analysis shall be performed in $< 38$ seconds for +up to 4 complete FPA images at one time. \VER{TEST}{TLR:17} \subsubsection{Calibration Stages} \label{mkcal} -\tbd{Requirements on the speed of processing the calibration images.} - -The Calibration analysis stages must construct the various types of -calibration frames needed by the IPP. The requirements for each of -these stages are discussed in detail below. +The Calibration analysis stages construct the various types of +calibration frames needed by the IPP. Requirements for the +calibration processing include the following: + +\begin{enumerate} +\item The IPP Calibration Analysis shall produce master calibration images +from the raw calibration images in less \tbr{2 hours}.\VER{TEST}{TLR:17, TLR:22} + +\item Master calibration images shall not introduce systematic + uncertainties in the photometry greater than \tbr{0.2\%}.\VER{TEST}{TLR:1} + +\end{enumerate} + +Requirements for each of the individual calibration analysis stages +are discussed in detail below. \paragraph{bias images} \begin{enumerate} -\item The \code{bias} calibration stage must construct a master bias - image from a collection of raw bias images. - -\item The \code{bias} calibration stage must correct the input images - based on the overscan region. - -\item The \code{bias} calibration stage must combine the input images +\item The \code{bias} calibration stage shall construct a master bias + image from a collection of raw bias images.\TASK + +\item The \code{bias} calibration stage shall correct the input images + based on the overscan region.\TASK + +\item The \code{bias} calibration stage shall combine the input images using the statistic specified by the user, selected from one of the following: sample mean, median, and mode, robust mean, median, and - mode, and the clipped mean and median. - -\item The \code{bias} calibration stage must construct residual - images, in which the master bias is applied to the input images. + mode, and the clipped mean and median.\TASK + +\item The \code{bias} calibration stage shall construct residual + images, in which the master bias is applied to the input images.\TASK \item Outlier residual images, those for which the residual bias and - variance in the bias image are excessive ($> 1DN$), must be excluded - from the input image stack the the bias image reconstructed. + variance in the bias image are excessive ($> 1DN$), shall be excluded + from the input image stack the the bias image reconstructed.\VER{TEST}{TLR:1} \end{enumerate} @@ -1441,25 +1549,25 @@ \begin{enumerate} -\item The \code{dark} calibration stage must construct a master dark - image from a collection of raw dark images. - -\item The \code{dark} calibration stage must raise an error if the +\item The \code{dark} calibration stage shall construct a master dark + image from a collection of raw dark images.\TASK + +\item The \code{dark} calibration stage shall raise an error if the input images have exposure time which deviate by more than - \tbr{2\%}. - -\item The \code{dark} calibration stage must correct the input dark - images for the bias. - -\item The \code{dark} calibration stage must combine the input images + \tbr{2\%}.\VER{TEST}{TLR:1} + +\item The \code{dark} calibration stage shall correct the input dark + images for the bias.\TASK + +\item The \code{dark} calibration stage shall combine the input images using the statistic specified by the user, selected from one of the following: sample mean, median, and mode, robust mean, median, and - mode, and the clipped mean and median. - -\item The \code{dark} calibration stage must construct residual - images, in which the master dark is applied to the input images. + mode, and the clipped mean and median.\VER{TEST}{TLR:1} + +\item The \code{dark} calibration stage shall construct residual + images, in which the master dark is applied to the input images.\TASK \item Outlier residual images, those for which the residual level and - variance are excessive ($> 1DN$), must be excluded from the input - image stack the the dark image reconstructed. + variance are excessive ($> 1DN$), shall be excluded from the input + image stack the the dark image reconstructed.\VER{TEST}{TLR:1} \end{enumerate} @@ -1467,31 +1575,31 @@ \begin{enumerate} -\item The \code{flat-field} calibration stage must construct a master - flat-field image from a collection of raw flat-field images. - -\item The \code{flat-field} calibration stage must accept a group of +\item The \code{flat-field} calibration stage shall construct a master + flat-field image from a collection of raw flat-field images.\VER{TEST}{TLR:1} + +\item The \code{flat-field} calibration stage shall accept a group of images from one of the following flat-field sources: dome, twilight, - night-sky. - -\item The \code{flat-field} calibration stage must raise an error if + night-sky.\VER{TEST}{TLR:1} + +\item The \code{flat-field} calibration stage shall raise an error if the input images in a single stack used more than one of the above - flat-field sources or multiple filters. - -\item The \code{flat-field} calibration stage must correct the input - flat-field images for the bias and dark. - -\item The \code{flat-field} calibration stage must combine the input + flat-field sources or multiple filters.\TASK + +\item The \code{flat-field} calibration stage shall correct the input + flat-field images for the bias and dark.\TASK + +\item The \code{flat-field} calibration stage shall combine the input images using the statistic specified by the user, selected from one of the following: sample mean, median, and mode, robust mean, - median, and mode, and the clipped mean and median. - -\item The \code{flat-field} calibration stage must construct residual + median, and mode, and the clipped mean and median.\VER{TEST}{TLR:1} + +\item The \code{flat-field} calibration stage shall construct residual images, in which the master flat-field is applied to the input - images. + images.\TASK \item Outlier residual images, those for which the residual level and variance are excessive ($> 0.1$\%, or 1.02 times the Poisson limit - of the flat-field image), must be excluded from the input image - stack the the flat-field image reconstructed. + of the flat-field image), shall be excluded from the input image + stack the the flat-field image reconstructed.\VER{TEST}{TLR:1} \end{enumerate} @@ -1499,25 +1607,25 @@ \begin{enumerate} -\item The \code{mask} calibration stage must construct a bad-pixel +\item The \code{mask} calibration stage shall construct a bad-pixel mask from a stack of raw flat-field images and a master flat-field - image. - -\item The \code{mask} calibration stage must mask the pixels which are + image.\VER{TEST}{TLR:1} + +\item The \code{mask} calibration stage shall mask the pixels which are inconsistent in the input flats by more than \tbr{1\%}, given - sufficient signal-to-noise in the input flats. - -\item The \code{mask} calibration stage must mask the pixels which are + sufficient signal-to-noise in the input flats.\VER{TEST}{TLR:1} + +\item The \code{mask} calibration stage shall mask the pixels which are consistently low or high in the input flats by more than a factor of - \tbr{3} beyond the typical pixel. - -\item The \code{mask} calibration stage must mask the pixels + \tbr{3} beyond the typical pixel.\VER{TEST}{TLR:1} + +\item The \code{mask} calibration stage shall mask the pixels identified in a table of bad pixels generated externally to the - calibration stage. - -\item The \code{mask} calibration stage must use multiple bit values - to identify the different types of masked pixels. - -\item The \code{mask} calibration stage must raise an error if the - input images generate too many bad pixels in the mask. + calibration stage.\TASK + +\item The \code{mask} calibration stage shall use multiple bit values + to identify the different types of masked pixels.\TASK + +\item The \code{mask} calibration stage shall raise an error if the + input images generate too many bad pixels in the mask.\TASK \end{enumerate} @@ -1525,63 +1633,61 @@ \begin{enumerate} -\item The \code{fringe} calibration stage must construct a master fringe +\item The \code{fringe} calibration stage shall construct a master fringe frame from a stack of raw night-time sky images or from a stack of -dome fringe frames. - -\item The \code{fringe} calibration stage must raise an error if the input +dome fringe frames.\VER{TEST}{TLR:1, TLR:5} + +\item The \code{fringe} calibration stage shall raise an error if the input stack consists is images generated with more than one type of fringe -source or with multiple filters. - -\item The \code{fringe} calibration stage must flatten the input images -to remove the pixel-to-pixel sensitivity variations of the detector. - -\item The \code{fringe} calibration stage must measure the fringe amplitude -on the input fringe images. - -\item The \code{fringe} calibration stage must scale the input fringe images -based on the fringe amplitude. - -\item The \code{fringe} calibration stage must combine the scaled input +source or with multiple filters.\TASK + +\item The \code{fringe} calibration stage shall flatten the input images +to remove the pixel-to-pixel sensitivity variations of the detector.\VER{TEST}{TLR:1} + +\item The \code{fringe} calibration stage shall measure the fringe amplitude +on the input fringe images.\TASK + +\item The \code{fringe} calibration stage shall scale the input fringe images +based on the fringe amplitude.\TASK + +\item The \code{fringe} calibration stage shall combine the scaled input images using the statistic specified by the user, selected from one of the following: sample mean, median, and mode, robust mean, median, and -mode, and the clipped mean and median. - -\item The \code{fringe} calibration stage must construct residual images, in +mode, and the clipped mean and median.\VER{TEST}{TLR:5} + +\item The \code{fringe} calibration stage shall construct residual images, in which the master fringe image is applied to the input images, along -with all necessary preceding calibration images. - -\item The \code{fringe} calibration stage must measure the residual fringe -amplitude on the residual images. +with all necessary preceding calibration images.\TASK + +\item The \code{fringe} calibration stage shall measure the residual fringe +amplitude on the residual images.\TASK \end{enumerate} \paragraph{low-spatial-frequency sky models} -The \code{sky model} calibration stage must construct a sky model -image from a stack of raw night-time sky images. - -\tbd{details of the image construction to be specified} +The \code{sky model} calibration stage shall construct a sky model +image from a stack of raw night-time sky images.\VER{TEST}{TLR:5} \paragraph{Flat-field correction frame} \begin{enumerate} -\item The \code{flat-field correction} calibration stage must construct a +\item The \code{flat-field correction} calibration stage shall construct a flat-field correction image from dithered observations of a stellar -field. - -\item The \code{flat-field correction} calibration stage must construct a -flat-field correction image for each filter and camera independently. - -\item The \code{flat-field correction} calibration stage must construct a +field.\VER{TEST}{TLR:1} + +\item The \code{flat-field correction} calibration stage shall construct a +flat-field correction image for each filter and camera independently.\TASK + +\item The \code{flat-field correction} calibration stage shall construct a correction image which makes the photometry of multiple observations of the same stellar source consistent at different locations in the -focal plane. - -\item The \code{flat-field correction} calibration stage must construct -corrected flat-field images using the measured correction. - -\item The \code{flat-field correction} calibration stage must determine the +focal plane.\VER{TEST}{TLR:1} + +\item The \code{flat-field correction} calibration stage shall construct +corrected flat-field images using the measured correction.\VER{TEST}{TLR:1} + +\item The \code{flat-field correction} calibration stage shall determine the consistency of the corrected flat-field images using the dithered stellar field observations flattened with the corrected flat-field -image. +image.\TASK \end{enumerate} @@ -1589,15 +1695,38 @@ \begin{enumerate} -\item The \code{non-linear correction} calibration stage must construct a +\item The \code{non-linear correction} calibration stage shall construct a non-linear correction from a collection of images of a constant source -with varying exposure times. - -\item The \code{non-linear correction} calibration stage must construct a -non-linear correction which linearizes the detector fluxes $<0.5\%$. - -\item The \code{non-linear correction} calibration stage must determine the +with varying exposure times.\VER{TEST}{TLR:1} + +\item The \code{non-linear correction} calibration stage shall construct a +non-linear correction which linearizes the detector fluxes $<0.5\%$.\VER{TEST}{TLR:1} + +\item The \code{non-linear correction} calibration stage shall determine the saturation regime, in which the non-linear correction is no longer -consistent to $<0.5\%$. -\end{enumerate} +consistent to $<0.5\%$.\VER{TEST}{TLR:1} +\end{enumerate} + +\paragraph{Telescope Astrometry Parameters} + +\begin{enumerate} +\item The IPP Calibration system shall construct static models of the + telescope astrometry parameters (e.g., distortion, detector warps) + once per week.\VER{INSPECT}{TLR:4} + +\item The IPP Calibration system shall construct static models of the + telescope astrometry parameters (e.g., distortion, detector warps) + with an accuracy to produce astrometry consistent to 30 + milliarcsec.\VER{TEST}{TLR:4} + +\item The IPP Calibration system shall monitor changes in the + telescope astrometry parameters and issue a warning if the + parameters change by more than \tbr{2\%}.\VER{INSPECT}{TLR:4} +\end{enumerate} + +\paragraph{Zero-Point Monitoring} + +The IPP Calibration system shall determine telescope filter and camera +zero-points on a \tbd{timescale} with an accuracy sufficient to +determine photometry in the native filter systems to 5 millimags. \subsubsection{Reference Catalog Creation} @@ -1607,5 +1736,5 @@ future Pan-STARRS calibration. The generation of these catalogs is inherently a research project, and will require human control and -intervention. The IPP must provide the data access, manipulation and +intervention. The IPP shall provide the data access, manipulation and visualization tools needed to construct these reference catalogs and to assess their quality. In this section, we list the requirements of @@ -1635,5 +1764,5 @@ The IPP will generate an astrometric reference on the basis of the -observations obtained by the AP survey. The IPP must provide the +observations obtained by the AP survey. The IPP shall provide the analysis tools needed to generate the master astrometric reference catalog. Much of the required functionality is covered by the AP @@ -1642,52 +1771,66 @@ \begin{enumerate} -\item The Astrometry Reference creation tools must return the match between +\item The IPP Reference Creation System shall produce an astrometric + reference catalog from the extracted objects within 6 months of the + end of the AP Survey.\VER{TEST}{TLR:3, TLR:4} + +\item The IPP Reference Creation System shall produce an astrometric + reference catalog with an absolute accuracy of \tbr{100 mas} and a + local relative accuracy of \tbr{30 mas} for bright objects.\VER{TEST}{TLR:3} + +\item The IPP Reference Creation System shall produce an astrometric + reference catalog with proper motions measurements for + non-solar-system objects with an accuracy of \tbr{20 mas / year} for + unsaturated, bright stars.\VER{TEST}{TLR:3} + +\item The Astrometry Reference creation tools shall return the match between stars observed with Pan-STARRS and any of several astrometric -reference catalogs listed in Table~\ref{AstroRefs}. - -\item The tools must convert the reference catalog object coordinates to all +reference catalogs listed in Table~\ref{AstroRefs}.\TASK + +\item The tools shall convert the reference catalog object coordinates to all of the coordinate frames of relevance in the telescope and camera system: \begin{enumerate} -\item Catalog to mean positions -\item Mean to apparent positions -\item Apparent positions + pointing to tangent plane coordinates -\item Apparent positions + pointing to focal plane coordinates -\item focal plane to specific detector (OTA) -\item specific detector to detector cell -\end{enumerate} - -\item The tools must provide the necessary calibration data: the telescope +\item Catalog to mean positions\VER{TEST}{TLR:3} +\item Mean to apparent positions\VER{TEST}{TLR:3} +\item Apparent positions + pointing to tangent plane coordinates\VER{TEST}{TLR:3} +\item Apparent positions + pointing to focal plane coordinates\VER{TEST}{TLR:3} +\item focal plane to specific detector (OTA)\VER{TEST}{TLR:3} +\item specific detector to detector cell\VER{TEST}{TLR:3} +\end{enumerate} + +\item The tools shall provide the necessary calibration data: the telescope and camera optical distortion models and the variation of the image -PSF across the camera field, as a function of color. - -\item The tools must fit the observed stellar coordinates to the astrometric +PSF across the camera field, as a function of color.\TASK + +\item The tools shall fit the observed stellar coordinates to the astrometric reference catalog coordinates to determine improved astrometric -solutions for both the stars and the detectors. - -\item The tools must determine improved telescope optical distortion models -based on the astrometric solutions. - -\item The tools must optionally determine the fit coefficients as a function -of position along, or with combinations of magnitude or color. - -\item The fitting method must include robust outlier rejection. - -\item The tools must identify objects which are detected in the catalog, but -not the science image or vice-versa. - -\item The tools must determine average centroiding errors for each object. - -\item The tools must plot the fit residuals against a wide variety of -parameters: the object positions, magnitudes, colors, etc. - -\item The tools must allow the fit to exclude subsets of objects from the -fits on the basis of these parameters. . - -\item The tools must provide coordinates of the guide stars in the same frame -of reference as the normal image data. - -\item The tools must perform the various fitting steps for the guide stars -rather than for the normal image data. +solutions for both the stars and the detectors. \TASK + +\item The tools shall determine improved telescope optical distortion models +based on the astrometric solutions. \VER{TEST}{TLR:3} + +\item The tools shall optionally determine the fit coefficients as a function +of position along, or with combinations of magnitude or color. \VER{TEST}{TLR:3} + +\item The fitting method shall include robust outlier rejection. \VER{TEST}{TLR:3} + +\item The tools shall identify objects which are detected in the catalog, but +not the science image or vice-versa.\TASK + +\item The tools shall determine average centroiding errors for each object.\TASK + +\item The tools shall plot the fit residuals against a wide variety of +parameters: the object positions, magnitudes, colors, etc.\TASK + +\item The tools shall allow the fit to exclude subsets of objects from the +fits on the basis of these parameters.\TASK + +\item The tools shall provide coordinates of the guide stars in the +same frame of reference as the normal image data to within 30 +mas.\VER{TEST}{TLR:3} + +\item The tools shall perform the various fitting steps for the guide stars +rather than for the normal image data.\TASK \end{enumerate} @@ -1703,7 +1846,7 @@ & mmag & mag & \\ \hline -SDSS & & & \\ -CFHT-LS & & & \\ -Landolt & & & \\ +SDSS & 15? & 16? & {\em u,g,r,i,z} \\ +CFHT-LS & 10? & 18 & {\em u,g,r,i,z} \\ +Landolt & 10-20 & 15 & {\em U,B,V,R,I} \\ \hline \end{tabular} @@ -1712,5 +1855,5 @@ The IPP will generate a photometric reference catalog on the basis of -the observations obtained by the AP survey. The IPP must provide the +the observations obtained by the AP survey. The IPP shall provide the analysis tools needed to generate a master photometric reference catalog. Much of the required functionality is covered by the AP @@ -1719,47 +1862,60 @@ \begin{enumerate} -\item The Photometry Reference creation tools must return the match between +\item The IPP Reference Creation System shall produce a photometric + reference catalog from the extracted point-source objects within 6 + months of the end of the AP Survey.\VER{TEST}{TLR:1} + +\item The IPP Reference Creation System shall produce a photometric + reference catalog with a consistency across the sky of \tbr{5 + millimag}.\VER{TEST}{TLR:1} + +\item The IPP Reference Creation System shall produce a photometric + reference catalog with an absolute calibration to the external + system (defined by \tbr{SDSS} and the CFHT Legacy Survey Standards) + with an accuracy of \tbr{10 millimag} (for bright objects).\VER{TEST}{TLR:1} + +\item The Photometry Reference creation tools shall return the match between stars observed with Pan-STARRS and any of several photometric -reference catalogs listed in Table~\ref{PhotoRefs}. - -\item The tools must convert between different photometric systems, including: -\begin{enumerate} -\item instrumental to nominal detector magnitude -\item nominal detector magnitude to average filter system -\item average filter system to reference photometry system -\end{enumerate} - -\item These transformations must account for color and airmass terms. - -\item The tools must measure and apply relative photometry corrections -between images. - -\item The tools must determine photometric transformation fit coefficients +reference catalogs listed in Table~\ref{PhotoRefs}.\TASK + +\item The tools shall convert between different photometric systems, including: +\begin{enumerate} +\item instrumental to nominal detector magnitude\VER{TEST}{TLR:1} +\item nominal detector magnitude to average filter system\VER{TEST}{TLR:1} +\item average filter system to reference photometry system\VER{TEST}{TLR:1} +\end{enumerate} + +\item These transformations shall account for color and airmass terms. \VER{TEST}{TLR:1} + +\item The tools shall measure and apply relative photometry corrections +between images.\VER{TEST}{TLR:1} + +\item The tools shall determine photometric transformation fit coefficients as a function of airmass, magnitude, color and detector coordinates, -or with combinations of the above. - -\item The fitting method must include robust outlier rejection. - -\item The tools must reject specific objects from the fit on the basis of +or with combinations of the above.\TASK + +\item The fitting method shall include robust outlier rejection.\VER{TEST}{TLR:1} + +\item The tools shall reject specific objects from the fit on the basis of object locations, instrumental magnitudes, observed and reference -errors, and in particular time of the observations. - -\item The tools must plot the fit residuals against a wide variety of -parameters, including the object positions, magnitudes, colors, etc. - -\item The tools must provide photometry from the guide stars in the same -system as observations of stars from the normal imaging data. - -\item The tools must perform the above fitting steps for the guide stars -rather than for the normal image data. +errors, and in particular time of the observations. \TASK + +\item The tools shall plot the fit residuals against a wide variety of +parameters, including the object positions, magnitudes, colors, etc.\TASK + +\item The tools shall provide photometry from the guide stars in the same +system as observations of stars from the normal imaging data.\VER{TEST}{TLR:1} + +\item The tools shall perform the above fitting steps for the guide stars +rather than for the normal image data.\TASK \end{enumerate} \subsection{Modules} -In order to encapsulation functionality, the analysis stages are +In order to encapsulate functionality, the analysis stages are constructed of a sequence of steps. The analysis stages consist of a -\tbd{python} script which executes a sequence of C-level functions. -The C-level functions called by the \tbd{python} script are called -{\em modules} and represent basic data analysis operations. +high-level script which executes a sequence of C-level functions. The +C-level functions executed by the script are called {\em modules} and +represent basic data analysis operations. The required set of Pan-STARRS modules and their functionality is @@ -1794,68 +1950,107 @@ \begin{enumerate} -\item Certain IPP programs must be able to read and write standard - FITS images. - -\item Certain IPP programs must be able to read and write files in +\item Certain IPP programs shall be able to read and write standard + FITS images.\VER{TEST}{allocated} + +\item Certain IPP programs shall be able to read and write files in modified FITS format with Pan-STARRS definitions for non-square - pixel arrays. + pixel arrays.\VER{TEST}{allocated} \end{enumerate} \subsubsection{Table Formats} -Certain IPP programs must be able to read and write FITS tables. +Certain IPP programs shall be able to read and write FITS tables.\VER{TEST}{allocated} \subsubsection{Other Data Formats} -Certain IPP program must be able to read and write XML files. +Certain IPP program shall be able to read and write XML files.\VER{TEST}{allocated} \subsubsection{External Catalogs} -The IPP AP Database must be able to interact with the following -externally provided reference catalogs: - -\begin{enumerate} -\item Hipparcos -\item Tycho2 -\item HST-GSC -\item USNO-A -\item UCAC -\item 2Mass -\item USNO-Bx -\item YBx -\end{enumerate} +The IPP AP Database shall be able to interact with the following +externally provided reference catalogs listed in Table~\ref{AstroRefs} +and Table~\ref{PhotoRefs}.\VER{TEST}{TLR:1, TLR:3} \subsubsection{Analysis Reference Data} -The IPP must store reference data describing the following entities: - -\begin{enumerate} -\item Telescopes -\item Cameras -\item Detectors -\item Filters -\item software basic parameters -\item computer configuration -\end{enumerate} +The IPP shall store reference data describing the relevant Pan-STARRS +and IPP components, including the telescope, camera, detectors, +filters, clustered computers, and IPP software parameters. + +\subsubsection{Static Sky Pixel Size} + +The IPP static sky shall have a pixel scale of \tbr{0.2\arcsec}. \subsection{External Interfaces} -Images from OATS / Camera data store. - -Summit Metadata from OATS. - -Image Q/A assessment to OATS - -3$\sigma$ transient non-orphaned detections to MOPS - -3$\sigma$ transient detections to Transient Science Client - -Published static sky images to Science Database - -Published objects (P2, P4S, P4D, SS) to Science Database +The IPP shall interact with several Pan-STARRS systems and with the +Client Science Pipelines, but it has no requirements to interact with +external systems which are not associated with the Pan-STARRS project. + +\begin{enumerate} + +\item The IPP shall receive Metadata data from OTIS.\TASK + +\item The IPP shall send image quality assessments to OTIS.\TASK + +\item The IPP shall receive raw images from the Camera System.\TASK + +\item The IPP shall send 3$\sigma$ transient non-orphaned detections + to MOPS.\TASK + +\item The IPP shall send all 3$\sigma$ transient detections to + Transient Science Client.\TASK + +\item The IPP shall send static sky images to Science Database after + they are released for publication.\TASK + +\item The IPP shall send object detections from Phase 2, Phase 4 (Sum + and Difference detections) and the static sky images to the + Published Science Products System after they are release for + publication.\TASK +\end{enumerate} \subsection{Internal Interfaces} +The IPP has internal interfaces between several of the architectural +components and between the architectural components and the analysis +stages. + +\begin{enumerate} + +\item IPP Scheduler - IPP Controller. The IPP Scheduler shall send to +the IPP Controller information about the tasks to be performed and +shall receive from the IPP Controller descriptions of the success or +failure of these tasks.\TASK + +\item IPP Scheduler - Metadata DB. The IPP Scheduler shall query the +Metadata DB to determine an appropriate course of action. The IPP +Scheduler shall send result and status information to the Metadata +DB.\TASK + +\item IPP Controller - Analysis Tasks. The IPP Controller shall +initiate the Analysis Tasks and monitor their output and exit +status.\TASK + +\item Analysis Tasks - Metadata DB. The Analysis Tasks shall be able +to query the Metadata DB as needed to extract metadata needed for a +given task. The Analysis Tasks shall be able to send results and +updates to the Metadata DB.\TASK + +\item Analysis Tasks - Image Server. The Analysis Tasks shall be able +to extract relevant images from the Image Server. The Analysis Tasks +shall be able to send output images to the Image Server.\TASK + +\item Analysis Tasks - AP DB. The Analysis Tasks shall be able to +extract information related to specific objects from the Astrometric +and Photometric Database. The Analysis Tasks shall be able to send +result detections to the AP Database.\TASK +\end{enumerate} + \subsection{Internal Data Requirements} + +The internal data requirements of the IPP are left as detailed design +decisions, and are specified within the IPP Supplementary Design +Requirements Documents for the IPP Modules and Library. \subsection{Computer Hardware} @@ -1866,5 +2061,5 @@ hardware requirements addressed in this section consist of: -\begin{enumerate} +\begin{itemize} \item Total Disk Volume \item Total Processing Power @@ -1872,5 +2067,6 @@ \item Sustained Node Network I/O \item Sustained Disk I/O -\end{enumerate} +\item Availabilty +\end{itemize} The report, `The Pan-STARRS Image Processing Pipeline Computational @@ -1879,5 +2075,5 @@ configuration and the PS-4 configuration, under multiple assumptions regarding the data volume. The requirements in this section are -derived from that report, and follow the minimal data volumne +derived from that report, and follow the minimal data volume assumptions for PS-1. @@ -1889,10 +2085,10 @@ \hline Raw data & 200 TB \\ -static sky & 235 TB \\ -calibration frames & 1.8 TB \\ -metadata db & 0.2 TB \\ -AP db & 24 TB \\ +static sky & 350 TB \\ +calibration frames & 2.8 TB \\ +metadata db & 0.3 TB \\ +AP db & 55 TB \\ \hline -total & 461 TB \\ +total & 610 TB \\ \hline \end{tabular} @@ -1909,53 +2105,46 @@ \begin{enumerate} -\item The IPP must store all raw images from the first year from the +\item The IPP shall store all raw images from the first year from the AP and IVP surveys. This corresponds to 175,000 images, or 175 TB, - assuming 1 GB per image and compression. The IPP will require space - for 200 TB of raw imagery to store the data from these two survey - components along with raw calibration, test, and other raw images - not in the AP and IVP surveys. - -\item The IPP must store a single copy of the complete static sky in - all four filters. With the assumed image sampling of 0.2 arcsec per - pixel, this corresponds to 9.7 Tpix per filter, or a total of 235 TB - for the 6 filters, with 2 bytes for the noise map and 2 bytes for - the image map. - -\item The IPP must also store other, smaller collections of data. The - other components contribute only a small fraction of the data - storage requirement. The metadata is a fraction of a terabyte, - while the calibration frames (all master detrend frames) represent - at most a few terabytes. The AP object and detection data make up a - total of 24 terabytes (see Table~\ref{APrates}). - -\item The IPP must have storage capacity for a total of 461 TB of data. + assuming 1 GB per image with compression. The IPP will require + space for 200 TB of raw imagery to store the data from these two + survey components along with raw calibration, test, and short-term + storage of other raw images not in the AP and IVP + surveys.\VER{INSPECT}{TLR:23} + +\item The IPP shall store a single copy of the complete static sky in + all 6 filters. With the assumed image sampling of 0.2 arcsec per + pixel, this corresponds to 9.7 Tpix per filter, or a total of 350 TB + for the 6 filters, with 4 bytes for the image pixels and 2 bytes for + the noise map pixesl.\VER{INSPECT}{TLR:6, TLR:11} + +\item The IPP shall store all detections from the AP, IVP, and MVP + Surveys. These detections make up a total of 55 terabytes (see + Table~\ref{APrates}). \VER{INSPECT}{TLR:24} + +\item The IPP shall store all metadata and master calibration images + from two years of PS-1 operation. The metadata is a fraction of a + terabyte, while the calibration frames (all master detrend frames) + represent at most 2 terabytes. \VER{INSPECT}{TLR:25} + +\item The IPP shall have storage capacity for a total of 610 TB of data. \end{enumerate} \subsubsection{CPU Requirements} -The IPP must provide sufficient computing resources to keep up with -the data analysis tasks. The minimal processing requirement is that -the analysis of a typical night's worth of data be completed within 12 -hours of the start of the night. With a typical night length of 8 -hours, and a maximum read rate of 1 image every 30 seconds, this -implies an average of 45 seconds per image. - -The science image analysis dominates the processing requirements. -Within the science image analysis, Phase 2 and Phase 4 dominate the -processing requirements. These two phases are performed in sequence -with separate computers performing the analyses. They may therefore -be addressed independently. - -\begin{enumerate} -\item The IPP must perform the Phase 2 analysis within an average time of 45 -seconds per single Gigapixel camera image. The Phase 2 analysis has -been measured to require 3200 GHz-sec on a x86/32 bit machine, -implying a requirement of NN GHz for the Phase 2 analysis, if NN sec -are devoted to I/O. - -\item The IPP must perform the Phase 4 analysis on a set of 4 input frames -within an average time of 180 seconds. The Phase 4 analysis has been -measured to require a total of 7800 GHz-sec on an x86/32 bit machine -for a major frame of 4 input Gigapixel camera images. +\begin{enumerate} +\item The IPP shall provide sufficient computing resources to process +images obtained at a cadence of 1 image per 40 seconds.\VER{TEST}{TLR:17} + +\item The IPP shall perform the Phase 2 analysis within an average +time of 40 seconds per single Gigapixel camera image. The Phase 2 +analysis has been measured to require 3200 GHz-sec on a Pentium-4 +machine.\VER{TEST}{TLR:17} + +\item The IPP shall perform the Phase 4 analysis on a set of 4 input +frames within an average time of 180 seconds. The Phase 4 analysis +has been measured to require a total of 7800 GHz-sec on a Pentium-4 +machine for a major frame of 4 input Gigapixel camera +images.\VER{TEST}{TLR:17} \end{enumerate} @@ -1967,7 +2156,7 @@ raw images and the corresponding detrend images, and that all Phase 4 processing requires complete network distribution of both the initial -and updated static sky images, the total I/O for a 180 second +and updated static sky images, the total I/O for a 160 second major-frame period is: -\begin{enumerate} +\begin{itemize} \item 8 GB from summit to Phase 2 (4 images @ 2 GB each) \item 18 GB from Phase 2 to Phase 4 (3 bytes per pixel for image + @@ -1976,16 +2165,14 @@ input image pixel, 4 bytes per pixel). \item 9 GB from Phase 4 to Static Sky -\end{enumerate} -for a grand total of 44 GB over 180 seconds, or 244 MB/second, of -which 26 GB are processed by the Phase 2 nodes and 36 are processed by -the Phase 4 nodes. The IPP must be capable of sustaining this network -load. - -\paragraph{Disk I/O Requirements} - -The disk I/O requirements are determined by the total number of bytes -read from and written to disk. For each major frame processed, the -total I/O to and from disk for Phase 2 is: -\begin{enumerate} +\end{itemize} +for a total of 44 GB, of which 26 GB are used by the Phase 2 nodes and +36 are used by the Phase 4 nodes. The IPP shall be capable of +sustaining this network load.\VER{TEST}{TLR:17} + +\paragraph{Phase 2 Disk I/O Requirements} + +For each major frame processed, the total I/O to and from disk for +Phase 2 is: +\begin{itemize} \item 8 GB raw image from summit to Phase 2 nodes (4 images @ 2 GB each) \item 8 GB raw image from Phase 2 disk to memory @@ -1994,26 +2181,34 @@ + 1 byte mask). \item 18 GB processed image from Phase 2 disk to Phase 4 -\end{enumerate} -for a grand total of 86 GB I/O for Phase 2. Equivalently, for each -major frame processed, the total I/O to and from disk for Phase 4 is: -\begin{enumerate} -\item 18 GB processed image from Phase 2 disk to Phase 4 +\end{itemize} +for a total of 86 GB Disk I/O for Phase 2 for a complete major frame.\VER{TEST}{TLR:17} + +\paragraph{Phase 4 Disk I/O Requirements} +For each major frame processed, the total I/O to and from disk for +Phase 4 is: +\begin{itemize} \item 9 GB static image from Phase 4 disk to memory \item 9 GB static image from memory to Phase 4 disk -\end{enumerate} -for a total of 36 GB I/O for Phase 4. - -\subsubsection{Per-Node I/O Requirements} - -Data I/O per node is defined as the number of bytes per second passed -through the node's network adapter. The data I/O per node is tied to -the total processing power and the total number of nodes. A useful -way to examine the per-node I/O requirements is to compare the I/O and -CPU requirements to determine the required number of processing nodes. -The assumption is made that each CPU is associated with a single disk -RAID which may deliver data at a rate of 100 MB/sec and a GigE -ethernet controller which may deliver data at a sustained rate of 50 -MB/sec, and that each CPU is equivalent to 4 GHz. The IPP must -therefore have a total of 26 Phase 2 nodes and 16 Phase 4 nodes. +\end{itemize} +for a total of 18 GB I/O for Phase 4 for a complete major frame.\VER{TEST}{TLR:17} + +\subsubsection{Total Node Requirements} + +The I/O and CPU requirements above may be confronted with reasonable +assumptions of bandwidth and CPU speeds to estimate the number of +nodes required for the IPP. Each CPU is matched with one network +adapter and one disk array. : +\begin{enumerate} +\item The IPP requires at least 40 Phase 2 Nodes (OTA Nodes)\VER{TEST}{TLR:17} +\item The IPP requires at least 5 TB for each Phase 2 node\VER{TEST}{TLR:17} +\item The IPP requires at least 25 Phase 4 Nodes (Static Sky Nodes)\VER{TEST}{TLR:17} +\item The IPP requires at least 14 TB for each Phase 4 node\VER{TEST}{TLR:17} +\item The IPP requires at least 10 AP DB Nodes\VER{TEST}{TLR:17} +\end{enumerate} + +\subsubsection{Availability} + +The IPP Image Server nodes shall not be offline for more than 12 hours + consecutively or 36 hours per year.\VER{ANALYSIS}{TLR:17} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -2021,6 +2216,6 @@ \section{Test Verification} -A testing regime must be implemented to demonstrate the working state -of the provided software. Certain tests as specified must be +A testing regime shall be implemented to demonstrate the working state +of the provided software. Certain tests as specified shall be performed by MHPCC, with code release contingent on success. Other specified tests will be performed by IfA to verify the validity of the @@ -2031,5 +2226,5 @@ \subsection{Software Configuration Tests} -MHPCC must test the validity of the software configuration, +MHPCC shall test the validity of the software configuration, specifically to check that the code can be compiled on the specified platforms and that the compilation produces no errors or warnings, @@ -2039,26 +2234,27 @@ \begin{enumerate} -\item MHPCC must test that the code does not produce memory leaks. - -\item MHPCC must test that the code does not produce segmentation faults. +\item MHPCC shall test that the code does not produce memory leaks. + +\item MHPCC shall test that the code does not produce segmentation faults. \end{enumerate} \subsection{Basic Unit Tests} -MHPCC must perform basic unit tests with sample input data and known +MHPCC shall perform basic unit tests with sample input data and known output results, including invalid input data to test error handling. -These tests must exercise the complete range of module options. +These tests shall exercise the complete range of module options. \subsection{Detailed Functional Analysis} -IfA must perform detailed tests with a wide range of input data and +IfA shall perform detailed tests with a wide range of input data and compare the results with existing software system. \subsection{Test Verification Matrix} -\subsubsection{Pan-STARRS IPP Library} - -See Appendix A \& B of the IPP Library SDR (PSDC-430-007) for the test -verification matrices for the Pan-STARRS IPP Library +Test Verification Matrix information is supplied with each identified +requirement in this document. + +\subsection{Trace Matrix} +\input{ippSRStrace.tex} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%