Index: trunk/doc/design/ippSRS.tex =================================================================== --- trunk/doc/design/ippSRS.tex (revision 771) +++ trunk/doc/design/ippSRS.tex (revision 810) @@ -1,3 +1,3 @@ -%%% $Id: ippSRS.tex,v 1.1 2004-05-25 00:38:56 eugene Exp $ +%%% $Id: ippSRS.tex,v 1.2 2004-05-29 00:56:14 eugene Exp $ \documentclass[panstarrs]{panstarrs} @@ -25,11 +25,13 @@ DR.02 & 2003.03.10 & Second draft \\ \hline DR.03 & 2003.04.13 & Most paragraphs fleshed out \\ \hline +DR.04 & 2003.04.27 & Basic text frozen for internal review \\ \hline +DR.05 & 2003.05.24 & Incorporating comments from internal review \\ \hline \RevisionsEnd + +\tableofcontents +\pagebreak \listoffigures \pagebreak - -\tableofcontents -\pagebreak \pagenumbering{arabic} @@ -40,5 +42,5 @@ \subsection{Identification} -This document establishes the system requirements for the Pan-STARRS +This document establishes the software requirements for the Pan-STARRS Image Processing Pipeline (IPP) as applied to Pan-STARRS 1 (PS-1), the initial demonstration telescope to be constructed on Haleakala by Jan @@ -56,16 +58,17 @@ that series is implied. -Open Issues and TBDs in this document are marked \tbd{in bold, red -with surrounding square brackets}. - -All timing measurements are to execution time as measured on a -\tbd{Reference Pan-Starrs Computation Node} and assumed to be not -limited by network bandwidth. - -\subsubsection{Definitions} +Open issues (TBDs) in this document are marked \tbd{in bold, red with +surrounding square brackets}. + +Quantities which should be reviewed (TBRs) are marked \tbr{in bold, +blue with surrounding square brackets}. + +\subsubsection{Requirements Definitions} \paragraph{``Must''} When used in this specification, the word ``must'' refers to an explicit requirement of a system component or -the complete system. +the complete system. In this document, the use of the word ``must'' +replaces, and is equivalent to, use of the word ``shall'' found in +many requirements documents. \paragraph{``Should''} When used in this specification, the word @@ -80,8 +83,8 @@ \DocumentsInternalSection -PSDC-430-xxx & PS-1 Design Reference Mission \\ \hline +PSDC-130-001 & PS-1 Design Reference Mission \\ \hline PSDC-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline PSDC-430-006 & Pan-STARRS IPP ADD \\ \hline -PSDC-430-007 & Pan-STARRS IPP PSLib SDR \\ \hline +PSDC-430-007 & Pan-STARRS IPP PSLib SDRS \\ \hline \DocumentsExternalSection Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ @@ -92,32 +95,9 @@ \section{Requirements} -\subsection{Required States} - -The IPP must have 3 states: active, paused, and interactive. - -\subsubsection{Active State} -\label{req:active-state} - -In active state, the IPP must accept images and metadata from OATS and -automatically perform the complete set of image processing tasks, -including both calibration and science image processing. The IPP must -respond to requests for data from the client science pipelines -\tbd{and IPP monitoring team}. - -\subsubsection{Paused State} -\label{req:paused-state} - -In paused state, the IPP must refuse data and metadata from OATS and -data requests from the client science pipelines. - -\subsubsection{Interactive State} -\label{req:interactive-state} - -In interactive state, the IPP must accept data and metadata from OATS, -but must not automatically process the data. The IPP must respond to -user commands to initiate portions of the data analysis. - -\subsection{System Capability Requirements} +\subsection{Science Requirements} \label{req:system-capabilities} + +\tbd{distinguish data products in commissioning, during PA survey, +after PA survey} The IPP must perform the following tasks: @@ -186,7 +166,33 @@ \end{enumerate} -\subsubsection{Software Coding Requirements} - -\paragraph{Languages} +\subsection{Required States} + +The IPP must have 3 states: active, paused, and interactive. + +\subsubsection{Active State} +\label{req:active-state} + +In active state, the IPP must accept images and metadata from the +external sources (i.e., the summit) and automatically perform the +complete set of image processing tasks, including both calibration and +science image processing. The IPP must respond to requests for data +from client science pipelines. + +\subsubsection{Paused State} +\label{req:paused-state} + +In paused state, the IPP must refuse incoming data and metadata and +data requests from the client science pipelines. + +\subsubsection{Interactive State} +\label{req:interactive-state} + +In interactive state, the IPP must accept imcoming data and metadata, +but must not automatically process the data. The IPP must respond to +user commands to initiate portions of the data analysis. + +\subsection{Software Coding Requirements} + +\subsubsection{Languages} \label{req:languages} @@ -196,26 +202,32 @@ Scripting language must be \tbd{Python, version TBD}. -\paragraph{Interfaces} -\label{req:interfaces} - -Access to low-level Library functions must be provided via C APIs -consisting of the function calls and the defined data structures and -other data types. Access to high-level functions must be provided -via C APIs as well as SWIG interfaces, where specified. Access to -processing jobs must be available via the UNIX shell. - -\paragraph{Coding Standards} - -The C code must comply with ANSI Standard C99. Because the delivered -code is required to run on UNIX machines, the delivered code must be -in compliance with the language-independent UNIX operating system -standard POSIX (Open Group Based Specifications Issue 6, IEEE Std -1003.1, 2003). Source code files must use the UNIX line-break -convention (line-feed only). C coding style must adhere to the -standard defined in the document 'Pan-STARRS C-coding standard' -(PSDC-430-004). \tbd{Python coding must follow the Python standard -defined in the document TBD}. - -\paragraph{Naming Conventions} +\subsubsection{Interfaces} +We require the following types of interfaces: +\begin{enumerate} +\item Access to low-level Library functions must be provided via C +APIs consisting of the function calls and the defined data structures +and other data types. +\item Access to high-level functions must be provided via C APIs as +well as SWIG interfaces, where specified. +\item Access to processing jobs must be available via the UNIX shell. +\end{enumerate} + +\subsubsection{Coding Standards} + +\begin{enumerate} +\item The C code must comply with ANSI Standard C99. +\item Because the delivered code is required to run on UNIX machines, +the delivered code must be in compliance with the language-independent +UNIX operating system standard POSIX (Open Group Based Specifications +Issue 6, IEEE Std 1003.1, 2003). +\item Source code files must use the UNIX line-break +convention (line-feed only). +\item C coding style must adhere to the standard defined in the +document 'Pan-STARRS C-coding standard' (PSDC-430-004). +\item \tbd{Python} coding must follow the standard defined in the +document \tbd{TBD}. +\end{enumerate} + +\subsubsection{Naming Conventions} Header files must have names starting \code{ps} or \code{p_ps} for @@ -224,10 +236,10 @@ for the public header files. -Functions visible at global scope which are part of the public API -must have names begining with \code{ps}, and follow the naming -conventions in the coding standard. Functions that are visible at -global scope but which are not part of the public interface must have -names begining with \code{p_ps}. Functions that are local to a file -must \textit{not} start \code{ps} (or \code{p_ps}). +Functions visible at global scope that are part of the public API must +have names begining with \code{ps} and follow the naming conventions +in the coding standard. Functions visible at global scope but which +are not part of the public interface must have names begining with +\code{p_ps}. Functions that are local to a file must \textit{not} +start \code{ps} (or \code{p_ps}). Variables visible at global scope which are part of the public API @@ -251,5 +263,5 @@ \code{psEquatorial2Ecliptic}). -\paragraph{C Programming Guidelines} +\subsubsection{C Programming Guidelines} Functions that assign to a variable must list that argument @@ -290,5 +302,5 @@ \end{itemize} -\paragraph{Commenting and Documentation} +\subsubsection{Commenting and Documentation} Commenting of delivered C and Python code must follow the C and @@ -304,9 +316,9 @@ documentation must be delivered as PDF documents. -\paragraph{Version Control} +\subsubsection{Version Control} Source code version control must be implemented with CVS. -\paragraph{CSCI Deliverable} +\subsubsection{CSCI Deliverable} All final source code generated for the IPP is to be delivered via @@ -314,5 +326,5 @@ and made available via CVS. -\paragraph{Platform architectures and operating systems} +\subsubsection{Platform architectures and operating systems} Makefiles must be provided with appropriate flags set so that all @@ -333,22 +345,32 @@ x86/Linux combination. -\paragraph{Software Configuration} +All timing measurements are to execution time as measured on a +\tbd{Reference Pan-Starrs Computation Node} and assumed to be not +limited by network bandwidth. + +\subsubsection{Software Configuration} \tbd{deferred} -\subsubsection{Architectural Components} - -In order to achieve the required functionality, it is necessary to -divide the IPP into a number of clearly-defined software elements, -listed as follows: +\subsection{Architectural Components} + +As discussed in the Pan-STARRS System Concept Definition, the IPP is +organized into a number of clearly-defined software elements. The SCD +provides a detailed description of the roles and responsibilities of +these subsystems. In brief, the IPP consists of a collection of +science analysis stages, a set of architectural components which +provide the infrastructure needed to run the analysis programs, and a +collection of hardware on which all of the software elements exist. + +The architectural components consist of: \begin{enumerate} -\item {\bf Pixel Server:} This component is a large data store for all +\item {\bf Image Server:} This component is a large data store for all images used by the IPP, including the raw images from the telescope, the master calibration images, the reference static-sky images, and - any temporary image data products produced by the IPP. The Pixel + any temporary image data products produced by the IPP. The Image Server is required to meet all of the image storage needs identified - in the top-level requirements above. The Pixel Server must accept + in the top-level requirements above. The Image Server must accept the incoming data and store it until it is no longer needed by other portions of the IPP. @@ -364,10 +386,4 @@ as needed to perform the analysis specified above. -\item {\bf Analysis Stages:} Specific programs are required to perform - the processing steps listed above. These can be divided into - well-defined analysis stages, each of which operates on a particular - unit of data, such as a single OTA image or a collection of - astronomical objets. - \item {\bf Controller:} In order to perform the analysis stages required by the IPP, it is necessary to use distributed computing @@ -389,5 +405,5 @@ \begin{figure} \begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/overview.ps}} +\resizebox{8cm}{!}{\includegraphics{pics/overview}} \caption{ \label{overview} IPP System Overview} \end{center} @@ -396,158 +412,172 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\paragraph{Pixel Server} - -The IPP Pixel Server \tbd{rename as Image Server?} is a large data -store for all images used by the IPP. The Pixel Server is required to -store all of the images needed by the IPP for the length of time they -are required; total data volume is specified in detail in the hardware -summary, but is in the vicinity of \tbd{700 TB}. - -The IPP Pixel Server must maintain a record of all images currently -available in the repository \tbd{and all no longer available}. This -record must include the image name, location (which machine), the -state of the image (available, deleted), the image size, the image -type, and the existence and location of secondary copies of the image. -This information need not include other metadata such as the image -summary statistics or the state of the image processing for the image, -as these aspects are included in the Metadata DB. - -The IPP Pixel Server must store images as FITS files on disk. Raw -images from the telescope must be stored as individual OTA images for -each file, with multiple Cell images per file as well as video -sequences from the guide stars. Images of the Static Sky must be -stored in the form of \tbd{triangular segments} to minimize the total -data volume and pixel overlap. - -The IPP Pixel Server must distribute images across a cluster of -machines. The IPP Pixel Server must be capable of honoring requests -to store an image on a specific machine. If such a request cannot be -honored, the IPP Pixel Server must select an appropriate machine and -notify the requesting agent of the new locations. The IPP Pixel -Server must provide a mechanism to maintain multiple (at least two) -copies of each image. - -The IPP Pixel Server must interface with other subsystems of the IPP. -It must provide an interface to other IPP subsystems to identify the -image location (the computer on which it resides). It must provide a -mechanism to serve a specified image to another IPP or Pan-STARRS -subsystem. It must provide a mechanism for deletion of images in the -Pixel Server. It must have a mechanism to accept or retrieve an image -from another Pan-STARRS subsystem, in particular OATS. Communication -of messages between the IPP Pixel Server and other subsystem must be -via \tbd{XML messages} passed via \tbd{some transport}. - -The IPP Pixel Server must accept images at the telescope maximum rate -of 1 full-camera image every 30 seconds. The IPP Pixel Server must -therefore accept notifications and process retrievals at a rate of 64 -raw OTAs in 30 seconds. - -\tbd{O/S, language, SQL, ODBC requirements?} - -\tbd{hardware requirements?} - -\tbd{communication protocols?} - -\paragraph{P\&A Database} - -The IPP requires a mechanism to store data related to astronomical -objects derived from various sources with a variety of associations. -The PnA (Photometry and Astrometry) Database serves this function. -The PnA Database deals with two related concepts: {\em objects} and -{\em detections}. The objects are descriptions of astronomical -objects while the detections are the specific measurements of those -objects on an image. A collection of {\em detections} may be used to -derive average quantities which describe a particular {\em object}. - -The PnA Database must store the collections of detections which were -derived from specific images from any of the analysis stages. It must -be possible to determine and locate (perhaps via interactions with the -pixel server) the image from which a specific detection was derived. -It must also be possible to extract all detections derived from a -specific image. These associations must include descriptive -information including the coordinates of the detection on the image. - -The PnA Database must provide a mechanism to associate together -multiple detections of a specific object. Several major classes of -objects will be present, each of which must be handled correctly. - -First, the most distant stars, compact galaxies, and QSOs will have -nearly fixed locations relative to other nearby stars, with only small -deviations for individual measurements. The association between -multiple detections of such objects must be made on the basis of their -coincident positions. The PnA Database must be able to determine the -average position of the object and the deviations of the individual -detections from that average. - -Second, solar system objects do not have a fixed location and -detections of such objects must associated on the basis of their -coincidence with the orbit of the objects. The PnA Database must be -able to associate detections with the orbits of known objects. The -determination of this association is the responsibility of the MOPS -and must be communicated to the IPP PnA Database on \tbd{some -timescale}. The PnD Database must be able to retrieve the detections -associated with the object and to provide the object associated with -the specific detections. This association must include descriptive -information such as the offset of the detection from the predicted -location of the detection based on the orbit. This functionality is -required to allow the PnA Database to ignore known moving object -detections from other types of queries. - -Third, stars in the general vicinity of the solar system fall in -between these first two classes of objects. Their proper motion and -parallax response is significant enough ($>1$ arcsec in 10 years) that -they are not well-described by an average location and a collection of -offsets. These objects must be described by a distance and a proper -motion vector. The PnA Database must be able to find and associate -detections of objects for which either of the parallax or the proper -motion are substantial. - -Fourth, many detections, especially in their initial states, will not -be associated with a specific astronomical object of any of the above -classes and should be treated as orphans. Some of these will be -suprious (not represent real objects), some will be from solar system -objects for which orbits are not yet determined, some will be from -faint stars near the detection limits, some will be from short-term -transients which have only been detected once. The PnA Database must -be able to carry these detections until they have been associated with -one of the objects above. It must be possible to migrate individual -detections associated with an astronomical object back to the orphan -state. - -For every object, and all orphaned detections, it must be possible to -determine the images for which the coordinates were included but for -which no detection was made. The minimum set of information which -must be carried for these non-detections is the image and the -associated object or orphan. - -The PnA Database must store the relationships between various -photometric systems and, in some cases, the evolution of that -relationship. It must be possible, given a determined set of -calibrations, to convert between the measured instrumental magnitude -of a detection with a specific filter, detector, and telescope, and at -particular time and the implied magnitude in the average Pan-STARRS -magnitude systems. It must also be possible, given the magnitudes of -an object in one system to convert those to the magnitudes in another -system; an example of such a conversion is between the average -Pan-STARRS filter systems and the various reference systems -appropriate for those filters. - -The PnA Database must provide interfaces to extract lists of objects -and detections based on various query parameters. It must be possible -to extract all detections associated with a specific object, all -non-detections of that object (or orphan) and summary statistics from -these collections. It must be possible to extract all objects or -detections within specified spatial regions including regions bounded -by great circles (RA,DEC; GLAT,GLON; ELAT,ELON) and regions described -by a location and a search radius. It must be possible to extract the -image parameters associated with a specific detection including image -coordinates of the detection, exposure time, time and date of the -detection, etc. - -\tbd{volume requirements} - -\tbd{speed / access requirements} - -\paragraph{Metadata Database} +\subsubsection{Image Server} + +The IPP Image Server must store images on a distributed collection of +computer disks. Individual instinces of a file are only required to +be stored on a single machine (striping across computers is not a +requirement). + +The IPP Image Server must be capable of honoring requests to store an +image on a specific machine. If such a request cannot be honored (ie, +the machine is down), the IPP Image Server must select an appropriate +machine and notify the requesting agent of the new locations. + +The IPP Image Server be able to maintain multiple copies of each +image, as specified by the user. + +The IPP Image Server must maintain a record of all image copies +currently available in the repository. This record must include the +image name, location (which machine), the image size, and the state of +the image. + +The IPP Image Server must lock images in the repository on request. +Both read (shared) and write (exclusive) locks must be provided. A +read lock must prevent write access to the file; a write lock must +prevent both read and write access. + +The IPP Image Server must return the image location (the computer on +which it resides) upon request. + +The IPP Image Server must return a specified image upon request. + +The IPP Image Server must delete images in the repository on request. + +The IPP Image Server must accept images from the summit at the maximum +rate of 1 full-camera image every 30 seconds. The IPP Image Server +must therefore accept new images into the repository at a rate of 64 +raw OTAs in 30 seconds and a total input data volume rate of 75 +MB/sec. + +\subsubsection{PA Database} + +\begin{table} +\begin{center} +\caption{PA Detection Classes \& Object Parameters\label{PAdetections}} +\begin{tabular}{lrrrr} +\hline +\hline +Object Parameter & P2 & P4S & P4D & SS \\ +\hline +PSF x,y, M, $\sigma_{\rm M}$ & + & + & + & + \\ +$\sigma_x$, $\sigma_y$, covar. & + & + & + & + \\ +exp. spaced aps., Poisson noise, variance & - & - & - & + \\ +streak L, $\phi$, $\sigma_L$, $\sigma_\phi$ & - & - & + & + \\ +$x_g$, $y_g$, flag & + & + & - & + \\ +local sky data & + & + & + & + \\ +Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ +S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ +W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ +star/gal sep, star/streak sep. & - & + & + & + \\ +\hline +deVeucaleur R, M, AB, $\phi$ & - & + & - & + \\ +exponential R, M, AB, $\phi$ & - & + & - & + \\ +\hline +\end{tabular} +\end{center} +\end{table} + +The PA Database must accept and store individual detections and +collections of detections along with information about the image which +provided the detections. + +Detections must be saved as one of several detection classes (P2, P4S, +P4D, SS) and the PA Database must store the appropriate parameters, +listed in Table~\ref{PAdetections}, for each class. + +The PA Database must identify the image which provided the detection, +or in the case of external references, an identifier specific to the +reference source. + +The PA Database must group detections into objects and measure average +parameters of those objects. + +The PA Database must store parallax and proper motion parameters for a +subset of the average objects. + +The PA Database must store image and filter calibration information +necessary to convert between instrumental magnitudes and calibrated +magnitudes in standard systems. + +The PA Database must perform at least the follow queries, with +constraints on the output based on at least time ranges, magnitude +limits, error limits: +\begin{enumerate} +\item given (RA,DEC) and a Radius, return all objects and/or +detections in the region. + +\item given (RA,DEC)_0 - (RA,DEC)_1, return all objects and/or + detections in the region. + +\item given (RA,DEC), return closest object. + +\item given object ID, return all detections + +\item given detection, return source image data. + +\item given (RA,DEC), return all images overlapping coordinate. + +\item given (RA,DEC) and a Radius, return all images overlapping region. + +\item given (RA,DEC)_0 - (RA,DEC)_1, return all images overlapping + region. + +\item given detection instrumental magnitude, return derived + magnitudes based on calibration information. + +\item given a collection of detections, determine the object avergae + magnitude. + +\item given a collection of objects and detections, determine the + individual image zero-points. + +\item given a region, return all possible combinations of the object + or detection magnitudes (M1 - M2). + +\item given a list of (RA,DEC) entries, return all nearest objects. + +\item given a filter, telescope, or detector, return all calibration + terms and history. + +\item given a detection, return all non-detections from images which + overlapped the detection coordinates. + +\end{enumerate} + +The PA Database must accept detection IDs of moving objects and label +the detections with the identified object. + +\begin{table} +\begin{center} +\caption{PA Detection Classes \& Object Parameters\label{PAdetections}} +\begin{tabular}{lrrrr} +\hline +\hline +Quantity & P2 & P4$\Sigma$ & P4$\Delta$ & SS \\ +\hline +detection limit & $20 \sigma$ & $5 \sigma$ & $3 \sigma$ & \\ +depth (r') & 20.8 & 23.0 & & \\ +stars deg$^{-2}$ ($|b|>10$) & $1 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ \\ +stars FPA$^{-1}$ ($|b|>10$) & $7 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ \\ +stars sec$^{-1}$ ($|b|>10$) & $2.3 \times 10^4$ & $1 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ \\ +bytes star$^{-1}$ & 64 & 100 & 64 & \\ +MB sec$^{-1}$ & 1.4 & 2.2 & 0.7 & \\ +PS-1 total TB & 8 & 12 & 4 & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +The PA Database must accept new detections at the rate generated by +the telescope from the Phase 2 and Phase 4 analysis. Except within 10 +degrees of the galactic plane, the PA Database must keep up with the +incoming rates. The expected rates are listed in Table~\ref{PArates}, +along with the total data volume required for storage space over the +PS-1 lifetime. The PA Database must be able to keep up with these +rates. + +\subsubsection{Metadata Database} + +\tbd{this section needs to be reviewed and revised} The IPP requires a Metadata Database to store and provide access to @@ -568,12 +598,14 @@ avoid slowing down the analysis systems. +\tbd{need to extract specific requirements from this} + \tbd{volume requirements} -\tbd{does the description of images belong in the Metadata database or - in the Pixel / Image Server?} - \tbd{queries} -\subparagraph{Configuration Database -- a subset of the metadata database?} +{\bf note: description of images belong in the Metadata database, + location of images is in the Image server} + +\paragraph{Configuration Database} The IPP requires a Configuration Database to store and provide access to @@ -581,210 +613,98 @@ configuration database include the default parameters for the various analysis programs, the description of the computing environment, the -process status information, etc. - -\paragraph{Controller} - -The IPP uses a collection of computers to store and process images and -to manipulate collections of detections. These computers perform any -of a large number of analysis stages or other processing tasks without -significant interprocess communication. It is necessary to have a -mechanism which initiates computing tasks on the different computers, -which monitors the tasks as they are executed, which handles the -output and the errors from these tasks, and which reacts to the -failure of any of the computing nodes. The system responsible for the -tasks in the IPP is the Controller. - -The Controller must interact with the collection of computers under -its management and with other subsystems in the IPP. The controller -must accept a variety of inputs from other subsystems, described -below, and respond accordingly. The controller must also provide -information to other subsystems on demand. - -Computers managed by the controller are allowed to be in one of -several states, and the controller must interact with it in an -appropriate way for each of those states. A computer may be {\tt -alive}, {\tt dead} or {\tt off}. If the computer is {\tt alive}, it -responds to commands from the controller and may be used for tasks -subject to other constraints. If it is {\tt dead}, the computer is -not responsive and must not be used for executing tasks. The -controller must identify computers which have died and occasionally -test them to see if they are {\tt alive} again. Computers which are -{\tt off} are not available for tests and must not be tested. -Computers may be set to the {\tt off} or {\tt dead} states by external -subsystems; it is the responsibility of the Controller to return a -computer to the {\tt alive} state if possible. - -Computers which are in the {\tt alive} state may be in one of two -modes: {\tt busy} and {\tt free}. A computer which is {\tt busy} -currently has a task assigned to it. The controller may only assign -one task to one computer at a time\footnote{A physical piece of -hardware may be defined to the Controller as multiple computers to -allow multi-processor nodes to execute more than one simultaneous -task.}. Computers which are in the {\tt free} state may have tasks -assigned to it. The controller must also manage an additional set of -constraint tables for each machine: the allowed tasks. Each computer -may have a list of allowed tasks which may include {\tt all} tasks, -{\tt none} of the tasks, or specified task names. The controller must -only execute the allowed tasks on a machine. - -The Controller must accept tasks from other IPP subsystems. The task -requests must include the specific command to be executed. The -commands must be in the form of a UNIX command which could be -performed on any of the computing nodes. Any input or output data -structures in the commands must be a valid resource regardless of the -node on which the task is executed. Input and output data resources -must be unique where necessary to avoid conflicts. Tasks must be -given an identifier, which must be returned to the requesting agent, -to be used to control the specific task. - -Task requests may specify a desired node for the task execution. The -Controller must attempt to honor the request if the node is {\tt -alive}, but must execute on another node if the requested one is {\tt -dead} or {\tt off}. Even if a node is {\tt alive} the controller must -choose another node if the specified tasks is not allowed on the -requested node. In all other cases, the controller must wait until -executing processes, and processes with higher priority, are completed -before executing the specified task on the requested node. - -Task requests may specify an urgency level. The controller determines -the priority of the task by sorting first by priority and next by the -sequence of the request. An executing task must be completed before -any new task is started, regardless of priority. Tasks may be -assigned a priority of 0 in which case they are maintained in the -queue and never executed. - -The controller must monitor the output streams from the executing -tasks and the exit status of the tasks. \tbd{where do we send the -output logs?}. The status, including the exit status, of each task -must be maintained for other subsystems to query as needed. \tbd{how -long? on disk / database?} - -The controller must accept commands from other IPP subsystems. These -commands include those which govern the processing of specified tasks, -those which govern the behavior of specific computing nodes, and those -which request information from the controller. The controller must be -able to halt the execution of a specified task, delete an unexecuted -task from the task list, change the priority of tasks, change the -requested nodes for tasks. The controller must also be able to stop -the current execution of a task and push it to the end of the queue -and also change its priority. - -The controller must honor requests (normally from the users) to change -the mode of any computing node on demand between {\tt off} and {\tt -dead}. It must also be able to change the list of allowed tasks as -requested by external commands. - -The controller must respond to informational requests regarding the -collection of machines and their states as well as the collection of -tasks and their states. The controller must monitor the execution -times of the different tasks and provide summary statistics. Finally, -the controller must respond to three top-level commands: {\tt finish}, -{\tt stop} and {\tt abort}. When {\tt finish} is requested, no more -new tasks are accepted on the stack of task, and when all tasks in the -stack have completed, the controller must exit. When {\tt stop} is -requested, the currently executing tasks must be completed at which -point the controller must exit, but tasks remaining in the stack which -have not been started are flushed. When {\tt abort} is issued, the -controller immediately kills all executing tasks and exits. - -\paragraph{Scheduler} - -The IPP is responsible for a variety of analysis tasks: several stages -of processing of the science images; routine assessment of the detrend -images used in processing the science images; construction of -replacement detrend images when needed; generation of astrometric and -photometric reference catalogs based on the collected dataset; and the -performance of test analysis programs. At any point, decisions need -to be made about which of these tasks should be performed, based on an -analysis of the contents of the image database tables, the -requirements of the people monitoring the IPP, and the near-term -observing plans. The IPP Scheduler is a mechanism to manage these -various inputs to guide the decisions and initiate the actions. - -The Scheduler acts as an intermediate between several components of -the IPP and also between the IPP and external agents such as the OATS -system and the users who must monitor the behavior of the IPP. - -The Scheduler must send commands to the Controller for execution. It -is the Controller's responsibility to manage the specific analysis -jobs executing on a given processing node. These analyses may include -the process of copying of moving data from OATS to the pixel server -nodes, or it may involve image processing stages performed on the -science images by the appropriate processing nodes, or it may involve -analysis of the data in the PnA object database. In order to isolate -and encapsulate the responsibilities of the Scheduler and the -Controller, the Scheduler must initiate the tasks which the controller -manages; in this way, the controller does not need to have any -information about the details of the tasks which it executes. -Communication between the Scheduler and the Controller must be -bi-directional; the Scheduler must send tasks to the Controller which -the Controller must inform the Scheduler of the outcome of those -tasks. \tbd{it is not specified whether the scheduler and controller -are components of a single software system or interacting but distinct -software components.} - -The Scheduler must take as input the current list of pending images, -both science and calibration, and a description of the current -observing plan or strategy on some time-scale. The Scheduler must -also take input from humans who manage the IPP. - -The Scheduler must choose between several types of analysis stages -based on the contents of those lists and on the requirements of the -users. The list of tasks which the Scheduler must decide between -includes: -\begin{itemize} -\item moving data to and from the pixel server ($\sim 30$ second timescales) -\item running the science analysis stages ($\sim 30$ second timescales) -\item testing the validity of the current detrend images ($\sim$ - nightly) -\item constructing new detrend images ($\sim$ weekly) -\item updating and improving the photometric and astrometric reference - catalogs ($\sim$ yearly). -\end{itemize} - -The Scheduler must choose between tasks which are relevant on several -different time-scales. The time-scale range from 2 times per minute -to once or twice a year, as noted in the list above. The Scheduler -must also make use of the human input in managing such choices. The -human users must be able to specify that a particular task or set of -tasks is of higher or lower priority than the norm. - -The Scheduler must maintain a set of rules defining the dependency of -one type of analysis stage on other analysis products. For example, -the nightly science image processing depends on the existence of valid -detrend images. The Scheduler must be able to recognize the -dependency and initiate the required analysis needed to perform other -analysis tasks. The Scheduler must have the ability to decide between -postponing an analysis task until the required data are available or -to initiate the task using a lower-quality or less appropriate -substitute. For example, in normal circumstances, a science image -must not be processed until the corresponding detrend frame has been -produced. However, if such a frame is unlikely to appear soon, and -the pressure to process the science image is sufficiently high, then -the frame could be processed with an older detrend frame of known -lower quality. The Scheduler must have the ability to choose the -best, if not ideal, reference data for a particular circumstance. - -The Scheduler is responsible for setting the operating mode of the -IPP. When the IPP is in the automatic operating mode, this implies -that the Scheduler is performing the most appropriate tasks at a -particular time. When the IPP is in the interactive mode, the -Scheduler must perform the requested action regardless of the outcome -of the decision trees. In addition, the Scheduler must only perform -the requested actions and not attempt to perform the other -normally-required actions. The only exception to this exclusion is -that, in the interactive mode, data must still be copied from the -summit system. A human-sent command must be able to change the -Scheduler priorities from the automatic to the interactive modes -\tbd{with a CLI or GUI}. An additional IPP mode is the {\em paused -mode}, intended for tests or maintenance, in which case the Scheduler -does not perform even the data copy tasks. Every task is performed on -demand by the user. - -\subsubsection{Analysis Stages} - -\paragraph{Overview} - -We now consider the collection of analysis tasks which must be +process status information, etc. \tbd{part of metadata database?}. + +\subsubsection{Controller} + +The IPP Controller must manage tasks on a cluster of up to 128 +computers. + +On startup, the IPP Controller must attempt to establish communication +with all of its computers and set their state to be {\tt alive} or +{\tt dead} based on the success of the connection. + +The IPP Controller must detect computers which crash or stop +responding. + +The IPP Controller must attempt to re-establish communication with +{\tt dead} computers. + +The IPP Controller must accept tasks from external users and systems, +which may specify a desired CPU (node) and priority in addition to the +task command. + +The IPP Controller must attempt to run pending tasks on the desired +node, if available (not {\tt dead} or {\tt off}). If the node is +unavailable, the IPP Controller must attempt to run the task on +another node. If the node is available, the IPP Controller must +attempt to run the next task when the current task is completed. + +The IPP Controller must monitor the output from the task and write it +to an associated log file. + +The IPP Controller must monitor the execution status of the task and +perform the following actions: +\begin{enumerate} +\item identify the task as successful if it has a valid exit status. +\item identify the task as unsuccessful if it has an error exit + status. +\item identify the task as unattempted if the computer crashed. +\end{enumerate} + +The IPP Controller must accept and perform the following external +commands: +\begin{enumerate} +\item add a task to the pending task list. +\item delete a specific task from the pending task list. +\item return the current status of a specific task. +\item return a list of all pending and non-pending tasks. +\item set a specified computer state to {\tt off} or {\tt dead}. +\item restrict a specified CPU to a class of tasks. +\item halt execution of a specified task. +\item set the IPP Controller state to {\tt finish}, {\tt abort}, or + {\tt stop}. +\end{enumerate} + +\subsubsection{Scheduler} + +The IPP Scheduler intiates analysis tasks which it must send to the +IPP Controller. + +All analysis tasks sent by the IPP Scheduler must include a complete +UNIX command with necessary arguments, the priority of the task, and +optionally the desired processing node. + +The IPP Scheduler must refer to several input data sources to decide +what tasks to intiate. These data sources include the IPP Metadata +Database, the Summit Metadata Database, and User requests. + +The IPP Scheduler must query the Databases on a regular basis to check +for new input information. These queries must take place at least +once every \tbr{5 seconds}. + +The IPP Scheduler must accept new User input in real-time (within 0.1 +seconds of the request). + +The IPP Scheduler must construct new tasks on the basis of the inputs +and a task dependency table. + +When the IPP Scheduler is placed in the {\em paused state}, it must +only intiate User-requested tasks. + +When the IPP Scheduler is placed in the {\em interactive state}, it +must intiate User-requested tasks as well as data transfer tasks. + +When the IPP Scheduler is placed in the {\em automatic state}, it must +intiate the most appropriate task based on the inputs. + +The IPP Scheduler must receive the exit status of tasks from the IPP +Controller. + +The IPP Scheduler must send the exit status of the analysis tasks to +the appropriate destination as defined by the task dependency table. + +\subsection{Analysis Stages} + +We now consider the requirements of the analysis tasks which must be performed by the IPP. These tasks represent the core of the required IPP functionality; the architectural components discussed above can be @@ -792,174 +712,59 @@ tasks to be executed on the appropriate data and to store the results. -Depending on the task, the basic data unit may be individual images, -collections of images, or derived data products such as a collection of -detections of astronomical objects. Because of the granularity of -these data units, many of the analysis tasks can be performed in -parallel because, for example, the intial analysis of an OTA in one -image does not depend on the results from another OTA. We define the -term `analysis stage' to refer to the largest complete analysis task -which may be performed on a single data item. The analysis stages are -divided into three categories, and further subdivided as follows: - -\begin{enumerate} - \item {\bf Science Image Analysis} is performed on the night-sky - science images to extract the science data from these images. The - science image analysis is divided into 4 phases: - - \begin{itemize} - \item {\bf Phase 1:} The image processing preparation phase, in - which basic astrometric analysis of the complete FPA image is - performed. - - \item {\bf Phase 2:} The image reduction phase, in which the - individual detector images (OTAs) are processed as much as possible - without reference to other chips in the same FPA image or other - exposures. - - \item {\bf Phase 3:} The exposure analysis phase, in which the - results of the multiple detectors are combined to improve the - calibrations for the complete FPA images. - - \item {\bf Phase 4:} The image combination phase, in which several - different exposures of the same part of the sky are combined to - produce high-quality difference and summed images. - \end{itemize} - - \item {\bf Calibration Image Analysis} is required to generate the - calibration images used in the science image analysis. There are - three types of calibration images which are produced. \tbd{make this - consistent with other sections which use the basic / other - calibration distinction} - - \begin{enumerate} - \item {\bf Calibration 1:} The basic master-detrend creation images, - which are constructed from a simple stack of multiple input - calibration images. - - \item {\bf Calibration 2:} Sky-model \& fringe-model images, which - are constructed by combining a collection of images which require - substantial processing before the combination. - - \item {\bf Calibration 3:} Flat-field correction image, which is - constructed on the basis of photometry observations of objects from - certain science images. - - \end{enumerate} - - \item {\bf Reference Catalog Creation} is required by the IPP to - generate improved astrometric and photometric reference catalogs on - the basis of Pan-STARRS observations. - -\end{enumerate} - -Figure~\ref{stages} shows the flow of data between the various IPP -software systems and the different analysis stages, each managed by -the Controller. The thick lines represent the flow of pixel data, the -thin lines represent the flow of metadata and object data, and the -grey lines represent the flow of commands. The hatched systems -represent external PanSTARRS systems (OATS, the Sky Server, the SAIC -Object Database, the Moving/Transient Object Pipeline, and other -Client Science Pipelines. - -The individual analysis stages can be accessed as a UNIX command-line -program. Each command represents the action of the stage on a single -quantum of data. These analysis stages are built of lower-level -C-functions wrapped in a higher-level programming language, -\tbd{Python}. - -The decision to execute a specific analysis stage for a specific -dataset is made by the Scheduler, which sends the infomation to the -Controller. The Controller executes the analysis stage for the data -on an appropriate machine and monitors the success or failure of the -job. - -\begin{figure} -\begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/stages.ps}} -\caption{ \label{stages} IPP System Overview} -\end{center} -\end{figure} - -\paragraph{Science Image Analysis} +\subsubsection{Science Image Analysis} The Science Image analysis stages together represent the basic data -analysis required by the IPP. These analysis stages must process the -images in a timely manner so that the incoming data stream will not -overload the Pixel Server. The required processing time is derived -from the rate at which science images are obtained by PS-1. At a -minimum, the Science Image Analysis must keep up with the average -image rate over the course of 1 day. \tbd{The Science image analysis -is required to process images at the maximum science image rate from -PS-1 of 1 image every 30 seconds -- does this fall out of the science -requirements?} \tbd{In order to give time for uncertainties in the -Pan-STARRS system as a whole, the Science Image Analysis must be able -to process all images from a night within 12 hours.} - -\tbd{number of images per night, data volume per image, output -products} - -The science image analysis which must be performed by the IPP consists -of: - -\begin{itemize} -\item detrending the images to remove the instrumental signature - -\item astrometric and photometric calibration of the individual images - -\item merging a collection of several images of the same portion of -the sky obtained over a short period of time (to remove image defects -and gaps) - -\item subtracting the appropriate reference static-sky image - -\item cleaning the image of any transients - -\item adding the cleaned image to the static sky - -\item object detection of images at specific stages -\end{itemize} - -These analysis steps can be grouped into four phases, each of which -deals with a single data unit. We identify and discuss the -requirements of the four phases below. - -\paragraph{Phase 1 : image processing preparation} - -The Phase 1 analysis stage is performed on each science FPA to -calculate basic astrometric \tbd{and photometric} data needed by the -later stages. Phase 1 must use the static (pre-determined) telescope -distortion model and table of nominal OTA positions and rotations, -combined with the guide star pixel and celestial coordinates, to -determine the correct telescope bore-sight, field rotation and -magnification. The astrometric accuracy required from this analysis -stage is \tbd{2 arcsec} across the field, sufficient to match the vast -majority of reference stars with their detections. - -In some circumstances, science images may have no guide stars. This -may occur if the detectors are not run in OTA mode, especially for -short snapshot images of if IPP is being run on non-Pan-STARRS data. -In such a circumstance, the Phase 1 stage must perform extremely basic -object detection, determining the detector coordinates for stars which -are not excessively saturated and which are significantly above the -background level. The threshold levels for this object detection -stage must be configurable. The object extraction must be performed -in less than \tbd{3 seconds}. - -In order for astrometry of an image to succeed, it is necessary that -approximate image coordinates be known. The Phase 1 analysis must be -able to succeed despite initial coordinate errors as large as \tbd{5 -times} the field width. However, the search process must attempt the -near matches first in the assumption that the given coordinates are -accurate. - -A table of the overlaps between the science image to be processed and -the static sky images must be constructed. This table will be used to -guide the processing of the static sky in Phase 4. The overlaps must -be generously calculated so that small errors in astrometry at Phase 1 -will not cause any valid static sky / science image pairs to be missed -because of the astrometric error at this phase. It is acceptable for -a small number of invalid overlaps to be identified as these will be -excluded in Phase 4. Sky cells which do not have sufficient science -image overlap \tbd{$< 10\%$} need not be processed. +analysis required by the IPP. There are several requirements which +must be met by the collection of science image analysis stages as a +group. + +The science image analysis stages must perform their analyses quickly +enoough to keep up with the incoming data stream. The required +processing time is derived from the rate at which science images are +obtained by PS-1. At a minimum, the Science Image Analysis must keep +up with the average image rate over the course of 1 day. In order to +provide a sufficient buffer for variations in the processing speed, +the Science Image Analysis must be able to process all images from a +night within 12 hours. + +The maximum latency between the aquisition of an image and the +completion of the science image analysis is set by the science +requirements of the fast transient recovery programs. The science +image analysis must process images from these observing programs +within \tbr{5 min} of their arrival time in the IPP Image Server. + +The science image analysis stages must processes up to 1000 science +images per night. + +\subsubsection{Phase 1 : image processing preparation} + +The Phase 1 analysis stage must determine the astrometric solution of +the complete camera (FPA image) with an accuracy of \tbr{1 arcsec} +peak-to-peak deviation. + +The Phase 1 analysis stage must load the guide star pixel and +celestial coordinates from the \tbd{IPP Metadata Database}\comment{or +from the image header?}. + +If guide stars are not available, the Phase 1 analysis stage must +extract bright stars from the image. This extraction must be done in +less than \tbr{1 second}. The total number of stars and size of the +bright-star aquisition box must be a user-configurable parameter. + +In order for blind astrometry of an image to succeed, it is necessary +that approximate image coordinates be known. The Phase 1 analysis +must be able to succeed despite initial coordinate errors as large as +\tbr{20\arcsec}. + +The Phase 1 analysis stage must construct a table of the overlaps +between the science image to be processed and the static sky images. + +The overlaps must overestimated by a small amount so that errors in +astrometry at Phase 1 will not cause any valid static sky / science +image pairs to be missed. The amount of overlap must be a +user-configurable parameter. + +Sky cells which do not have sufficient science image overlap \tbd{$< +5\%$} must be excluded. It is not unusual that an image be obtained with invalid coordinates @@ -967,108 +772,109 @@ may make an error and report the wrong time or coordinates. Or, the image may be obtained in exceptionally poor conditions with no -detected stars. Phase 1 must fail gracefully in these conditions, -reporting an appropriate error. Such images must be identified for -possible human intervention, or future follow-up after metadata -repairs are made. - -\paragraph{Phase 2 : image reduction} +detected stars. Phase 1 must return a descriptive error message in +these conditions. + +\subsubsection{Phase 2 : image reduction} The Phase~2 analysis is the detrend stage, in which the images from -the detector are processed to remove instrumental signatures. In -addition, basic object detection is performed along with improved -astrometric and photometric calibration. \tbd{what component selects -the appropriate calibration data? is it the phase~2 program, the -individual modules, or the scheduler above it?} In each step of the -analysis process, an image mask and noise map must be carried and -updated when appropriate. The following operations need to occur -within Phase~2 processing: - +the detector are processed to remove instrumental signatures. + +Phase 2 must perform the analysis steps only if required by the +processing recipe. The processing recipe must respect exposure time +and background flux limits to select certain stages. + +\paragraph{Detrend Image Convolutions} + +The Phase 2 analysis stage must determine the OT kernel from the IPP +Metadata Database\comment{or image header}. + +The Phase 2 analysis stage must convolve the flat-field and +high-spatial-frequency fringe images with the OT kernel. If no OT +kernel exists, this step must be silently skipped. + +\paragraph{Flag bad and saturated pixels} + +The Phase 2 analysis must load the basic bad pixel map appropriate to +the detector of interest. + +The Phase 2 analysis must use the OT kernel to grow the traps in the +raw bad pixel mag. + +The Phase 2 analysis must mask saturated pixels and a user-specified +number of surrounding pixels. + +Different bits must be set to identify different reasons for masking +the pixels. + +\paragraph{Bias correction via overscan subtraction} + +Phase 2 must be perform bias subtraction on the image. + +Phase 2 must choose the bias subtraction method and applied statistics +based on a user-configured parameter. + +The bias correction must be measured from the image overscan region. + +The overscan region must be determined from the image +header\comment{or Metadata DB}. + +The bias subtraction must apply one of the following bias corrections, +depending on the user parameters: \begin{enumerate} -\item Convolve detrend images with the OT kernel, if available -\item Flag bad and saturated pixels -\item Bias correction via overscan subtraction -\item Trim object image to remove overscan and edges corrupted by OT -\item Correct for non-linearity -\item Flat-field correction -\item Sky subtraction -\item Identify CRs -\item Find objects in the image -\item Make postage stamps of bright objects. +\item subtract a single constant from the image. + +\item subtract a 1-D bias which varies along the overscan. The function to be used must include +a spline or a chebychev polynomial derived from the data values along +the overscan, as specified by the user parameters. + +\item correct the overscan {\em and} subtract a 2-D bias image which + has been overscan corrected using one of the two methods above. \end{enumerate} -\subparagraph{Convolve detrend images with the OT kernel} - -Detrend images must be convolved by the OT kernel, so that they -accurately represent the detrend images appropriate for the object -images, which have been shifted using OT. The detrend images which -must be convolved include: the flat-field and the -high-spatial-frequency fringe images. \tbd{Must this be a formal -convolution with the analytical OT kernel, or can it be a convolution -with a decomposed kernel?} The appropriate kernel for each cell of an -OTA must be determined from the guide star history. \tbd{what is the -source of the OT kernel? pixel server?} - -\subparagraph{Flag bad and saturated pixels} - -A static bad pixel mask needs to be used to identify pixels which are -bad. Note that bad pixels which are charge traps need to be grown by -the extent of the OT convolution kernel, while those pixels above a -charge trap (i.e.\ bad colums) must not be grown, since they were not -affected by pixel shifting, but only became bad at read-out. - -Pixels saturated in the A/D converter must also be masked, and this -area must be grown by an additional pixel to mask excess charge -spillover. - -The bad pixel mask must be carried with the science images. Different -bits must be set to identify different reasons for masking the pixel. - -\subparagraph{Bias correction via overscan subtraction} - -The image bias must be subtracted. Since different detectors behave in -different ways, several options for modelling the bias must be -available. The bias must be measured from the image overscan region. -The bias subtraction method must be capable of applying a single -constant to the complete image, or to represent the bias as a function -which varies along the overscan. The function to be used must include -a spline or a chebychev polynomial derived from the data values along -the overscan. The values used to determine both the single constant -or the inputs to the spline and polynomial fits must be derived from -groups of pixels on the basis of one of several statistics, including -the sample and robust mean, median, and modes. In the case of a -single constant, all of the overscan pixel values are used in the -calculation of this statistic. In the case of the 1D functional -representation, the input values to the fit must represent the -coordinate along the overscan, with the statistic derived from the -pixels in the perpedicular direction at each location. Sigma-clipping -on the input data values must be an option. \tbd{accuracy of the bias -subtraction?} - -\subparagraph{Trim object image} - -The image must be trimmed to remove the non-imaging pixels, such as -the overscan and any pre-scan pixels, along with those pixels near the -edges that have been compromised due to OT operation. The definition -of the imaging area of the detector must be determined from the camera -configuration data or from the metadata associated with the image, -with the choice a user-configurable option. - -\subparagraph{Correct for non-linearity} - -If required, the object image (after bias correction) must be -corrected for the effects of non-linearity through a provided -polynomial fit to the pixel data values. The choice to apply the -correction must be set by the user. - -\subparagraph{Flat-field correction} +The statistic used to calculate the overscan constant or the inputs to +the spline and polynomial fits must be derived from groups of pixels +on the basis of one of several statistics, as specified by the user +parameters. The choice of statistics must include the sample and +robust mean, median, and modes. + +In the case of a single constant, all of the overscan pixel values are +used in the calculation of this statistic. In the case of the 1D +functional representation, the input values to the fit must represent +the coordinate along the overscan, with the statistic derived from the +pixels in the perpedicular direction at each location. + +If specified in the user parameters, sigma-clipping must be performed +on the input data values. + +The bias subtraction must leave no residuals greater than \tbr{1 DN} +peak-to-peak. + +\paragraph{Trim object image} + +The Phase 2 analysis must trim the non-imaging pixels from the image. + +The definition of the imaging area must be determined from the +Metadata Database\comment{or image header?}. + +Phase 2 must trim pixel near the edges that have been compromised due +to OT operation. + +\paragraph{Correct for non-linearity} + +If required, the science image must be corrected for the effects of +non-linearity. The correction must be a function of chip. + +\paragraph{Flat-field correction} The object image (after bias correction and non-linearity correction) must be corrected for sensitivity variations as a function of -position, dividing by a flat-field image. The flat-field images must -be appropriately normalized (see section \ref{mkcal}). The -flat-fielded image must have a consistent photometric zero-point -across the chip, and across the full FPA, to within 0.2\%. - -\subparagraph{Sky \& Fringe subtraction} +position, dividing by a flat-field image. + +The flat-field images must be appropriately normalized (see section +\ref{mkcal}). The flat-fielded image must have a consistent +photometric zero-point across the chip, and across the full FPA, to +within 0.2\% with peak-to-peak deviations of \tbr{0.5\%}. + +\paragraph{Sky \& Fringe subtraction} The flux contribution of the sky (from both continuum emission and the @@ -1087,5 +893,5 @@ \tbd{What is allowed power-spectrum of background variations?} -\subparagraph{Identify `cosmic rays'} +\paragraph{Identify `cosmic rays'} Charged particles in the detector frequently cause features which do @@ -1101,5 +907,5 @@ Phase~2.} -\subparagraph{Find objects in the image} +\paragraph{Find objects in the image} Objects on the flat-fielded object image must be found, and general @@ -1114,5 +920,5 @@ relevant image metadata (\ie filter, exposure time, etc). -\subparagraph{Astrometry} +\paragraph{Astrometry} Objects detected in Phase~2 must be matched with known astrometric @@ -1128,5 +934,5 @@ arcsec}. -\subparagraph{Postage Stamps} +\paragraph{Postage Stamps} The IPP must have the capability of extracting regions surrounding a @@ -1136,5 +942,5 @@ of a set of rules applied to the object magnitude and position. -\paragraph{Phase 3 : exposure analysis} +\subsubsection{Phase 3 : exposure analysis} The Phase 3 analysis stage works with the results from a complete FPA @@ -1158,5 +964,5 @@ limited by the astrometric reference catalog \tbd{30 mas for USNO?} -\paragraph{Phase 4 : image combination} +\subsubsection{Phase 4 : image combination} Phase 4 is the image combination stage, in which multiple images of @@ -1177,5 +983,5 @@ into several stages, each of which are discussed in detail below. -\subparagraph{Extract image pixels} +\paragraph{Extract image pixels} For the given sky cell, the corresponding set of image pixels must be @@ -1185,5 +991,5 @@ than 20\% more pixels than necessary from the input images. -\subparagraph{Transform pixel coordinates} +\paragraph{Transform pixel coordinates} Pixels which have been extracted from the input images must be mapped @@ -1196,5 +1002,5 @@ \tbd{interpolation method?} -\subparagraph{Flux matching} +\paragraph{Flux matching} The multiple input images must have their object fluxes intercompared @@ -1203,5 +1009,5 @@ photometrically. -\subparagraph{Image outlier pixel rejection} +\paragraph{Image outlier pixel rejection} Pixels from the group of images which are inconsistent with the @@ -1212,10 +1018,10 @@ obtained over a wide range of times. -\subparagraph{PSF matching} +\paragraph{PSF matching} The multiple input images must have their PSF mutually matched to allow for proper image subtraction. -\subparagraph{Image Subtraction} +\paragraph{Image Subtraction} The static sky image must be subtracted from the stacked, cleaned @@ -1224,5 +1030,5 @@ Object detection at this stage is the same as that used for Phase 2. -\subparagraph{Cleaned Input Image} +\paragraph{Cleaned Input Image} The flagged pixels must be excluded from the input images and a new, @@ -1230,5 +1036,5 @@ applied to it. \tbd{parameters} -\subparagraph{Update static sky} +\paragraph{Update static sky} The final, cleaned input image must be added to the static sky so that @@ -1236,5 +1042,5 @@ \tbd{parameters, weight map} -\subparagraph{Products} +\paragraph{Products} Phase 4 must produce the following data products at a minimum: @@ -1249,5 +1055,5 @@ \end{enumerate} -\subparagraph{Timing} +\paragraph{Timing} It is required that the {\em total} processing for each exposure by @@ -1264,5 +1070,5 @@ second. -\subparagraph{Accuracies} +\paragraph{Accuracies} Transformations/mappings from detector to sky must preserve both @@ -1275,5 +1081,5 @@ \end{itemize} -\subparagraph{Robustness} +\paragraph{Robustness} It is essential that the static sky image (which may have been @@ -1282,5 +1088,5 @@ to an error upstream in the processing). -\paragraph{Calibration Stages} +\subsubsection{Calibration Stages} \label{mkcal} @@ -1294,5 +1100,5 @@ below. -\paragraph{Basic Calibration Stages} +\subsubsection{Basic Calibration Stages} The IPP must generate basic calibration images using the raw bias, @@ -1308,5 +1114,5 @@ see which input images are consistent and valid. -\subparagraph{bias images} +\paragraph{bias images} Bias images may be needed to correct for structure in the bias. The @@ -1322,5 +1128,5 @@ used to exclude any significant outlier input images. -\subparagraph{dark images} +\paragraph{dark images} Dark images may be needed to correct for structure in the dark @@ -1341,5 +1147,5 @@ -- by what component?}. -\subparagraph{flat-field images} +\paragraph{flat-field images} Master flat-field images must be constructed from a collection of @@ -1356,7 +1162,7 @@ exclude any significant outlier input images. -\paragraph{Other Calibration Stages} - -\subparagraph{mask images} +\subsubsection{Other Calibration Stages} + +\paragraph{mask images} Initial bad-pixel mask images must be generated on the basis of @@ -1367,5 +1173,5 @@ inconsistent, an error must be raised. -\subparagraph{fringe frames} +\paragraph{fringe frames} Fringe-correction frames must be generated to remove the fringe @@ -1382,5 +1188,5 @@ standard combination statistics (mean, median, mode, etc). -\subparagraph{low-k sky models} +\paragraph{low-k sky models} Large-scale background structure in images which is not caused by @@ -1391,5 +1197,5 @@ telescope. \tbd{discuss principal components, SVD?} -\subparagraph{Flat-field correction frame} +\paragraph{Flat-field correction frame} Flat-field images, whether constructed from the dome, twilight, or @@ -1401,5 +1207,5 @@ sequence of images. -\subparagraph{Non-linearity correction frames} +\paragraph{Non-linearity correction frames} The IPP must have the capability of constructing non-linear correction @@ -1410,5 +1216,5 @@ from a linear detector. -\paragraph{Reference Catalog Creation} +\subsubsection{Reference Catalog Creation} For PS-1, one of the primary goals is the creation of photometric and astrometric @@ -1421,5 +1227,5 @@ list the requirements of the tools needed for this effort. -\paragraph{Astrometry Reference Creation} +\subsubsection{Astrometry Reference Creation} The existing astrometric reference catalogs are known to have @@ -1487,5 +1293,5 @@ stars rather than for the normal image data. -\paragraph{Photometry Reference Creation} +\subsubsection{Photometry Reference Creation} The IPP must provide the analysis tools needed to generate a master @@ -1545,5 +1351,5 @@ stars rather than for the normal image data. -\subsubsection{Modules} +\subsection{Modules} In order to encapsulation functionality, the analysis stages are @@ -1559,5 +1365,5 @@ Processing Pipeline Algorithm Design Document' (PSDC-430-006). -\subsubsection{PanSTARRS IPP Library} +\subsection{PanSTARRS IPP Library} In order to facilitate testing and development, and to encourage @@ -1579,19 +1385,19 @@ PSDC-430-006). -\subsubsection{Data Sources and Formats} - -\paragraph{Image Formats} +\subsection{Data Sources and Formats} + +\subsubsection{Image Formats} FITS images -\paragraph{Table Formats} +\subsubsection{Table Formats} FITS tables -\paragraph{Other Data Formats} +\subsubsection{Other Data Formats} XML files -\paragraph{External Catalogs} +\subsubsection{External Catalogs} \begin{itemize} @@ -1606,5 +1412,5 @@ \end{itemize} -\paragraph{Analysis Reference Data} +\subsubsection{Analysis Reference Data} \begin{itemize} @@ -1616,5 +1422,5 @@ \end{itemize} -\paragraph{Installation Reference Data} +\subsubsection{Installation Reference Data} \begin{itemize}