Context Navigation

← Previous Change
Next Change →

ippSRS.tex

Timestamp:

May 28, 2004, 2:56:14 PM (22 years ago)

Author:

eugene

Message:

substantial changes to SRS: split into SRS & SCD

Location:

trunk/doc/design

Files:

: 2 edited

. (modified) (1 prop)
ippSRS.tex (modified) (56 diffs)

Legend:

: Unmodified
: Added
: Removed

trunk/doc/design

Property svn:ignore

-              old
+              new
 *.out
 *.lof
+design.pdf
+specs.pdf
+ippSCD.pdf
+ippSRS.pdf
+ippSDRS.pdf
+hardware.pdf
+notes
+old
+.mana
+.status

trunk/doc/design/ippSRS.tex

-              r771
+              r810
 %%% $Id: ippSRS.tex,v 1.1 2004-05-25 00:38:56 eugene Exp $
+%%% $Id: ippSRS.tex,v 1.2 2004-05-29 00:56:14 eugene Exp $
 \documentclass[panstarrs]{panstarrs}
 …
 DR.02 & 2003.03.10 & Second draft \\ \hline
 DR.03 & 2003.04.13 & Most paragraphs fleshed out \\ \hline
+DR.04 & 2003.04.27 & Basic text frozen for internal review \\ \hline
+DR.05 & 2003.05.24 & Incorporating comments from internal review \\ \hline
 \RevisionsEnd
+\tableofcontents
+\pagebreak
 \listoffigures
 \pagebreak
-\tableofcontents
-\pagebreak
 \pagenumbering{arabic}
 …
 \subsection{Identification}
 This document establishes the system requirements for the Pan-STARRS
+This document establishes the software requirements for the Pan-STARRS
 Image Processing Pipeline (IPP) as applied to Pan-STARRS 1 (PS-1), the
 initial demonstration telescope to be constructed on Haleakala by Jan
 …
 that series is implied.
+Open Issues and TBDs in this document are marked \tbd{in bold, red
+with surrounding square brackets}.
+All timing measurements are to execution time as measured on a
+\tbd{Reference Pan-Starrs Computation Node} and assumed to be not
+limited by network bandwidth.
+\subsubsection{Definitions}
+Open issues (TBDs) in this document are marked \tbd{in bold, red with
+surrounding square brackets}.
+Quantities which should be reviewed (TBRs) are marked \tbr{in bold,
+blue with surrounding square brackets}.
+\subsubsection{Requirements Definitions}
 \paragraph{``Must''}  When used in this specification, the word
 ``must'' refers to an explicit requirement of a system component or
+the complete system.
+the complete system.  In this document, the use of the word ``must''
+replaces, and is equivalent to, use of the word ``shall'' found in
+many requirements documents.
 \paragraph{``Should''}  When used in this specification, the word
 …
 \DocumentsInternalSection
 PSDC-430-xxx  &   PS-1 Design Reference Mission \\ \hline
+PSDC-130-001  &   PS-1 Design Reference Mission \\ \hline
 PSDC-430-004  &   Pan-STARRS IPP C Code Conventions \\ \hline
 PSDC-430-006  &   Pan-STARRS IPP ADD \\ \hline
 PSDC-430-007  &   Pan-STARRS IPP PSLib SDR \\ \hline
+PSDC-430-007  &   Pan-STARRS IPP PSLib SDRS \\ \hline
 \DocumentsExternalSection
 Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\
 …
 \section{Requirements}
+\subsection{Required States}
+The IPP must have 3 states: active, paused, and interactive.
+\subsubsection{Active State}
+\label{req:active-state}
+In active state, the IPP must accept images and metadata from OATS and
+automatically perform the complete set of image processing tasks,
+including both calibration and science image processing.  The IPP must
+respond to requests for data from the client science pipelines
+\tbd{and IPP monitoring team}.
+\subsubsection{Paused State}
+\label{req:paused-state}
+In paused state, the IPP must refuse data and metadata from OATS and
+data requests from the client science pipelines.
+\subsubsection{Interactive State}
+\label{req:interactive-state}
+In interactive state, the IPP must accept data and metadata from OATS,
+but must not automatically process the data.  The IPP must respond to
+user commands to initiate portions of the data analysis.
+\subsection{System Capability Requirements}
+\subsection{Science Requirements}
 \label{req:system-capabilities}
+\tbd{distinguish data products in commissioning, during PA survey,
+after PA survey}
 The IPP must perform the following tasks:
 …
 \end{enumerate}
+\subsubsection{Software Coding Requirements}
+\paragraph{Languages}
+\subsection{Required States}
+The IPP must have 3 states: active, paused, and interactive.
+\subsubsection{Active State}
+\label{req:active-state}
+In active state, the IPP must accept images and metadata from the
+external sources (i.e., the summit) and automatically perform the
+complete set of image processing tasks, including both calibration and
+science image processing.  The IPP must respond to requests for data
+from client science pipelines.
+\subsubsection{Paused State}
+\label{req:paused-state}
+In paused state, the IPP must refuse incoming data and metadata and
+data requests from the client science pipelines.
+\subsubsection{Interactive State}
+\label{req:interactive-state}
+In interactive state, the IPP must accept imcoming data and metadata,
+but must not automatically process the data.  The IPP must respond to
+user commands to initiate portions of the data analysis.
+\subsection{Software Coding Requirements}
+\subsubsection{Languages}
 \label{req:languages}
 …
 Scripting language must be \tbd{Python, version TBD}.
+\paragraph{Interfaces}
+\label{req:interfaces}
+Access to low-level Library functions must be provided via C APIs
+consisting of the function calls and the defined data structures and
+other data types.  Access to high-level functions must be provided
+via C APIs as well as SWIG interfaces, where specified.  Access to
+processing jobs must be available via the UNIX shell.
+\paragraph{Coding Standards}
+The C code must comply with ANSI Standard C99.  Because the delivered
+code is required to run on UNIX machines, the delivered code must be
+in compliance with the language-independent UNIX operating system
+standard POSIX (Open Group Based Specifications Issue 6, IEEE Std
+.1, 2003).  Source code files must use the UNIX line-break
+convention (line-feed only).  C coding style must adhere to the
+standard defined in the document 'Pan-STARRS C-coding standard'
+(PSDC-430-004).  \tbd{Python coding must follow the Python standard
+defined in the document TBD}.
+\paragraph{Naming Conventions}
+\subsubsection{Interfaces}
+We require the following types of interfaces:
+\begin{enumerate}
+\item Access to low-level Library functions must be provided via C
+APIs consisting of the function calls and the defined data structures
+and other data types.
+\item Access to high-level functions must be provided via C APIs as
+well as SWIG interfaces, where specified.
+\item Access to processing jobs must be available via the UNIX shell.
+\end{enumerate}
+\subsubsection{Coding Standards}
+\begin{enumerate}
+\item The C code must comply with ANSI Standard C99.
+\item Because the delivered code is required to run on UNIX machines,
+the delivered code must be in compliance with the language-independent
+UNIX operating system standard POSIX (Open Group Based Specifications
+Issue 6, IEEE Std 1003.1, 2003).
+\item Source code files must use the UNIX line-break
+convention (line-feed only).
+\item C coding style must adhere to the standard defined in the
+document 'Pan-STARRS C-coding standard' (PSDC-430-004).
+\item \tbd{Python} coding must follow the standard defined in the
+document \tbd{TBD}.
+\end{enumerate}
+\subsubsection{Naming Conventions}
 Header files must have names starting \code{ps} or \code{p_ps} for
 …
 for the public header files.
 Functions visible at global scope which are part of the public API
+must have names begining with \code{ps}, and follow the naming
+conventions in the coding standard.  Functions that are visible at
+global scope but which are not part of the public interface must have
+names begining with \code{p_ps}.  Functions that are local to a file
 must \textit{not} start \code{ps} (or \code{p_ps}).
+Functions visible at global scope that are part of the public API must
+have names begining with \code{ps} and follow the naming conventions
+in the coding standard.  Functions visible at global scope but which
+are not part of the public interface must have names begining with
+\code{p_ps}.  Functions that are local to a file must \textit{not}
+start \code{ps} (or \code{p_ps}).
 Variables visible at global scope which are part of the public API
 …
 \code{psEquatorial2Ecliptic}).
 \paragraph{C Programming Guidelines}
+\subsubsection{C Programming Guidelines}
 Functions that assign to a variable must list that argument
 …
 \end{itemize}
 \paragraph{Commenting and Documentation}
+\subsubsection{Commenting and Documentation}
 Commenting of delivered C and Python code must follow the C and
 …
 documentation must be delivered as PDF documents.
 \paragraph{Version Control}
+\subsubsection{Version Control}
 Source code version control must be implemented with CVS.
 \paragraph{CSCI Deliverable}
+\subsubsection{CSCI Deliverable}
 All final source code generated for the IPP is to be delivered via
 …
 and made available via CVS.
 \paragraph{Platform architectures and operating systems}
+\subsubsection{Platform architectures and operating systems}
 Makefiles must be provided with appropriate flags set so that all
 …
 x86/Linux combination.
+\paragraph{Software Configuration}
+All timing measurements are to execution time as measured on a
+\tbd{Reference Pan-Starrs Computation Node} and assumed to be not
+limited by network bandwidth.
+\subsubsection{Software Configuration}
 \tbd{deferred}
+\subsubsection{Architectural Components}
+In order to achieve the required functionality, it is necessary to
+divide the IPP into a number of clearly-defined software elements,
+listed as follows:
+\subsection{Architectural Components}
+As discussed in the Pan-STARRS System Concept Definition, the IPP is
+organized into a number of clearly-defined software elements.  The SCD
+provides a detailed description of the roles and responsibilities of
+these subsystems.  In brief, the IPP consists of a collection of
+science analysis stages, a set of architectural components which
+provide the infrastructure needed to run the analysis programs, and a
+collection of hardware on which all of the software elements exist.
+The architectural components consist of:
 \begin{enumerate}
 \item {\bf Pixel Server:} This component is a large data store for all
+\item {\bf Image Server:} This component is a large data store for all
  images used by the IPP, including the raw images from the telescope,
  the master calibration images, the reference static-sky images, and
  any temporary image data products produced by the IPP.  The Pixel
+ any temporary image data products produced by the IPP.  The Image
  Server is required to meet all of the image storage needs identified
  in the top-level requirements above.  The Pixel Server must accept
+ in the top-level requirements above.  The Image Server must accept
  the incoming data and store it until it is no longer needed by other
  portions of the IPP.
 …
   as needed to perform the analysis specified above.
-\item {\bf Analysis Stages:} Specific programs are required to perform
-  the processing steps listed above.  These can be divided into
-  well-defined analysis stages, each of which operates on a particular
-  unit of data, such as a single OTA image or a collection of
-  astronomical objets.
 \item {\bf Controller:} In order to perform the analysis stages
   required by the IPP, it is necessary to use distributed computing
 …
 \begin{figure}
 \begin{center}
 \resizebox{8cm}{!}{\includegraphics{pics/overview.ps}}
+\resizebox{8cm}{!}{\includegraphics{pics/overview}}
 \caption{ \label{overview} IPP System Overview}
 \end{center}
 …
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\paragraph{Pixel Server}
+The IPP Pixel Server \tbd{rename as Image Server?} is a large data
+store for all images used by the IPP.  The Pixel Server is required to
+store all of the images needed by the IPP for the length of time they
+are required; total data volume is specified in detail in the hardware
+summary, but is in the vicinity of \tbd{700 TB}.
+The IPP Pixel Server must maintain a record of all images currently
+available in the repository \tbd{and all no longer available}.  This
+record must include the image name, location (which machine), the
+state of the image (available, deleted), the image size, the image
+type, and the existence and location of secondary copies of the image.
+This information need not include other metadata such as the image
+summary statistics or the state of the image processing for the image,
+as these aspects are included in the Metadata DB.
+The IPP Pixel Server must store images as FITS files on disk.  Raw
+images from the telescope must be stored as individual OTA images for
+each file, with multiple Cell images per file as well as video
+sequences from the guide stars.  Images of the Static Sky must be
+stored in the form of \tbd{triangular segments} to minimize the total
+data volume and pixel overlap.
+The IPP Pixel Server must distribute images across a cluster of
+machines.  The IPP Pixel Server must be capable of honoring requests
+to store an image on a specific machine.  If such a request cannot be
+honored, the IPP Pixel Server must select an appropriate machine and
+notify the requesting agent of the new locations.  The IPP Pixel
+Server must provide a mechanism to maintain multiple (at least two)
+copies of each image.
+The IPP Pixel Server must interface with other subsystems of the IPP.
+It must provide an interface to other IPP subsystems to identify the
+image location (the computer on which it resides).  It must provide a
+mechanism to serve a specified image to another IPP or Pan-STARRS
+subsystem.  It must provide a mechanism for deletion of images in the
+Pixel Server.  It must have a mechanism to accept or retrieve an image
+from another Pan-STARRS subsystem, in particular OATS.  Communication
+of messages between the IPP Pixel Server and other subsystem must be
+via \tbd{XML messages} passed via \tbd{some transport}.
+The IPP Pixel Server must accept images at the telescope maximum rate
+of 1 full-camera image every 30 seconds.  The IPP Pixel Server must
+therefore accept notifications and process retrievals at a rate of 64
+raw OTAs in 30 seconds.
+\tbd{O/S, language, SQL, ODBC requirements?}
+\tbd{hardware requirements?}
+\tbd{communication protocols?}
+\paragraph{P\&A Database}
+The IPP requires a mechanism to store data related to astronomical
+objects derived from various sources with a variety of associations.
+The PnA (Photometry and Astrometry) Database serves this function.
+The PnA Database deals with two related concepts: {\em objects} and
+{\em detections}.  The objects are descriptions of astronomical
+objects while the detections are the specific measurements of those
+objects on an image.  A collection of {\em detections} may be used to
+derive average quantities which describe a particular {\em object}.
+The PnA Database must store the collections of detections which were
+derived from specific images from any of the analysis stages.  It must
+be possible to determine and locate (perhaps via interactions with the
+pixel server) the image from which a specific detection was derived.
+It must also be possible to extract all detections derived from a
+specific image.  These associations must include descriptive
+information including the coordinates of the detection on the image.
+The PnA Database must provide a mechanism to associate together
+multiple detections of a specific object.  Several major classes of
+objects will be present, each of which must be handled correctly.
+First, the most distant stars, compact galaxies, and QSOs will have
+nearly fixed locations relative to other nearby stars, with only small
+deviations for individual measurements.  The association between
+multiple detections of such objects must be made on the basis of their
+coincident positions.  The PnA Database must be able to determine the
+average position of the object and the deviations of the individual
+detections from that average.
+Second, solar system objects do not have a fixed location and
+detections of such objects must associated on the basis of their
+coincidence with the orbit of the objects.  The PnA Database must be
+able to associate detections with the orbits of known objects.  The
+determination of this association is the responsibility of the MOPS
+and must be communicated to the IPP PnA Database on \tbd{some
+timescale}.  The PnD Database must be able to retrieve the detections
+associated with the object and to provide the object associated with
+the specific detections.  This association must include descriptive
+information such as the offset of the detection from the predicted
+location of the detection based on the orbit.  This functionality is
+required to allow the PnA Database to ignore known moving object
+detections from other types of queries.
+Third, stars in the general vicinity of the solar system fall in
+between these first two classes of objects.  Their proper motion and
+parallax response is significant enough ($>1$ arcsec in 10 years) that
+they are not well-described by an average location and a collection of
+offsets.  These objects must be described by a distance and a proper
+motion vector.  The PnA Database must be able to find and associate
+detections of objects for which either of the parallax or the proper
+motion are substantial.
+Fourth, many detections, especially in their initial states, will not
+be associated with a specific astronomical object of any of the above
+classes and should be treated as orphans.  Some of these will be
+suprious (not represent real objects), some will be from solar system
+objects for which orbits are not yet determined, some will be from
+faint stars near the detection limits, some will be from short-term
+transients which have only been detected once.  The PnA Database must
+be able to carry these detections until they have been associated with
+one of the objects above.  It must be possible to migrate individual
+detections associated with an astronomical object back to the orphan
+state.
+For every object, and all orphaned detections, it must be possible to
+determine the images for which the coordinates were included but for
+which no detection was made.  The minimum set of information which
+must be carried for these non-detections is the image and the
+associated object or orphan.
+The PnA Database must store the relationships between various
+photometric systems and, in some cases, the evolution of that
+relationship.  It must be possible, given a determined set of
+calibrations, to convert between the measured instrumental magnitude
+of a detection with a specific filter, detector, and telescope, and at
+particular time and the implied magnitude in the average Pan-STARRS
+magnitude systems.  It must also be possible, given the magnitudes of
+an object in one system to convert those to the magnitudes in another
+system; an example of such a conversion is between the average
+Pan-STARRS filter systems and the various reference systems
+appropriate for those filters.
+The PnA Database must provide interfaces to extract lists of objects
+and detections based on various query parameters.  It must be possible
+to extract all detections associated with a specific object, all
+non-detections of that object (or orphan) and summary statistics from
+these collections.  It must be possible to extract all objects or
+detections within specified spatial regions including regions bounded
+by great circles (RA,DEC; GLAT,GLON; ELAT,ELON) and regions described
+by a location and a search radius.  It must be possible to extract the
+image parameters associated with a specific detection including image
+coordinates of the detection, exposure time, time and date of the
+detection, etc.
+\tbd{volume requirements}
+\tbd{speed / access requirements}
+\paragraph{Metadata Database}
+\subsubsection{Image Server}
+The IPP Image Server must store images on a distributed collection of
+computer disks.  Individual instinces of a file are only required to
+be stored on a single machine (striping across computers is not a
+requirement).
+The IPP Image Server must be capable of honoring requests to store an
+image on a specific machine.  If such a request cannot be honored (ie,
+the machine is down), the IPP Image Server must select an appropriate
+machine and notify the requesting agent of the new locations.
+The IPP Image Server be able to maintain multiple copies of each
+image, as specified by the user.
+The IPP Image Server must maintain a record of all image copies
+currently available in the repository.  This record must include the
+image name, location (which machine), the image size, and the state of
+the image.
+The IPP Image Server must lock images in the repository on request.
+Both read (shared) and write (exclusive) locks must be provided.  A
+read lock must prevent write access to the file; a write lock must
+prevent both read and write access.
+The IPP Image Server must return the image location (the computer on
+which it resides) upon request.
+The IPP Image Server must return a specified image upon request.
+The IPP Image Server must delete images in the repository on request.
+The IPP Image Server must accept images from the summit at the maximum
+rate of 1 full-camera image every 30 seconds.  The IPP Image Server
+must therefore accept new images into the repository at a rate of 64
+raw OTAs in 30 seconds and a total input data volume rate of 75
+MB/sec.
+\subsubsection{PA Database}
+\begin{table}
+\begin{center}
+\caption{PA Detection Classes \& Object Parameters\label{PAdetections}}
+\begin{tabular}{lrrrr}
+\hline
+\hline
+Object Parameter & P2 & P4S & P4D & SS \\
+\hline
+PSF x,y, M, $\sigma_{\rm M}$                & + & + & + & + \\
+$\sigma_x$, $\sigma_y$, covar.              & + & + & + & + \\
+exp. spaced aps., Poisson noise, variance   & - & - & - & + \\
+streak L, $\phi$, $\sigma_L$, $\sigma_\phi$ & - & - & + & + \\
+$x_g$, $y_g$, flag                          & + & + & - & + \\
+local sky data                              & + & + & + & + \\
+Petrosian R, M, $R_{50}$, $R_{90}$          & - & + & - & + \\
+S\'ersic R, M, AB, $\phi$, $\nu$            & - & + & - & + \\
+W.L. $\gamma_1$, $\gamma_2$, pol. terms     & - & - & - & + \\
+star/gal sep, star/streak sep.              & - & + & + & + \\
+\hline
+deVeucaleur R, M, AB, $\phi$                & - & + & - & + \\
+exponential R, M, AB, $\phi$                & - & + & - & + \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+The PA Database must accept and store individual detections and
+collections of detections along with information about the image which
+provided the detections.
+Detections must be saved as one of several detection classes (P2, P4S,
+P4D, SS) and the PA Database must store the appropriate parameters,
+listed in Table~\ref{PAdetections}, for each class.
+The PA Database must identify the image which provided the detection,
+or in the case of external references, an identifier specific to the
+reference source.
+The PA Database must group detections into objects and measure average
+parameters of those objects.
+The PA Database must store parallax and proper motion parameters for a
+subset of the average objects.
+The PA Database must store image and filter calibration information
+necessary to convert between instrumental magnitudes and calibrated
+magnitudes in standard systems.
+The PA Database must perform at least the follow queries, with
+constraints on the output based on at least time ranges, magnitude
+limits, error limits:
+\begin{enumerate}
+\item given (RA,DEC) and a Radius, return all objects and/or
+detections in the region.
+\item given (RA,DEC)_0 - (RA,DEC)_1, return all objects and/or
+  detections in the region.
+\item given (RA,DEC), return closest object.
+\item given object ID, return all detections
+\item given detection, return source image data.
+\item given (RA,DEC), return all images overlapping coordinate.
+\item given (RA,DEC) and a Radius, return all images overlapping region.
+\item given (RA,DEC)_0 - (RA,DEC)_1, return all images overlapping
+  region.
+\item given detection instrumental magnitude, return derived
+  magnitudes based on calibration information.
+\item given a collection of detections, determine the object avergae
+  magnitude.
+\item given a collection of objects and detections, determine the
+  individual image zero-points.
+\item given a region, return all possible combinations of the object
+  or detection magnitudes (M1 - M2).
+\item given a list of (RA,DEC) entries, return all nearest objects.
+\item given a filter, telescope, or detector, return all calibration
+  terms and history.
+\item given a detection, return all non-detections from images which
+  overlapped the detection coordinates.
+\end{enumerate}
+The PA Database must accept detection IDs of moving objects and label
+the detections with the identified object.
+\begin{table}
+\begin{center}
+\caption{PA Detection Classes \& Object Parameters\label{PAdetections}}
+\begin{tabular}{lrrrr}
+\hline
+\hline
+Quantity & P2 & P4$\Sigma$ & P4$\Delta$ & SS \\
+\hline
+detection limit             & $20 \sigma$       & $5 \sigma$      & $3 \sigma$      & \\
+depth (r')                  & 20.8              & 23.0            &                 & \\
+stars deg$^{-2}$ ($|b|>10$) &   $1 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ \\
+stars FPA$^{-1}$ ($|b|>10$) &   $7 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ \\
+stars sec$^{-1}$ ($|b|>10$) & $2.3 \times 10^4$ & $1 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ \\
+bytes star$^{-1}$           & 64                & 100             & 64              &                 \\
+MB sec$^{-1}$               & 1.4               & 2.2             & 0.7             &                 \\
+PS-1 total TB               & 8                 & 12              & 4               &                 \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+The PA Database must accept new detections at the rate generated by
+the telescope from the Phase 2 and Phase 4 analysis.  Except within 10
+degrees of the galactic plane, the PA Database must keep up with the
+incoming rates.  The expected rates are listed in Table~\ref{PArates},
+along with the total data volume required for storage space over the
+PS-1 lifetime.  The PA Database must be able to keep up with these
+rates.
+\subsubsection{Metadata Database}
+\tbd{this section needs to be reviewed and revised}
 The IPP requires a Metadata Database to store and provide access to
 …
 avoid slowing down the analysis systems.
+\tbd{need to extract specific requirements from this}
 \tbd{volume requirements}
-\tbd{does the description of images belong in the Metadata database or
-  in the Pixel / Image Server?}
 \tbd{queries}
+\subparagraph{Configuration Database -- a subset of the metadata database?}
+{\bf note: description of images belong in the Metadata database,
+  location of images is in the Image server}
+\paragraph{Configuration Database}
 The IPP requires a Configuration Database to store and provide access to
 …
 configuration database include the default parameters for the various
 analysis programs, the description of the computing environment, the
+process status information, etc.
+\paragraph{Controller}
+The IPP uses a collection of computers to store and process images and
+to manipulate collections of detections.  These computers perform any
+of a large number of analysis stages or other processing tasks without
+significant interprocess communication.  It is necessary to have a
+mechanism which initiates computing tasks on the different computers,
+which monitors the tasks as they are executed, which handles the
+output and the errors from these tasks, and which reacts to the
+failure of any of the computing nodes.  The system responsible for the
+tasks in the IPP is the Controller.
+The Controller must interact with the collection of computers under
+its management and with other subsystems in the IPP.  The controller
+must accept a variety of inputs from other subsystems, described
+below, and respond accordingly.  The controller must also provide
+information to other subsystems on demand.
+Computers managed by the controller are allowed to be in one of
+several states, and the controller must interact with it in an
+appropriate way for each of those states.  A computer may be {\tt
+alive}, {\tt dead} or {\tt off}.  If the computer is {\tt alive}, it
+responds to commands from the controller and may be used for tasks
+subject to other constraints.  If it is {\tt dead}, the computer is
+not responsive and must not be used for executing tasks.  The
+controller must identify computers which have died and occasionally
+test them to see if they are {\tt alive} again.  Computers which are
+{\tt off} are not available for tests and must not be tested.
+Computers may be set to the {\tt off} or {\tt dead} states by external
+subsystems; it is the responsibility of the Controller to return a
+computer to the {\tt alive} state if possible.
+Computers which are in the {\tt alive} state may be in one of two
+modes: {\tt busy} and {\tt free}.  A computer which is {\tt busy}
+currently has a task assigned to it.  The controller may only assign
+one task to one computer at a time\footnote{A physical piece of
+hardware may be defined to the Controller as multiple computers to
+allow multi-processor nodes to execute more than one simultaneous
+task.}.  Computers which are in the {\tt free} state may have tasks
+assigned to it.  The controller must also manage an additional set of
+constraint tables for each machine: the allowed tasks.  Each computer
+may have a list of allowed tasks which may include {\tt all} tasks,
+{\tt none} of the tasks, or specified task names.  The controller must
+only execute the allowed tasks on a machine.
+The Controller must accept tasks from other IPP subsystems.  The task
+requests must include the specific command to be executed.  The
+commands must be in the form of a UNIX command which could be
+performed on any of the computing nodes.  Any input or output data
+structures in the commands must be a valid resource regardless of the
+node on which the task is executed.  Input and output data resources
+must be unique where necessary to avoid conflicts.  Tasks must be
+given an identifier, which must be returned to the requesting agent,
+to be used to control the specific task.
+Task requests may specify a desired node for the task execution.  The
+Controller must attempt to honor the request if the node is {\tt
+alive}, but must execute on another node if the requested one is {\tt
+dead} or {\tt off}.  Even if a node is {\tt alive} the controller must
+choose another node if the specified tasks is not allowed on the
+requested node.  In all other cases, the controller must wait until
+executing processes, and processes with higher priority, are completed
+before executing the specified task on the requested node.
+Task requests may specify an urgency level.  The controller determines
+the priority of the task by sorting first by priority and next by the
+sequence of the request.  An executing task must be completed before
+any new task is started, regardless of priority.  Tasks may be
+assigned a priority of 0 in which case they are maintained in the
+queue and never executed.
+The controller must monitor the output streams from the executing
+tasks and the exit status of the tasks.  \tbd{where do we send the
+output logs?}.  The status, including the exit status, of each task
+must be maintained for other subsystems to query as needed.  \tbd{how
+long?  on disk / database?}
+The controller must accept commands from other IPP subsystems.  These
+commands include those which govern the processing of specified tasks,
+those which govern the behavior of specific computing nodes, and those
+which request information from the controller.  The controller must be
+able to halt the execution of a specified task, delete an unexecuted
+task from the task list, change the priority of tasks, change the
+requested nodes for tasks.  The controller must also be able to stop
+the current execution of a task and push it to the end of the queue
+and also change its priority.
+The controller must honor requests (normally from the users) to change
+the mode of any computing node on demand between {\tt off} and {\tt
+dead}.  It must also be able to change the list of allowed tasks as
+requested by external commands.
+The controller must respond to informational requests regarding the
+collection of machines and their states as well as the collection of
+tasks and their states.  The controller must monitor the execution
+times of the different tasks and provide summary statistics.  Finally,
+the controller must respond to three top-level commands: {\tt finish},
+{\tt stop} and {\tt abort}.  When {\tt finish} is requested, no more
+new tasks are accepted on the stack of task, and when all tasks in the
+stack have completed, the controller must exit.  When {\tt stop} is
+requested, the currently executing tasks must be completed at which
+point the controller must exit, but tasks remaining in the stack which
+have not been started are flushed.  When {\tt abort} is issued, the
+controller immediately kills all executing tasks and exits.
+\paragraph{Scheduler}
+The IPP is responsible for a variety of analysis tasks: several stages
+of processing of the science images; routine assessment of the detrend
+images used in processing the science images; construction of
+replacement detrend images when needed; generation of astrometric and
+photometric reference catalogs based on the collected dataset; and the
+performance of test analysis programs.  At any point, decisions need
+to be made about which of these tasks should be performed, based on an
+analysis of the contents of the image database tables, the
+requirements of the people monitoring the IPP, and the near-term
+observing plans.  The IPP Scheduler is a mechanism to manage these
+various inputs to guide the decisions and initiate the actions.
+The Scheduler acts as an intermediate between several components of
+the IPP and also between the IPP and external agents such as the OATS
+system and the users who must monitor the behavior of the IPP.
+The Scheduler must send commands to the Controller for execution.  It
+is the Controller's responsibility to manage the specific analysis
+jobs executing on a given processing node.  These analyses may include
+the process of copying of moving data from OATS to the pixel server
+nodes, or it may involve image processing stages performed on the
+science images by the appropriate processing nodes, or it may involve
+analysis of the data in the PnA object database.  In order to isolate
+and encapsulate the responsibilities of the Scheduler and the
+Controller, the Scheduler must initiate the tasks which the controller
+manages; in this way, the controller does not need to have any
+information about the details of the tasks which it executes.
+Communication between the Scheduler and the Controller must be
+bi-directional; the Scheduler must send tasks to the Controller which
+the Controller must inform the Scheduler of the outcome of those
+tasks.  \tbd{it is not specified whether the scheduler and controller
+are components of a single software system or interacting but distinct
+software components.}
+The Scheduler must take as input the current list of pending images,
+both science and calibration, and a description of the current
+observing plan or strategy on some time-scale.  The Scheduler must
+also take input from humans who manage the IPP.
+The Scheduler must choose between several types of analysis stages
+based on the contents of those lists and on the requirements of the
+users.  The list of tasks which the Scheduler must decide between
+includes:
+\begin{itemize}
+\item moving data to and from the pixel server ($\sim 30$ second timescales)
+\item running the science analysis stages ($\sim 30$ second timescales)
+\item testing the validity of the current detrend images ($\sim$
+  nightly)
+\item constructing new detrend images ($\sim$ weekly)
+\item updating and improving the photometric and astrometric reference
+  catalogs ($\sim$ yearly).
+\end{itemize}
+The Scheduler must choose between tasks which are relevant on several
+different time-scales.  The time-scale range from 2 times per minute
+to once or twice a year, as noted in the list above.  The Scheduler
+must also make use of the human input in managing such choices.  The
+human users must be able to specify that a particular task or set of
+tasks is of higher or lower priority than the norm.
+The Scheduler must maintain a set of rules defining the dependency of
+one type of analysis stage on other analysis products.  For example,
+the nightly science image processing depends on the existence of valid
+detrend images.  The Scheduler must be able to recognize the
+dependency and initiate the required analysis needed to perform other
+analysis tasks.  The Scheduler must have the ability to decide between
+postponing an analysis task until the required data are available or
+to initiate the task using a lower-quality or less appropriate
+substitute.  For example, in normal circumstances, a science image
+must not be processed until the corresponding detrend frame has been
+produced.  However, if such a frame is unlikely to appear soon, and
+the pressure to process the science image is sufficiently high, then
+the frame could be processed with an older detrend frame of known
+lower quality.  The Scheduler must have the ability to choose the
+best, if not ideal, reference data for a particular circumstance.
+The Scheduler is responsible for setting the operating mode of the
+IPP.  When the IPP is in the automatic operating mode, this implies
+that the Scheduler is performing the most appropriate tasks at a
+particular time.  When the IPP is in the interactive mode, the
+Scheduler must perform the requested action regardless of the outcome
+of the decision trees.  In addition, the Scheduler must only perform
+the requested actions and not attempt to perform the other
+normally-required actions.  The only exception to this exclusion is
+that, in the interactive mode, data must still be copied from the
+summit system.  A human-sent command must be able to change the
+Scheduler priorities from the automatic to the interactive modes
+\tbd{with a CLI or GUI}.  An additional IPP mode is the {\em paused
+mode}, intended for tests or maintenance, in which case the Scheduler
+does not perform even the data copy tasks.  Every task is performed on
+demand by the user.
+\subsubsection{Analysis Stages}
+\paragraph{Overview}
+We now consider the collection of analysis tasks which must be
+process status information, etc.  \tbd{part of metadata database?}.
+\subsubsection{Controller}
+The IPP Controller must manage tasks on a cluster of up to 128
+computers.
+On startup, the IPP Controller must attempt to establish communication
+with all of its computers and set their state to be {\tt alive} or
+{\tt dead} based on the success of the connection.
+The IPP Controller must detect computers which crash or stop
+responding.
+The IPP Controller must attempt to re-establish communication with
+{\tt dead} computers.
+The IPP Controller must accept tasks from external users and systems,
+which may specify a desired CPU (node) and priority in addition to the
+task command.
+The IPP Controller must attempt to run pending tasks on the desired
+node, if available (not {\tt dead} or {\tt off}).  If the node is
+unavailable, the IPP Controller must attempt to run the task on
+another node.  If the node is available, the IPP Controller must
+attempt to run the next task when the current task is completed.
+The IPP Controller must monitor the output from the task and write it
+to an associated log file.
+The IPP Controller must monitor the execution status of the task and
+perform the following actions:
+\begin{enumerate}
+\item identify the task as successful if it has a valid exit status.
+\item identify the task as unsuccessful if it has an error exit
+  status.
+\item identify the task as unattempted if the computer crashed.
+\end{enumerate}
+The IPP Controller must accept and perform the following external
+commands:
+\begin{enumerate}
+\item add a task to the pending task list.
+\item delete a specific task from the pending task list.
+\item return the current status of a specific task.
+\item return a list of all pending and non-pending tasks.
+\item set a specified computer state to {\tt off} or {\tt dead}.
+\item restrict a specified CPU to a class of tasks.
+\item halt execution of a specified task.
+\item set the IPP Controller state to {\tt finish}, {\tt abort}, or
+  {\tt stop}.
+\end{enumerate}
+\subsubsection{Scheduler}
+The IPP Scheduler intiates analysis tasks which it must send to the
+IPP Controller.
+All analysis tasks sent by the IPP Scheduler must include a complete
+UNIX command with necessary arguments, the priority of the task, and
+optionally the desired processing node.
+The IPP Scheduler must refer to several input data sources to decide
+what tasks to intiate.  These data sources include the IPP Metadata
+Database, the Summit Metadata Database, and User requests.
+The IPP Scheduler must query the Databases on a regular basis to check
+for new input information.  These queries must take place at least
+once every \tbr{5 seconds}.
+The IPP Scheduler must accept new User input in real-time (within 0.1
+seconds of the request).
+The IPP Scheduler must construct new tasks on the basis of the inputs
+and a task dependency table.
+When the IPP Scheduler is placed in the {\em paused state}, it must
+only intiate User-requested tasks.
+When the IPP Scheduler is placed in the {\em interactive state}, it
+must intiate User-requested tasks as well as data transfer tasks.
+When the IPP Scheduler is placed in the {\em automatic state}, it must
+intiate the most appropriate task based on the inputs.
+The IPP Scheduler must receive the exit status of tasks from the IPP
+Controller.
+The IPP Scheduler must send the exit status of the analysis tasks to
+the appropriate destination as defined by the task dependency table.
+\subsection{Analysis Stages}
+We now consider the requirements of the analysis tasks which must be
 performed by the IPP.  These tasks represent the core of the required
 IPP functionality; the architectural components discussed above can be
 …
 tasks to be executed on the appropriate data and to store the results.
+Depending on the task, the basic data unit may be individual images,
+collections of images, or derived data products such as a collection of
+detections of astronomical objects.  Because of the granularity of
+these data units, many of the analysis tasks can be performed in
+parallel because, for example, the intial analysis of an OTA in one
+image does not depend on the results from another OTA.  We define the
+term `analysis stage' to refer to the largest complete analysis task
+which may be performed on a single data item.  The analysis stages are
+divided into three categories, and further subdivided as follows:
+\begin{enumerate}
+ \item {\bf Science Image Analysis} is performed on the night-sky
+ science images to extract the science data from these images.  The
+ science image analysis is divided into 4 phases:
+ \begin{itemize}
+  \item {\bf Phase 1:} The image processing preparation phase, in
+  which basic astrometric analysis of the complete FPA image is
+  performed.
+  \item {\bf Phase 2:} The image reduction phase, in which the
+  individual detector images (OTAs) are processed as much as possible
+  without reference to other chips in the same FPA image or other
+  exposures.
+  \item {\bf Phase 3:} The exposure analysis phase, in which the
+  results of the multiple detectors are combined to improve the
+  calibrations for the complete FPA images.
+  \item {\bf Phase 4:} The image combination phase, in which several
+  different exposures of the same part of the sky are combined to
+  produce high-quality difference and summed images.
+ \end{itemize}
+ \item {\bf Calibration Image Analysis} is required to generate the
+ calibration images used in the science image analysis.  There are
+ three types of calibration images which are produced. \tbd{make this
+ consistent with other sections which use the basic / other
+ calibration distinction}
+ \begin{enumerate}
+  \item {\bf Calibration 1:} The basic master-detrend creation images,
+  which are constructed from a simple stack of multiple input
+  calibration images.
+  \item {\bf Calibration 2:} Sky-model \& fringe-model images, which
+  are constructed by combining a collection of images which require
+  substantial processing before the combination.
+  \item {\bf Calibration 3:} Flat-field correction image, which is
+  constructed on the basis of photometry observations of objects from
+  certain science images.
+ \end{enumerate}
+ \item {\bf Reference Catalog Creation} is required by the IPP to
+ generate improved astrometric and photometric reference catalogs on
+ the basis of Pan-STARRS observations.
+\end{enumerate}
+Figure~\ref{stages} shows the flow of data between the various IPP
+software systems and the different analysis stages, each managed by
+the Controller.  The thick lines represent the flow of pixel data, the
+thin lines represent the flow of metadata and object data, and the
+grey lines represent the flow of commands.  The hatched systems
+represent external PanSTARRS systems (OATS, the Sky Server, the SAIC
+Object Database, the Moving/Transient Object Pipeline, and other
+Client Science Pipelines.
+The individual analysis stages can be accessed as a UNIX command-line
+program.  Each command represents the action of the stage on a single
+quantum of data.  These analysis stages are built of lower-level
+C-functions wrapped in a higher-level programming language,
+\tbd{Python}.
+The decision to execute a specific analysis stage for a specific
+dataset is made by the Scheduler, which sends the infomation to the
+Controller.  The Controller executes the analysis stage for the data
+on an appropriate machine and monitors the success or failure of the
+job.
+\begin{figure}
+\begin{center}
+\resizebox{8cm}{!}{\includegraphics{pics/stages.ps}}
+\caption{ \label{stages} IPP System Overview}
+\end{center}
+\end{figure}
+\paragraph{Science Image Analysis}
+\subsubsection{Science Image Analysis}
 The Science Image analysis stages together represent the basic data
+analysis required by the IPP.  These analysis stages must process the
+images in a timely manner so that the incoming data stream will not
+overload the Pixel Server.  The required processing time is derived
+from the rate at which science images are obtained by PS-1.  At a
+minimum, the Science Image Analysis must keep up with the average
+image rate over the course of 1 day.  \tbd{The Science image analysis
+is required to process images at the maximum science image rate from
+PS-1 of 1 image every 30 seconds -- does this fall out of the science
+requirements?}  \tbd{In order to give time for uncertainties in the
+Pan-STARRS system as a whole, the Science Image Analysis must be able
+to process all images from a night within 12 hours.}
+\tbd{number of images per night, data volume per image, output
+products}
+The science image analysis which must be performed by the IPP consists
+of:
+\begin{itemize}
+\item detrending the images to remove the instrumental signature
+\item astrometric and photometric calibration of the individual images
+\item merging a collection of several images of the same portion of
+the sky obtained over a short period of time (to remove image defects
+and gaps)
+\item subtracting the appropriate reference static-sky image
+\item cleaning the image of any transients
+\item adding the cleaned image to the static sky
+\item object detection of images at specific stages
+\end{itemize}
+These analysis steps can be grouped into four phases, each of which
+deals with a single data unit.  We identify and discuss the
+requirements of the four phases below.
+\paragraph{Phase 1 : image processing preparation}
+The Phase 1 analysis stage is performed on each science FPA to
+calculate basic astrometric \tbd{and photometric} data needed by the
+later stages.  Phase 1 must use the static (pre-determined) telescope
+distortion model and table of nominal OTA positions and rotations,
+combined with the guide star pixel and celestial coordinates, to
+determine the correct telescope bore-sight, field rotation and
+magnification.  The astrometric accuracy required from this analysis
+stage is \tbd{2 arcsec} across the field, sufficient to match the vast
+majority of reference stars with their detections.
+In some circumstances, science images may have no guide stars.  This
+may occur if the detectors are not run in OTA mode, especially for
+short snapshot images of if IPP is being run on non-Pan-STARRS data.
+In such a circumstance, the Phase 1 stage must perform extremely basic
+object detection, determining the detector coordinates for stars which
+are not excessively saturated and which are significantly above the
+background level.  The threshold levels for this object detection
+stage must be configurable.  The object extraction must be performed
+in less than \tbd{3 seconds}.
+In order for astrometry of an image to succeed, it is necessary that
+approximate image coordinates be known.  The Phase 1 analysis must be
+able to succeed despite initial coordinate errors as large as \tbd{5
+times} the field width.  However, the search process must attempt the
+near matches first in the assumption that the given coordinates are
+accurate.
+A table of the overlaps between the science image to be processed and
+the static sky images must be constructed.  This table will be used to
+guide the processing of the static sky in Phase 4.  The overlaps must
+be generously calculated so that small errors in astrometry at Phase 1
+will not cause any valid static sky / science image pairs to be missed
+because of the astrometric error at this phase.  It is acceptable for
+a small number of invalid overlaps to be identified as these will be
+excluded in Phase 4.  Sky cells which do not have sufficient science
+image overlap \tbd{$< 10\%$} need not be processed.
+analysis required by the IPP.  There are several requirements which
+must be met by the collection of science image analysis stages as a
+group.
+The science image analysis stages must perform their analyses quickly
+enoough to keep up with the incoming data stream.  The required
+processing time is derived from the rate at which science images are
+obtained by PS-1.  At a minimum, the Science Image Analysis must keep
+up with the average image rate over the course of 1 day.  In order to
+provide a sufficient buffer for variations in the processing speed,
+the Science Image Analysis must be able to process all images from a
+night within 12 hours.
+The maximum latency between the aquisition of an image and the
+completion of the science image analysis is set by the science
+requirements of the fast transient recovery programs.  The science
+image analysis must process images from these observing programs
+within \tbr{5 min} of their arrival time in the IPP Image Server.
+The science image analysis stages must processes up to 1000 science
+images per night.
+\subsubsection{Phase 1 : image processing preparation}
+The Phase 1 analysis stage must determine the astrometric solution of
+the complete camera (FPA image) with an accuracy of \tbr{1 arcsec}
+peak-to-peak deviation.
+The Phase 1 analysis stage must load the guide star pixel and
+celestial coordinates from the \tbd{IPP Metadata Database}\comment{or
+from the image header?}.
+If guide stars are not available, the Phase 1 analysis stage must
+extract bright stars from the image.  This extraction must be done in
+less than \tbr{1 second}.  The total number of stars and size of the
+bright-star aquisition box must be a user-configurable parameter.
+In order for blind astrometry of an image to succeed, it is necessary
+that approximate image coordinates be known.  The Phase 1 analysis
+must be able to succeed despite initial coordinate errors as large as
+\tbr{20\arcsec}.
+The Phase 1 analysis stage must construct a table of the overlaps
+between the science image to be processed and the static sky images.
+The overlaps must overestimated by a small amount so that errors in
+astrometry at Phase 1 will not cause any valid static sky / science
+image pairs to be missed.  The amount of overlap must be a
+user-configurable parameter.
+Sky cells which do not have sufficient science image overlap \tbd{$<
+\%$} must be excluded.
 It is not unusual that an image be obtained with invalid coordinates
 …
 may make an error and report the wrong time or coordinates.  Or, the
 image may be obtained in exceptionally poor conditions with no
+detected stars.  Phase 1 must fail gracefully in these conditions,
+reporting an appropriate error.  Such images must be identified for
+possible human intervention, or future follow-up after metadata
+repairs are made.
+\paragraph{Phase 2 : image reduction}
+detected stars.  Phase 1 must return a descriptive error message in
+these conditions.
+\subsubsection{Phase 2 : image reduction}
 The Phase~2 analysis is the detrend stage, in which the images from
+the detector are processed to remove instrumental signatures.  In
+addition, basic object detection is performed along with improved
+astrometric and photometric calibration.  \tbd{what component selects
+the appropriate calibration data?  is it the phase~2 program, the
+individual modules, or the scheduler above it?}  In each step of the
+analysis process, an image mask and noise map must be carried and
+updated when appropriate.  The following operations need to occur
+within Phase~2 processing:
+the detector are processed to remove instrumental signatures.
+Phase 2 must perform the analysis steps only if required by the
+processing recipe.  The processing recipe must respect exposure time
+and background flux limits to select certain stages.
+\paragraph{Detrend Image Convolutions}
+The Phase 2 analysis stage must determine the OT kernel from the IPP
+Metadata Database\comment{or image header}.
+The Phase 2 analysis stage must convolve the flat-field and
+high-spatial-frequency fringe images with the OT kernel.  If no OT
+kernel exists, this step must be silently skipped.
+\paragraph{Flag bad and saturated pixels}
+The Phase 2 analysis must load the basic bad pixel map appropriate to
+the detector of interest.
+The Phase 2 analysis must use the OT kernel to grow the traps in the
+raw bad pixel mag.
+The Phase 2 analysis must mask saturated pixels and a user-specified
+number of surrounding pixels.
+Different bits must be set to identify different reasons for masking
+the pixels.
+\paragraph{Bias correction via overscan subtraction}
+Phase 2 must be perform bias subtraction on the image.
+Phase 2 must choose the bias subtraction method and applied statistics
+based on a user-configured parameter.
+The bias correction must be measured from the image overscan region.
+The overscan region must be determined from the image
+header\comment{or Metadata DB}.
+The bias subtraction must apply one of the following bias corrections,
+depending on the user parameters:
 \begin{enumerate}
+\item Convolve detrend images with the OT kernel, if available
+\item Flag bad and saturated pixels
+\item Bias correction via overscan subtraction
+\item Trim object image to remove overscan and edges corrupted by OT
+\item Correct for non-linearity
+\item Flat-field correction
+\item Sky subtraction
+\item Identify CRs
+\item Find objects in the image
+\item Make postage stamps of bright objects.
+\item subtract a single constant from the image.
+\item subtract a 1-D bias which varies along the overscan.  The function to be used must include
+a spline or a chebychev polynomial derived from the data values along
+the overscan, as specified by the user parameters.
+\item correct the overscan {\em and} subtract a 2-D bias image which
+  has been overscan corrected using one of the two methods above.
 \end{enumerate}
+\subparagraph{Convolve detrend images with the OT kernel}
+Detrend images must be convolved by the OT kernel, so that they
+accurately represent the detrend images appropriate for the object
+images, which have been shifted using OT.  The detrend images which
+must be convolved include: the flat-field and the
+high-spatial-frequency fringe images. \tbd{Must this be a formal
+convolution with the analytical OT kernel, or can it be a convolution
+with a decomposed kernel?} The appropriate kernel for each cell of an
+OTA must be determined from the guide star history.  \tbd{what is the
+source of the OT kernel?  pixel server?}
+\subparagraph{Flag bad and saturated pixels}
+A static bad pixel mask needs to be used to identify pixels which are
+bad.  Note that bad pixels which are charge traps need to be grown by
+the extent of the OT convolution kernel, while those pixels above a
+charge trap (i.e.\ bad colums) must not be grown, since they were not
+affected by pixel shifting, but only became bad at read-out.
+Pixels saturated in the A/D converter must also be masked, and this
+area must be grown by an additional pixel to mask excess charge
+spillover.
+The bad pixel mask must be carried with the science images.  Different
+bits must be set to identify different reasons for masking the pixel.
+\subparagraph{Bias correction via overscan subtraction}
+The image bias must be subtracted. Since different detectors behave in
+different ways, several options for modelling the bias must be
+available.  The bias must be measured from the image overscan region.
+The bias subtraction method must be capable of applying a single
+constant to the complete image, or to represent the bias as a function
+which varies along the overscan.  The function to be used must include
+a spline or a chebychev polynomial derived from the data values along
+the overscan.  The values used to determine both the single constant
+or the inputs to the spline and polynomial fits must be derived from
+groups of pixels on the basis of one of several statistics, including
+the sample and robust mean, median, and modes.  In the case of a
+single constant, all of the overscan pixel values are used in the
+calculation of this statistic.  In the case of the 1D functional
+representation, the input values to the fit must represent the
+coordinate along the overscan, with the statistic derived from the
+pixels in the perpedicular direction at each location.  Sigma-clipping
+on the input data values must be an option.  \tbd{accuracy of the bias
+subtraction?}
+\subparagraph{Trim object image}
+The image must be trimmed to remove the non-imaging pixels, such as
+the overscan and any pre-scan pixels, along with those pixels near the
+edges that have been compromised due to OT operation.  The definition
+of the imaging area of the detector must be determined from the camera
+configuration data or from the metadata associated with the image,
+with the choice a user-configurable option.
+\subparagraph{Correct for non-linearity}
+If required, the object image (after bias correction) must be
+corrected for the effects of non-linearity through a provided
+polynomial fit to the pixel data values.  The choice to apply the
+correction must be set by the user.
+\subparagraph{Flat-field correction}
+The statistic used to calculate the overscan constant or the inputs to
+the spline and polynomial fits must be derived from groups of pixels
+on the basis of one of several statistics, as specified by the user
+parameters.  The choice of statistics must include the sample and
+robust mean, median, and modes.
+In the case of a single constant, all of the overscan pixel values are
+used in the calculation of this statistic.  In the case of the 1D
+functional representation, the input values to the fit must represent
+the coordinate along the overscan, with the statistic derived from the
+pixels in the perpedicular direction at each location.
+If specified in the user parameters, sigma-clipping must be performed
+on the input data values.
+The bias subtraction must leave no residuals greater than \tbr{1 DN}
+peak-to-peak.
+\paragraph{Trim object image}
+The Phase 2 analysis must trim the non-imaging pixels from the image.
+The definition of the imaging area must be determined from the
+Metadata Database\comment{or image header?}.
+Phase 2 must trim pixel near the edges that have been compromised due
+to OT operation.
+\paragraph{Correct for non-linearity}
+If required, the science image must be corrected for the effects of
+non-linearity.  The correction must be a function of chip.
+\paragraph{Flat-field correction}
 The object image (after bias correction and non-linearity correction)
 must be corrected for sensitivity variations as a function of
+position, dividing by a flat-field image.  The flat-field images must
+be appropriately normalized (see section \ref{mkcal}).  The
+flat-fielded image must have a consistent photometric zero-point
+across the chip, and across the full FPA, to within 0.2\%.
+\subparagraph{Sky \& Fringe subtraction}
+position, dividing by a flat-field image.
+The flat-field images must be appropriately normalized (see section
+\ref{mkcal}).  The flat-fielded image must have a consistent
+photometric zero-point across the chip, and across the full FPA, to
+within 0.2\% with peak-to-peak deviations of \tbr{0.5\%}.
+\paragraph{Sky \& Fringe subtraction}
 The flux contribution of the sky (from both continuum emission and the
 …
 \tbd{What is allowed power-spectrum of background variations?}
 \subparagraph{Identify `cosmic rays'}
+\paragraph{Identify `cosmic rays'}
 Charged particles in the detector frequently cause features which do
 …
 Phase~2.}
 \subparagraph{Find objects in the image}
+\paragraph{Find objects in the image}
 Objects on the flat-fielded object image must be found, and general
 …
 relevant image metadata (\ie filter, exposure time, etc).
 \subparagraph{Astrometry}
+\paragraph{Astrometry}
 Objects detected in Phase~2 must be matched with known astrometric
 …
 arcsec}.
 \subparagraph{Postage Stamps}
+\paragraph{Postage Stamps}
 The IPP must have the capability of extracting regions surrounding a
 …
 of a set of rules applied to the object magnitude and position.
 \paragraph{Phase 3 : exposure analysis}
+\subsubsection{Phase 3 : exposure analysis}
 The Phase 3 analysis stage works with the results from a complete FPA
 …
 limited by the astrometric reference catalog \tbd{30 mas for USNO?}
 \paragraph{Phase 4 : image combination}
+\subsubsection{Phase 4 : image combination}
 Phase 4 is the image combination stage, in which multiple images of
 …
 into several stages, each of which are discussed in detail below.
 \subparagraph{Extract image pixels}
+\paragraph{Extract image pixels}
 For the given sky cell, the corresponding set of image pixels must be
 …
 than 20\% more pixels than necessary from the input images.
 \subparagraph{Transform pixel coordinates}
+\paragraph{Transform pixel coordinates}
 Pixels which have been extracted from the input images must be mapped
 …
 \tbd{interpolation method?}
 \subparagraph{Flux matching}
+\paragraph{Flux matching}
 The multiple input images must have their object fluxes intercompared
 …
 photometrically.
 \subparagraph{Image outlier pixel rejection}
+\paragraph{Image outlier pixel rejection}
 Pixels from the group of images which are inconsistent with the
 …
 obtained over a wide range of times.
 \subparagraph{PSF matching}
+\paragraph{PSF matching}
 The multiple input images must have their PSF mutually matched to
 allow for proper image subtraction.
 \subparagraph{Image Subtraction}
+\paragraph{Image Subtraction}
 The static sky image must be subtracted from the stacked, cleaned
 …
 Object detection at this stage is the same as that used for Phase 2.
 \subparagraph{Cleaned Input Image}
+\paragraph{Cleaned Input Image}
 The flagged pixels must be excluded from the input images and a new,
 …
 applied to it.  \tbd{parameters}
 \subparagraph{Update static sky}
+\paragraph{Update static sky}
 The final, cleaned input image must be added to the static sky so that
 …
 \tbd{parameters, weight map}
 \subparagraph{Products}
+\paragraph{Products}
 Phase 4 must produce the following data products at a minimum:
 …
 \end{enumerate}
 \subparagraph{Timing}
+\paragraph{Timing}
 It is required that the {\em total} processing for each exposure by
 …
 second.
 \subparagraph{Accuracies}
+\paragraph{Accuracies}
 Transformations/mappings from detector to sky must preserve both
 …
 \end{itemize}
 \subparagraph{Robustness}
+\paragraph{Robustness}
 It is essential that the static sky image (which may have been
 …
 to an error upstream in the processing).
 \paragraph{Calibration Stages}
+\subsubsection{Calibration Stages}
 \label{mkcal}
 …
 below.
 \paragraph{Basic Calibration Stages}
+\subsubsection{Basic Calibration Stages}
 The IPP must generate basic calibration images using the raw bias,
 …
 see which input images are consistent and valid.
 \subparagraph{bias images}
+\paragraph{bias images}
 Bias images may be needed to correct for structure in the bias.  The
 …
 used to exclude any significant outlier input images.
 \subparagraph{dark images}
+\paragraph{dark images}
 Dark images may be needed to correct for structure in the dark
 …
 -- by what component?}.
 \subparagraph{flat-field images}
+\paragraph{flat-field images}
 Master flat-field images must be constructed from a collection of
 …
 exclude any significant outlier input images.
 \paragraph{Other Calibration Stages}
 \subparagraph{mask images}
+\subsubsection{Other Calibration Stages}
+\paragraph{mask images}
 Initial bad-pixel mask images must be generated on the basis of
 …
 inconsistent, an error must be raised.
 \subparagraph{fringe frames}
+\paragraph{fringe frames}
 Fringe-correction frames must be generated to remove the fringe
 …
 standard combination statistics (mean, median, mode, etc).
 \subparagraph{low-k sky models}
+\paragraph{low-k sky models}
 Large-scale background structure in images which is not caused by
 …
 telescope.  \tbd{discuss principal components, SVD?}
 \subparagraph{Flat-field correction frame}
+\paragraph{Flat-field correction frame}
 Flat-field images, whether constructed from the dome, twilight, or
 …
 sequence of images.
 \subparagraph{Non-linearity correction frames}
+\paragraph{Non-linearity correction frames}
 The IPP must have the capability of constructing non-linear correction
 …
 from a linear detector.
 \paragraph{Reference Catalog Creation}
+\subsubsection{Reference Catalog Creation}
 For PS-1, one of the primary goals is the creation of photometric and astrometric
 …
 list the requirements of the tools needed for this effort.
 \paragraph{Astrometry Reference Creation}
+\subsubsection{Astrometry Reference Creation}
 The existing astrometric reference catalogs are known to have
 …
 stars rather than for the normal image data.
 \paragraph{Photometry Reference Creation}
+\subsubsection{Photometry Reference Creation}
 The IPP must provide the analysis tools needed to generate a master
 …
 stars rather than for the normal image data.
 \subsubsection{Modules}
+\subsection{Modules}
 In order to encapsulation functionality, the analysis stages are
 …
 Processing Pipeline Algorithm Design Document' (PSDC-430-006).
 \subsubsection{PanSTARRS IPP Library}
+\subsection{PanSTARRS IPP Library}
 In order to facilitate testing and development, and to encourage
 …
 PSDC-430-006).
 \subsubsection{Data Sources and Formats}
 \paragraph{Image Formats}
+\subsection{Data Sources and Formats}
+\subsubsection{Image Formats}
 FITS images
 \paragraph{Table Formats}
+\subsubsection{Table Formats}
 FITS tables
 \paragraph{Other Data Formats}
+\subsubsection{Other Data Formats}
 XML files
 \paragraph{External Catalogs}
+\subsubsection{External Catalogs}
 \begin{itemize}
 …
 \end{itemize}
 \paragraph{Analysis Reference Data}
+\subsubsection{Analysis Reference Data}
 \begin{itemize}
 …
 \end{itemize}
 \paragraph{Installation Reference Data}
+\subsubsection{Installation Reference Data}
 \begin{itemize}

Note: See TracChangeset for help on using the changeset viewer.

Context Navigation

Changeset 810 for trunk/doc/design/ippSRS.tex

Legend:

trunk/doc/design

trunk/doc/design/ippSRS.tex

Download in other formats: