% -*- latex -*- 
%%% $Id: ippSCDdraft.tex,v 1.1 2004-11-30 23:16:57 eugene Exp $

\providecommand{\setflag}{\newif \ifwhole \wholefalse}
\setflag
\ifwhole\else
\documentclass[panstarrs,spec]{panstarrs}
\setcounter{secnumdepth}{5}
\begin{document}
\section{Pan-STARRS IPP SCD Standalone Section}
\newcommand\PSeten{$\gtrsim 40$\,m$^2$\,deg$^2$}
\newcommand\PSiqdeg{$<$ 27\%}
\newcommand\FRM[2]{\parbox[t]{#1}{\raggedright #2}}
\newcommand\FRA{100pt}
\newcommand\FRB{260pt}
\newcommand\FRC{40pt}
\newcommand\FRD{80pt}
\newcommand\FRN[1]{\FRM{50pt}{#1}}
\newcommand\FRS[1]{\FRM{190pt}{#1}}
\newcommand\grizy{$grizy$}
\fi 

% \section{Image Processing Pipeline}

\subsection{Subsystem Overview}
\label{IPP}

The Pan-STARRS Image Processing Pipeline (IPP) performs the image
processing and data analysis tasks needed to enable the scientific use
of the images obtained by the Pan-STARRS telescopes.  The primary
goals of the IPP are to process the science images from the Pan-STARRS
telescopes and make the results available to other systems within
Pan-STARRS.  It also is responsible for combining all of the science
images in a given filter into a single representation of the
non-variable component of the night sky called the ``Static Sky''.  To
achieve these goals, the IPP also performs other analysis functions to
generate the calibrations needed in the science image processing and
to occasionally use the derived data to generate improved astrometric
and photometric reference catalogs.  It also provides the
infrastructure needed to store the incoming data and the resulting
data products.

The IPP inherits lessons learned, and in some cases code and prototype
code, from several other astronomy image analysis systems, including
Imcat (Kaiser), the Sloan Digital Sky Survey (REF), the Elixir system
(Magnier \& Cuillandre), and Vista (Tonry).  Imcat and Vista have a
large number of robust image processing functions.  SDSS has
demonstrated a working analysis pipeline and large-scale database
system for a dedicated project.  The Elixir system has demonstrated an
automatic image processing system and an object database system for
operational usage.

The users of the IPP output are all systems internal to the Pan-STARRS
project.  They consist of the Transient Science Client, which will
receive the detections of transient objects on short time-scales; the
Moving Object Processing System (MOPS), which will receive the
detections of non-stationary transient objects on day-to-week
timescales; and the Published Science Products Subsystem (PSPS), which
will receive all data products of interest to the outside world, and
will act as the long-term archive and publishing clearinghouse.

An important operational choice for the IPP is the decision not to
attempt to save all raw data.  Once the IPP is running in a standard
operational mode, data will be processed by the pipeline and deleted
when it is no longer needed.  Raw images will only be saved for a
short period to allow tests and quality assurance, and potentially to
allow systems which study transient phenomena to return to recent data
for closer inspection.  In general, during normal operations, raw
science images will be deleted after $\sim$1 month.

The primary IPP hardware system on which the software operates will
not be located at the summit.  Instead, because of thermal, power, and
space constraints, the hardware will likely be located in a facility
off the mountain.  A subset of processing tasks may eventually be
assigned to machines at the summit if justified by the savings in data
transfer time and cost.

\subsection{Subsystem Top-level Requirements}

The IPP has the following top-level requirements derived from the
system requirements above (Section~\ref{sys:TLR}):


\begin{enumerate}
\item For images obtained in photometric weather with normal detector
  characteristics and providing appropriate flat-field images and
  correction data have been obtained, the IPP shall produce reduced
  science images for each full camera exposure with relative
  photometric zero-point scatter less than 1\% ($1 \sigma$) across the
  full field. 
  \label{TLR:1}

\item For images of reference fields calibrated for the IPP filter set
  and obtained in photometric weather with normal detector
  characteristics and providing appropriate flat-field images and
  correction data have been obtained, the IPP shall determine and
  track zero-points for these exposures with a 1$\sigma$ accuracy of
  1\%.
  \label{TLR:2}

\item For images obtained under normal seeing conditions and optical
  distortion, the IPP shall produce reduced science images for each
  full camera exposure with an astrometric calibration providing $<
  30$ milliarcsecond scatter (1$\sigma$) for sequential images of the
  same location.
  \label{TLR:4}

\item For images obtained under normal seeing conditions and optical
  distortion, the IPP shall produce reduced science images for each
  full camera exposure with an astrometric calibration providing $<
  100$ milliarcsecond scatter (1$\sigma$) relative to the ICRS
  reference system.
  \label{TLR:3}

\item In photometric weather and under moon conditions listed in
  Table~\ref{moonconditions}, the IPP shall produce reduced science
  images for each full camera exposure which have background
  variations of less than 1\% in regions free of large ($> 30$ pixels
  diameter) astronomical structures.
  \label{TLR:5}

\item In photometric weather, the IPP shall produce reduced science
  images for each full camera exposure which have background
  deviations from the static sky in the same filter of less than 1\%
  for the median in large ($> 30$ pixels diameter)
  regions.
  \label{TLR:5a}

\item The IPP shall merge all $g$ filter science images into a static sky image.
  \label{TLR:6}

\item The IPP shall merge all $r$ filter science images into a static sky image.
  \label{TLR:7}

\item The IPP shall merge all $i$ filter science images into a static sky image.
  \label{TLR:8}

\item The IPP shall merge all $z$ filter science images into a static sky image.
  \label{TLR:9}

\item The IPP shall merge all $y$ filter science images into a static sky image.
  \label{TLR:10}

\item The IPP shall merge all $w$ filter science images into a static sky image.
  \label{TLR:11}

\item The IPP shall detect and classify objects on the individual processed science
  images.
  \label{TLR:12}

\item The IPP shall detect and classify objects on the stacked groups
  of science images.
  \label{TLR:13}

\item The IPP shall detect and classify objects on the static sky
  image.
  \label{TLR:14}

\item The IPP shall detect transients with significance $>3\sigma$ in
  the individual science images relative to the static sky
  image.
  \label{TLR:15}

\item The IPP shall degrade the stacked image by no more than \tbr{10
  milliarcseconds (FWHM added in quadrature)} over the theoretical
  limit for the stack under infinite
  sampling.
  \label{TLR:16}

\item The IPP shall perform the processing of science images to the
  level of transient detection and static sky inclusion at a rate such
  that exposures taken at an \tbr{average cadence of 40 seconds} do
  not accumulate in the processing buffer (average throughput
  requirement).
  \label{TLR:17}

\item The IPP shall limit the false alarm rate (FAR) to less than 5\%
  for transient detections $> 5\sigma$ sent to the preferred client
  science pipelines.\footnote{note difference with PS-4: 1\%}
 \label{TLR:18}

\item The IPP shall perform transient detection to a completeness of
  99\% at the completeness for transient detections with a significant
  $> 5\sigma$.

\item The IPP shall publish the static sky images to the Pan-STARRS
  Published Science Products Subsystem (PSPS) at a rate so the full
  sky is transmitted once per year.
  \label{TLR:19}

\item The IPP shall publish the detected objects to the Pan-STARRS
  Published Science Products Subsystem (PSPS) at a rate such that the
  objects from the full sky are transmitted once per
  year.
  \label{TLR:20}

\item The IPP shall send the IPP metadata and received OTIS metadata
  to the Pan-STARRS Published Science Products Subsystem (PSPS)
  weekly.
  \label{TLR:21}

\item The IPP shall provide access to preferred Pan-STARRS science clients to the
  detected transient objects within \tbr{5 minutes}.
  \label{TLR:22}

\item The IPP shall provide sufficent storage volume for raw images from the AP and
  IVP Surveys and the \grizy\ Static Sky.\footnote{note difference with
  PS-4: 1 month of raw images}
  \label{TLR:23}

\item The IPP shall provide sufficient storage volume for all detections from the
  AP, IVP, and MVP Surveys.\footnote{note difference with PS-4: 1 year
  of detections}
  \label{TLR:24}

\item The IPP shall provide sufficient storage volume for 2 years of
 metadata.\footnote{note difference with PS-4: 10 years of
 metadata}
  \label{TLR:25}
\end{enumerate}

\subsection{Subsystem Top Level Description}

We now discuss three aspects of the IPP and their relationships: the
IPP Analysis Stages, the Architectural Components which make up the
IPP, and the Hardware System which provides the basic computer
resources needed by the IPP.  We discuss these aspects of the IPP in
general terms first, and in later sections, address the conceptual
design issues for each of the three aspects of the IPP.

\subsubsection{Analysis Tasks and Stages} 

Specific programs are required to perform the processing steps listed
above.  These can be divided into well-defined analysis stages, each
of which operates on a particular unit of data, such as a single OTA
image or a collection of astronomical objects.  Analysis tasks
representing the different analysis stages are performed on the IPP
computer cluster.  Note the distinction between the generic analysis
{\em stage} and a specific analysis {\em task}.  An analysis stage
represents a type of analysis which is performed, such as the basic
image calibration and object detection analysis.  An analysis task is
a particular realization of an analysis stage, e.g., the analysis of
OTA number 61 from exposure 654321 to produce a specific set of output
data products.  The analysis stages are discussed in detail in
Section~\ref{IPP:AnalysisStages}.

\subsubsection{Architectural Components}

In order to achieve the required functionality, the IPP provides an
infrastructure within which the Analysis Stages above are exectuted.
We have divided the IPP software infrastructure into a number of
clearly-defined architectural software units, listed as follows:

\begin{enumerate}

\item {\bf Image Server:} This component is a large data store for all
  images used by the IPP, including the raw images from the telescope,
  the master calibration images, the reference static-sky images, and
  any temporary image data products produced by the IPP.  The Image
  Server accepts the incoming data and stores it until it is no longer
  needed by other portions of the IPP.  The Image Server is not
  restricted to imaging data: it is capable of storing any large data
  files which are not well-suited for inclusion in a more structured
  relational database and for which access needs to be widely
  available beyond the individual process which created the file.

\item {\bf Astrometry \& Photometry Database (AP DB):} This component
  stores and manipulates astronomical objects detected in various
  images, as identified above, including individual measurements of
  objects on the images, the summary information about those objects,
  and reference object data.  It also provides mechanisms for users to
  query and manipulate the objects and detections.

\item {\bf Metadata Database:} This component stores the data which is
  not directly related to images or astronomical objects, but which is
  needed to perform the IPP analyses.  The metadata may include the
  summary weather information for each night, or details about the
  filters, camera, telescopes, etc.  

\item {\bf IPP Controller:} In order to perform the analysis stages
  required by the IPP, it is necessary to use distributed computing
  processes on a large number of computers.  The IPP Controller
  manages the collection of analysis tasks performed on these
  machines.

\item {\bf IPP Scheduler:} This component is a decision-making
  mechanism which guides the operation of the IPP.  It evaluates the
  currently available collection of data, identifies the necessary
  analysis, and assigns the analysis tasks to the IPP Controller.

\end{enumerate}

The relationship between these software units is shown in
Figure~\ref{overview}.  This figure also shows the interactions
between the IPP and other Pan-STARRS systems.  The architectural
components are discussed in detail in
Section~\ref{IPP:ArchComponents}.

\begin{figure}
\begin{center}
\resizebox{6in}{!}{\includegraphics{pics/IPPoverview}}
\caption{ \label{overview} IPP System Overview}
\end{center}
\end{figure}

\subsubsection{IPP Hardware Organization}

\begin{figure}
\begin{center}
\resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}}
\caption{ \label{hardware} IPP Hardware Organization}
\end{center}
\end{figure}

The IPP needs substantial computer resources, both in terms of
computational power and in terms of data storage and network
bandwidth.  The IPP requires relatively large amounts of data storage
space, primarily for the image data.  Image data is organized in two
categories.  First, there is the per-OTA data -- data associated with
specific OTAs, including the raw images, the calibration images, and
temporary processed images at various stages.  Second, there is the
data associated with the static sky imagery, which is in turn
organized into smaller sky-cell units.  In addition to image data,
there are the somewhat smaller data entities of the Metadata Database
and AP Database.

The computer hardware is organized into nodes which provide both data
storage and computational resources.  The data storage nodes are
divided into three classes: those which deal with the per-OTA image
data, those that provide the storage for the static sky images, and
those that provide the storage for the other data systems, the
Metadata Database and the AP Database.  In addition, the computational
tasks related to Phase 2 take place on the per-OTA storage nodes and
the Phase 4 computation takes place on the static sky storage nodes.

Figure~\ref{hardware} shows our basic concept for the hardware
organization for the IPP.  This diagram shows the two types of compute
nodes: OTA-level processing and storage nodes (dominated by Phase 2)
and static sky processing and storage nodes (mostly Phase 4).  Also
shown are two switches which divide the network into OTA and
Static-Sky portions.  In such an organization, the interswitch
communication must meet the throughput needs between these network
portions.  The additional data systems (Metadata Database and AP
Database) are also shown.

\subsubsection{Data Products}

The IPP data products are listed in Table~\ref{SYS:IPPProducts} and
are discussed in detail in the sections below.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Subsystem Tasks and Functions}
\label{IPP:Tasks}

In order to achieve the top-level requirements listed above, the IPP
performs the following tasks:

\begin{enumerate}

\item Accept raw images from OTIS.

\item Accept metadata from OTIS.

\item Produce high-quality calibration images from the raw calibration
  images.  

\item Pre-process the science images with the high-quality master
  calibration images.  This analysis is called ``Phase 2'' and the
  resulting images are called ``Phase 2'' or P2 images.

\item Perform an image-wide improvement to the astrometric and
  photometric calibrations and the sky background subtraction (``Phase
  3'').

\item Merge multiple pre-processed science images -- from multiple
  telescopes or from sequential, dithered exposures -- into single,
  cleaned, stacked images.  This analysis, and the next task,
  constitute ``Phase 4''.  The resulting images are called ``Phase 4
  Summed'' or P4$\Sigma$ images.

\item Subtract a static sky image from the cleaned, stacked images to
  produce an image of only the transient events.  The resulting images
  are called ``Phase 4 Difference'' or P4$\Delta$ images.

\item Excise the significant transients and outliers from the
  pre-processed science images and merge the cleaned images into the
  static sky image.

\item Detect objects on the four types of images: pre-processed
  images, the stacked image, the difference image, and the static sky
  image.

\item Determine astrometry of the detected objects relative to an
  astrometric reference.

\item Determine photometry of the detected objects relative to a
  photometric reference.

\item Provide the tools and data sources needed to construct a
  high-quality astrometric reference catalog from the extracted
  objects.

\item Provide the tools and data sources needed to construct a
  high-quality photometric reference catalog from the extracted
  objects.

\item Publish the static sky images to the Pan-STARRS PSPS after data
validation.

\item Publish the detected objects to the Pan-STARRS PSPS after data
validation.

\item Provide access to MOPS to the single-occurrence detections of
transient objects on short time scales.

\item Provide access to MOPS to the metadata specific to the images in
which single-occurrence detections were found.

\item Provide access to othe preferred Pan-STARRS science clients to
  the measurements of the detected transient objects on short time
  scales.

\item Store the raw images for a period of time, depending on the
  survey source of the data.  During normal operations, raw data is
  stored for \tbr{1 month}.

\item Store the detected objects for a period of time, depending on
  the type of detection.  Transients from the P4$\Delta$ images may be
  excised after \tbr{6 months}.

\end{enumerate}

\subsection{Operational Scenarios}

In the normal operational state, the IPP continuously accepts data
from the summit (to the Image Server) and schedules new analysis tasks
for operation.  At the start of a night, it will need to choose
between performing analysis on science images and analysis on
calibration images.  It will also feed data to the MOPS and other
preferred science clients (such as the Transient Science Client) in
real time.  For the MOPS, this data feed occurs at the end of each
night.  Other clients may require the data to be delivered more
rapidly.  The IPP has a top-level requirement to be capable of
delivering the transients to clients which need them within 5 minutes
of the image being taken.

As a continuous process, the IPP will perform detailed object analysis
on the static sky images.  The logical time to schedule this analysis
is to process those portions of the sky which are within $\sim 15$
degrees of the sun (in RA), since these are regions which are
guaranteed to be unobserved and unchanging for at least 2 months.

Some of the IPP computational and data resources will also be used to
construct the astrometric and photometric calibration catalogs.  The
processing tasks involved in this effort will be assigned to the IPP
cluster along with the standard, nightly processing tasks.  These
analysis tasks will be initiated by a human.

On a longer term timescale (possibly once a month when the system is
running in an operational mode), some of the data products will be
pushed to the PSPS.  In some cases (static sky images, most of the
metadata), this publication will involve the simple copying of the
data structures to an identical system on PSPS hardware.  In other
cases (object detections), data will be sent to the PSPS which will be
processed by the PSPS for incorporation in PSPS databases.  For the
static sky publication, it is again logical to send those portions of
the static sky which are within $\sim 15$ degrees of the sun in RA.

The start-up process for the IPP as a whole may be viewed as a
start-up procedure for each of the architectural systems
independently.  The three data storage systems (Image Server, AP
Database, and Metadata Database) need to check the existence and
validity of the hardware and data it manages.  The IPP Controller
needs to determine which computers are available before allowing
processing.  The IPP Scheduler will need to check that all of the
other systems are sufficiently operational before attempting to make
decisions and submit jobs to the IPP Controller.

\subsection{Conceptual Design}

\subsubsection{Architectural Components}
\label{IPP:ArchComponents}

\paragraph{Image Server}
\label{IPP:ImageServer}

The IPP Image Server is a large data store for all images used by the
IPP.  The Image Server stores all of the images needed by the IPP for
the length of time they are required; total data volume is specified
in detail in the separate report, `The Pan-STARRS Image Processing
Pipeline Computational Challange' (PSDC-4xx-xx); the total data volume
for PS-4 is approximately 1000 TB.

The IPP Image Server stores science and calibration images as FITS
files on disk Images of the Static Sky are stored as a variant of
FITS, using a data representation yet to be determined to minimize the
total data volume and pixel overlap and to minimize the losses from
warping of the images.  The optimal representation of the Static Sky
is a topic for study by the IfA IPP Team.  Two parameters which
critically affect the total data volume requirements of the Image
Server are 1) the pixel scale (arcseconds per pixel) and 2) the byte
representation of the data.  Current working numbers for these are
0.2\arcsec and 4 bytes per pixel (2 for the signal and 2 for the noise
map).

The IPP Image Server distributes images across a cluster of machines.
Multiple copies of each image may be requested for redundancy or for
improved throughput.  The image (or one of the copies) may be placed
on a specified machine.  The specific machine on which a particular
copy of an image resides may be determined via the user interface
methods.

The IPP Image Server maintains a record of all images currently
available in the repository.  The Image Server is only responsible for
tracking the location of the images, not for tracking metadata
information such as the image summary statistics or the state of the
image processing for the image.  These aspects are included in the
Metadata Database discussed below.

The IPP Image Server interfaces with other subsystems of the IPP.  It
provides a mechanism by which other IPP subsystems may identify the
image location (the computer on which it resides).  It also provides a
mechanism to serve a specified image to an IPP or Pan-STARRS subsystem
on request.  It also provides maintainence mechanisms for deletion,
relocation, and duplication of images in the Image Server as
necessary.

The IPP Image Server is not limited to image data.  Any large file
would be an appropriate object to store in the Image Server.  Raw
images from the telescope are stored as separate OTA images, with
multiple Cell images per file, as well as video sequences from the
guide stars in the form of MEF extensions.

%% IPP Image Server T & F

Image Server tasks and functions:

\begin{itemize}

\item The IPP Image Server stores images on a distributed collection
  of computer disks.  Individual instances of a file are only required
  to be stored on a single machine (striping across computers is not a
  requirement).

\item The IPP Image Server attempts to store an image on a specific
  machine if requested by the user.

\item If such a request cannot be honored (ie, the machine is down),
  the IPP Image Server selects an appropriate machine and notifies the
  requesting agent of the new location.

\item The IPP Image Server stores multiple copies of each image upon
  request, the number of copies specified independently for each file
  by the user.

\item The IPP Image Server maintains a record of all image copies
  currently available in the repository.  This record includes at
  least the image name, location (which machine), the image size, and
  the state of the image (available, locked,
  deleted).

\item The IPP Image Server locks images in the repository on request.
  Both read (shared) and write (exclusive) locks are provided.  A read
  lock prevents write access to the file; a write lock prevents both
  read and write access.  Access prevention may be advisory rather
  than rigorously enforced.

\item The IPP Image Server return the image location (the computer or
  computers on which it resides) upon request.

\item The IPP Image Server provides a specified image upon request.

\item The IPP Image Server deletes images in the repository on
  request.
\end{itemize}

\paragraph{AP Database}
\label{IPP:APDB}

The purpose of the AP Database is:
\begin{itemize}
\item to enable the photometric calibration of images
\item to enable the astrometric calibration of images
\item to enable the construction of flat-field correction frames
\item to enable the construction of a photometric calibration catalog
\item to enable the construction of an astrometric calibration catalog
\item to monitor the system photometry calibration parameters
\item to monitor the system astrometry calibration parameters
\item to perform the identification of single-occurance transients
\end{itemize}

The AP (Astrometry \& Photometry) Database is a mechanism to store
data related to astronomical objects derived from various sources with
a variety of associations.  The AP Database deals with two related
concepts: {\em objects} and {\em detections}.  The objects are
descriptions of astronomical objects while the detections are the
specific measurements of those objects, typically measured from
astronomical images.  A collection of {\em detections} may be used to
derive average quantities which describe a particular {\em object}.  A
third class of object information which must also be considered are
those supplied by external references.  These may be treated as {\em
detections}, with the caveat that access to the raw measurements and
metadata are usually unavailable; the reported measurements and errors
must be accepted as they are reported.

The AP Database stores the collections of detections which were
derived from specific images from any of the analysis stages.  It
provides a mechanism to determine and (in conjunction with the Image
Server) locate the image from which a specific detection was derived.
The AP Database also makes it possible to extract all detections
derived from a specific image and to determine quantities such as the
coordinates of the detection in pixel coordinates on the image.

The AP Database also has the capability to associate multiple
detections of a specific object.  Several major classes of objects
will be present, each of which must be handled correctly.

First, the most distant stars, compact galaxies, and QSOs will have
nearly fixed locations relative to other nearby stars, with only small
deviations for individual measurements.  The association between
multiple detections of such objects is made on the basis of their
coincident positions.  The AP Database determines the average position
of the object and the deviations of the individual detections from
that average on the basis of the ensemble of individual detection.

Second, solar system objects do not have a fixed location.  Detections
of such objects are linked by their orbits, and depend on both the
position and the time of the image.  The AP Database does not attempt
to make this link, which is the role of the MOPS system.  However, it
has the ability to accept identifications made externally with
specified detections and to return the identifier of the moving object
associated with the specific detections.  These associations also
include descriptive information such as the offset of the detection
from the predicted location of the detection based on the orbit.  This
functionality is required to allow the AP Database to ignore known
moving object detections from other types of queries.

Third, stars in the general vicinity of the solar system fall in
between these first two classes of objects.  Their proper motion and
parallax response is significant enough ($>1$ arcsec in 1 year) that
they are not well-described by an average location and a collection of
offsets.  These objects are described by a distance and a proper
motion vector.  The AP Database provides the association between the
specific detections and an average object which includes finite
parallax and proper motion.

Fourth, many detections, especially in their initial states, will not
be associated with a specific astronomical object of any of the above
classes and are treated as orphans.  Most of these will be spurious
(not represent real objects), some will be from solar system objects
for which orbits are not yet determined, some will be from faint stars
near the detection limits, some will be from short-term transients
which have only been detected once.  The AP Database maintains these
detections until they have been associated with one of the objects
above.  The AP Database provides mechanisms by which individual
detections may be migrated back and forth between the orphan state and
association with an astronomical object.

For every object, and all orphaned detections, the AP Database also
provides the capability to determine the images which observed the
location of the object but for which no detection was made.  The
minimum set of information which must be carried for these
non-detections is the image and the associated object or orphan.

The AP Database also stores the relationships between various
photometric systems and, in some cases, the evolution of that
relationship.  It provides mechanisms to convert between the measured
instrumental magnitude of a detection with a specific filter,
detector, and telescope, and at a particular time and the implied
magnitude in the average Pan-STARRS photometry system, given a
determined set of calibrations.  It also provides the capability to
convert magnitudes in one system to the magnitudes in another system;
an example of such a conversion is between the average Pan-STARRS
filter systems and the various reference systems appropriate for those
filters.

The AP Database provides interfaces to extract lists of objects and
detections based on various query parameters.  It provides the
capability to extract all detections associated with a specific
object, all non-detections of that object, all non-detections of an
orphan, and summary statistics from these collections.  It will also
return all objects or detections within specified spatial regions
including regions bounded by great circles (RA,DEC; GLAT,GLON;
ELAT,ELON) and regions described by a location and a search radius.
It will also return the image parameters associated with a specific
detection including image coordinates of the detection, exposure time,
time and date of the detection, etc.

\begin{table}
\begin{center}
\caption{AP Detection Classes \& Object Parameters\label{APdetections}}
\begin{tabular}{lrrrr}
\hline
\hline
Object Parameter & P2 & P4S & P4D & SS \\ 
\hline
PSF x,y, covar, $\alpha,\delta$               & + & + & + & + \\
PSF mag, $\sigma_{\rm mag}$                   & + & + & + & + \\
star/gal sep                                  & + & + & + & + \\
$\sigma_x$, $\sigma_y$, $\theta$              & + & + & + & + \\
local sky data                                & + & + & + & + \\
Petrosian R, M, $R_{50}$, $R_{90}$            & - & + & - & + \\
S\'ersic R, M, AB, $\phi$, $\nu$              & - & + & - & + \\
W.L. $\gamma_1$, $\gamma_2$, pol. terms       & - & - & - & + \\
exp. spaced aps., Poisson noise, variance     & - & - & - & + \\
\hline
\end{tabular}
\end{center}
\end{table}

The tasks and functions of the AP Database include:

\begin{itemize}
\item The AP Database accepts and stores individual detections and
  collections of detections along with information about the image
  which provided the detections.

\item Detections are saved as one of several detection classes (P2,
  P4$\Sigma$, P4$\Delta$, SS) and the AP Database stores the
  appropriate parameters, listed in Table~\ref{APdetections}, for each
  class.

\item The AP Database identifies the image which provided the
  detection, or in the case of external references, an identifier
  specific to the reference source.

\item The AP Database groups detections into objects on the basis of
  positional coincidence and measures average parameters of those
  objects.

\item The AP Database stores parallax and proper motion parameters for
  a subset of the average objects.

\item The AP Database stores image and filter calibration information
  necessary to convert between instrumental magnitudes and calibrated
  magnitudes in standard systems.

\item The AP Database performs at least the follow queries, with
  constraints on the output based on at least time ranges, magnitude
  limits, error limits:

 \begin{itemize}
 \item given $(RA,DEC)$ and a Radius, return all objects and/or
 detections in the region.

 \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all objects and/or
   detections in the region.

 \item given $(RA,DEC)$, return closest object.

 \item given object ID, return all detections.

 \item given detection, return source image data.

 \item given detection, return object.

 \item given $(RA,DEC)$, return all images overlapping coordinate.

 \item given $(RA,DEC)$ and a Radius, return all images overlapping region.

 \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping region.

 \item given detection instrumental magnitude, return derived
   magnitudes based on calibration information.

 \item given a collection of detections in a filter, determine the
   object average magnitude in that filter.

 \item given a collection of objects and detections, determine the
   individual image zero-points.

 \item given a region, return all possible combinations of the object
   or detection magnitudes $(M_1 - M_2)$.

 \item given a list of $(RA,DEC)$ entries, return all nearest objects.

 \item given a filter, telescope, or detector, return all calibration
   terms and history.

 \item given a detection, return all non-detections from images which
   overlapped the detection coordinates.
 \end{itemize}

\item The AP Database shall accept detection IDs of moving objects and
  label the detections with the identified object.
\end{itemize}

\paragraph{Metadata Database}
\label{IPP:MetadataDB}

The IPP requires a Metadata Database to store and provide access to
metadata of various types and from various sources.  Metadata in the
context of the IPP represents all data which is not included in the
two data stores discussed above (Images and Detection/Objects).
Metadata is generated at the telescope and during the various analysis
stages

The Metadata Database stores and provides metadata for all raw images,
for processed images, for the calibration images (both raw and
master), for the extracted object lists.  Metadata describing the
environmental conditions at the telescope must also be stored and
provided as needed.  The Metadata Database stores descriptive
information about the raw images received from the summit and the
current state of the data processing.  The Metadata Database also
stores descriptive information for each of the static sky images
currently available.

If analysis results are exchanged via the Metadata Database, it must
provide access to the queried data on timescales of $<2$ seconds to
avoid slowing down the analysis systems.

The IPP also requires a Configuration Database to store and provide
access to information about the IPP itself.  Examples of data in the
configuration database include the default parameters for the various
analysis programs, the description of the computing environment, the
process status information, etc.  Software configuration parameters
are stored in and extracted from the Metadata Database.
User-configurable software parameters are also stored in and extracted
from the Metadata Database.

%% Metadata DB T & F

The Metadata Database tasks and functions:

\begin{itemize}
\item The Metadata Database stores the classes of data listed in
  Table~\ref{metadata}.  Thus, the Metadata Database stores and serves
  metadata for all raw images, for processed images, for the
  calibration images (both raw and master), for the extracted object
  lists.  Metadata describing the environmental conditions at the
  telescope is also stored and provided as needed.  

\item The Metadata Database responds to simple queries which return
  the data in the categories listed in Table~\ref{metadata} based on
  the primary data key and with basic constraints of time ranges and
  other simple conditional constraints.

\item The Metadata database stores the configuration information with
  restricted access so that only specific people may change the
  information (eg, science parameters available to the science team;
  software configuration parameters available to the system
  maintainers).
\end{itemize}

\paragraph{IPP Controller}

The IPP uses a group of computers to store and process images and to
manipulate collections of detections.  These computers perform any of
a large number of analysis stages or other processing tasks without
significant interprocess communication.  It is necessary to have a
mechanism which initiates computing tasks on the different computers,
which monitors the tasks as they are executed, which handles the
output and the errors from these tasks, and which reacts to the
failure of any of the computing nodes.  The system responsible for the
tasks in the IPP is the IPP Controller.

The IPP Controller interacts with the collection of computers under
its management and with other subsystems in the IPP.  The IPP
Controller receives a variety of inputs from other subsystems,
described below, and initiates actions such as adding a new process to
its queue.  The IPP Controller also provides information to other
subsystems on demand about its processing history and current state.
Each physical computer may have multiple processors; since the IPP
Controller is managing processing tasks, it treats each processor
independently.  It is up to the system configuration if each computer
needs to reserve one of its CPUs to manage background tasks or if the
IPP Controller should attempt to send one task per CPU and let the
kernel handle the I/O load.

Computers managed by the IPP Controller are allowed to be in one of
several states, and the IPP Controller must interact with it in an
appropriate way for each of those states.  A computer may be {\tt
alive}, {\tt dead} or {\tt off}.  If the computer is {\tt alive}, it
responds to commands from the IPP Controller and may be used for tasks
subject to other constraints.  If it is {\tt dead}, the computer is
not responsive and must not be used for executing tasks.  The IPP
Controller must identify computers which have died and occasionally
test them to see if they are {\tt alive} again.  Computers which are
{\tt off} are not available for tests and must not be tested.
Computers may be set to the {\tt off} or {\tt dead} states by external
subsystems; it is the responsibility of the IPP Controller to return a
computer to the {\tt alive} state if possible.  An example scenario: a
computer crashes.  At this point the IPP Controller should detect that
the computer is no longer responsive and mark it {\tt dead}.  It
should occasionally try to re-establish communication with the
computer, potentially with longer and longer delays between attempts.
A human could be notified if the computer seems to remain {\tt dead}
for a very long time.  In another circumstance, a person needs to work
on a computer.  They should have the ability to notify the IPP
Controller that the machine is off, perhaps with a prior notification
that the machine should be prepared to go off.  Only when the person
is done working and testing the machine, and tells the IPP Controller
that the machine is now {\tt dead} can the IPP Controller attempt to
re-start communications and processing on that computer.

CPUs on computers which are in the {\tt alive} state may be in one of
two modes: {\tt busy} and {\tt free}.  A CPU which is {\tt busy}
currently has a task assigned to it.  The IPP Controller may only
assign one task to one CPU at a time.  A CPU which is in the {\tt
free} state may have tasks assigned to it.  The IPP Controller must
also respect a list of task restrictions which may require specific
tasks to run on specific CPUs or exclude specific tasks from specific
CPUs.

The IPP Controller accepts tasks from other IPP subsystems.  The task
requests include the specific command to be executed and are in the
form of a UNIX command which could be performed on any of the
computing nodes.  Any input or output data structures in the commands
must be a valid resource regardless of the node on which the task is
executed.  Input and output data resources must be unique where
necessary to avoid conflicts.  The IPP Controller gives each task a
unique identifier, which is returned to the requesting agent.  The
agent may then use that ID to obtain status information on that task
or to send control signals to the specific task.

Task requests may specify a desired node for the task execution.  The
IPP Controller attempts to honor the request if the node is {\tt
alive}, but will execute it on another node if the requested one is
{\tt dead} or {\tt off}.  Even if a node is {\tt alive}, the IPP
Controller will choose another node if the specified task is not
allowed on the requested node.  In all other cases, the IPP Controller
waits until the currently executing processes, and processes with
higher priority, are completed before executing the specified task on
the requested node.

Task requests may specify an urgency level.  The IPP Controller
determines the priority of the task on the basis of both the priority
and the age of the request.  An executing task must be completed on a
CPU before any new task is started on that CPU, regardless of
priority.  Tasks may be assigned a priority of 0 in which case they
are maintained in the queue and never executed.

The IPP Controller monitors the output streams from the executing
tasks and the exit status of the tasks.  Each task is associated with
a log file, to which all output is written.  The status, including the
exit status, of each task is maintained by the IPP Controller so that
other subsystems may determine if specific tasks have started or
completed.

The IPP Controller must accept commands from other IPP subsystems.
These commands include those which govern the processing of specified
tasks, those which govern the behavior of specific computing nodes,
and those which request information from the IPP Controller.  The IPP
Controller must be able to halt the execution of a specified task,
delete an unexecuted task from the task list, change the priority of
tasks, and change the requested nodes for tasks.  The IPP Controller
must also be able to stop the current execution of a task and push it
to the end of the queue and also change its priority.

The IPP Controller must honor requests (normally from the users) to
change the mode of any computing node on demand between {\tt off} and
{\tt dead}.  This would normally be done after a computer has been
rebooted and is release to the IPP Controller for its use.  It must
also be able to change the list of allowed tasks as requested by
external commands.

The IPP Controller must respond to informational requests regarding the
collection of machines and their states as well as the collection of
tasks and their states.  The IPP Controller must monitor the execution
times of the different tasks and provide summary statistics.  Finally,
the IPP Controller must respond to three top-level commands: {\tt finish},
{\tt stop} and {\tt abort}.  When {\tt finish} is requested, no more
new tasks are accepted on the stack of task, and when all tasks in the
stack have completed, the IPP Controller must exit.  When {\tt stop} is
requested, the currently executing tasks must be completed at which
point the IPP Controller must exit, but tasks remaining in the stack which
have not been started are flushed.  When {\tt abort} is issued, the
IPP Controller immediately kills all executing tasks and exits.

The IPP Controller and the IPP Image Server have related needs for
information from the combined storage-and-processing nodes regarding
which nodes are available.  It is not yet clear if this information is
best stored in a single location (either IPP Controller or IPP Image
Server), which provides the information to other systems on demand, or
if both systems should maintain the information.  Also, it may be
necessary to distinguish nodes which are available for processing from
those that are available to serve data as part of the IPP Image
Server.

It may be useful for the Controller to distinguish between tasks
dominated by I/O and tasks dominated by data processing.  It is
possible that one of each of these types of tasks may be sent to the
same node without significantly impacting the system performance.
Alternatively, it may be necessary to limit a single machine with 2
CPUs to only one of each of these types of tasks (i.e., one processor
will be working on I/O while the other is working on processing).
Such details will be studied by the IfA IPP Team.

%% IPP Controller T & F

 IPP Controller tasks and functions:

\begin{itemize}

\item On startup, the IPP Controller attempts to establish
  communication with all of its computers and set their state to be
  {\tt alive} or {\tt dead} based on the success of the
  connection.

\item The IPP Controller detects computers which crash or stop
  responding and set their state to {\tt dead}.

\item The IPP Controller attempts to re-establish communication with
  {\tt dead} computers.

\item The IPP Controller accepts tasks from external users and
  systems, which may specify a desired CPU (node) and priority in
  addition to the task command.

\item The IPP Controller attempts to run pending tasks on the desired
  node, if available (not {\tt dead} or {\tt off}).

\item If the node is unavailable, the IPP Controller attempts to run
  the task on another node.

\item If the node is available, the IPP Controller attempts to run a
  given task only if no higher-priority tasks are available and no
  task is currently being executed.

\item The IPP Controller monitors the output from the task and writes
  it to an associated log destination.

\item The IPP Controller monitors the execution status of each task
  currently executing on a node and performs the following actions:

  \begin{itemize}
  \item identify the task as successful if it has a valid exit status.
  \item identify the task as unsuccessful if it has an error exit status.
  \item identify the task as unattempted if the computer crashed.
  \end{itemize}

\item The IPP Controller accepts and performs the following external
  commands:
  \begin{itemize}
  \item add a task to the pending task list.
  \item delete a specific task from the pending task list.
  \item return the current status of a specific task.
  \item return a list of all pending and non-pending tasks.
  \item set a specified computer state to {\tt off} or {\tt dead}.
  \item restrict a specified CPU to a class of tasks.
  \item halt execution of a specified task.
  \item set the IPP Controller state to {\tt finish}, {\tt abort}, or {\tt stop}.
  \end{itemize}
\end{itemize}

\paragraph{IPP Scheduler}
\label{IPP:IPPscheduler}

The IPP is responsible for a variety of analysis tasks: processing of
the science images through several stages; routine assessment of the
detrend (instrumental calibration) images used in processing the
science images; construction of replacement detrend images when
needed; generation of astrometric and photometric reference catalogs
based on the collected dataset; and the performance of test analysis
programs.  At any point, decisions need to be made about which of
these tasks should be performed, based on an analysis of the contents
of the metadata database, the requirements of the people monitoring
the IPP, and the near-term observing plans.  The IPP Scheduler is the
mechanism that assesses these various inputs to guide the decisions
and initiate the actions.

The IPP Scheduler acts as an intermediary between several components
of the IPP and also between the IPP and external agents such as OTIS
and the users who must monitor the behavior of the IPP.

The IPP Scheduler sends commands to the IPP Controller for execution.
While the IPP Scheduler chooses the tasks to be performed, it is the
IPP Controller's responsibility to manage the specific tasks executing
on a given processing node.  Examples of these tasks include the
process of copying or moving data from the Summit data systems to the
IPP Image Server; image processing analysis stages performed on the
science images by the appropriate processing nodes; and the analysis
of the data in the AP Database.  This division of responsibilites
allows us to isolate and encapsulate the functionality of the IPP
Scheduler and the IPP Controller.  With this separation, the IPP
Controller does not need to have any information about the details of
the tasks which it executes, while the IPP Scheduler does not need to
have detailed information about the available computer hardware.

Communication between the IPP Scheduler and the IPP Controller is
bi-directional; the IPP Scheduler sends tasks to the IPP Controller,
while the IPP Controller informs the IPP Scheduler of the outcome of
those tasks.  It is not specified whether the IPP Scheduler and IPP
Controller are components of a single software system or interacting
but distinct software components.

The IPP Scheduler takes as input the current list of pending images,
both science and calibration, and a description of the current
observing plan or strategy on some time-scale.  The IPP Scheduler also
takes input from humans who manage the IPP.

The IPP Scheduler must choose between several types of analysis tasks
based on the contents of those lists and on the requirements of the
users.  The list of tasks which the IPP Scheduler must decide between
includes:
\begin{itemize}
\item moving data from the Summit pixel server ($\sim 30$ second timescales)
\item running the science analysis stages ($\sim 30$ second timescales)
\item testing the validity of the current detrend images ($\sim$
  nightly)
\item constructing new detrend images ($\sim$ weekly)
\item updating and improving the photometric and astrometric reference
  catalogs ($\sim$ yearly).
\end{itemize}

The IPP Scheduler chooses between tasks which are relevant on several
different time-scales.  The time-scales range from 2 times per minute
to once or twice a year, as noted in the list above.  The IPP
Scheduler must also make use of user input in managing such choices.
Users have the option to specify that a particular task or set of
tasks is of higher or lower priority than the norm.

The IPP Scheduler maintains a set of rules that define the dependency
of one type of analysis stage on other analysis products.  For
example, the nightly science image processing depends on the existence
of valid detrend images.  The IPP Scheduler must be able to recognize
the dependency and initiate the required analysis needed to perform
other analysis tasks.  The IPP Scheduler must have the ability to
decide between postponing an analysis task until the required data are
available or initiating the task using a lower-quality or less
appropriate substitute.  For example, in normal circumstances, a
science image must not be processed until the corresponding detrend
frame has been produced.  However, if such a frame is unlikely to
appear soon, and the pressure to process the science image is
sufficiently high, then the frame could be processed with an older
detrend frame of known lower quality.  The IPP Scheduler must have the
ability to choose the best, if not ideal, reference data for a
particular circumstance.  Note that these rules are defined for the
IPP Scheduler in an abstract way as a relationship between analysis
stages, rather than as a specific rule relating one task to another
task.

The IPP Scheduler defines the operating state of the IPP.  When the
IPP is in the {\em automatic state}, the IPP Scheduler performs the
most appropriate of all possible tasks at a particular time.  When the
IPP is in the {\em interactive state}, the IPP Scheduler performs only
the requested action regardless of the outcome of the decision trees.
In addition, in the interactive state, the IPP Scheduler must only
perform the requested actions and not attempt to perform the other
normally-required actions.  The only exception to this exclusion is
that, in the interactive state, data is still copied from the summit
system.  An additional IPP state is the {\em paused state}, intended
for tests or maintenance, in which case the IPP Scheduler does not
perform even the data copy tasks.  Every task is performed on demand
by the user.  The user command sets the IPP Scheduler in one of these
three states, {\em automatic}, {\em interactive}, and {\em paused}.

%% IPP Scheduler T & F

The IPP Scheduler tasks and functions:

\begin{itemize}
\item The IPP Scheduler sends the analysis tasks which it initiates to
  the IPP Controller.

\item All analysis tasks sent by the IPP Scheduler include a complete
  UNIX command with necessary arguments, the priority of the task, and
  optionally the desired processing node.

\item When the IPP Scheduler is placed in the {\em paused state}, it
  only initiates User-requested tasks.

\item When the IPP Scheduler is placed in the {\em interactive state},
  it initiates User-requested tasks as well as data transfer tasks.

\item When the IPP Scheduler is placed in the {\em automatic state},
  it initiates the most appropriate task based on the inputs and
  dependency rules.

\item The IPP Scheduler sends the exit status of the analysis tasks to
  the appropriate destination as defined by the task dependency table.
\end{itemize}

\subsubsection{Analysis Stages}
\label{IPP:AnalysisStages}

\begin{figure}
\begin{center}
\resizebox{6.4in}{!}{\includegraphics{pics/IPPstages}}
\caption{ \label{stages} IPP Analysis Stages}
\end{center}
\end{figure}

\paragraph{Overview}

We now consider the collection of analysis tasks which must be
performed by the IPP.  These tasks represent the core of the required
IPP functionality; the architectural components discussed above can be
viewed as primarily supporting infrastructure to enable the analysis
tasks to be executed on the appropriate data and to store the results.
The tasks are divided into well-defined analysis stages, each of which
operates on a particular unit of data, such as a single OTA image or a
collection of astronomical objets.

Depending on the analysis stage, the basic data unit may be individual
images, collections of images, or derived data products such as a
collection of detections of astronomical objects.  Because of the
granularity of these data units, a large number of analysis tasks
representing the same analysis stage many be performed in parallel.
This is particularly true because the analysis tasks from any
particular stage do not depend on the results of another analysis task
from that stage.  For example, the intial analysis of a chip from one
image does not depend on the results from another chip.  The analysis
stages are divided into three categories, and further subdivided as
follows:

\begin{enumerate}
 \item {\bf Science Image Analysis} is performed on the night-sky
 science images to extract the science data from these images.  The
 science image analysis is divided into 4 analysis stages, which we
 call Phases 1 - 4:

 \begin{itemize}
  \item {\bf Phase 1:} The image processing preparation stage, in
  which basic astrometric analysis of the complete FPA image is
  performed.

  \item {\bf Phase 2:} The image reduction stage, in which the
  individual detector images (OTAs) are processed as much as possible
  without reference to other chips in the same FPA image or other
  exposures.

  \item {\bf Phase 3:} The exposure analysis stage, in which the
  results of the multiple detectors are combined to improve the
  calibrations for the complete FPA images. 

  \item {\bf Phase 4:} The image combination stage, in which several
  different exposures of the same part of the sky are combined to
  produce high-quality difference and summed images.
 \end{itemize}

 \item {\bf Calibration Image Analysis} is required to generate the
 calibration images used in the science image analysis.  There are
 many different types of calibration images which must be produced,
 some of which are derived from other calibration images.  The type of
 calibration image is generated by its own analysis stage.  These
 include the construction of simple bias, dark, and flat-field stacked
 images, the generation of flat-field correction frames on the basis
 of stellar photometry, and the construction of sky-model and
 fringe-model images.  

 \item {\bf Reference Catalog Creation} is required by the IPP to
 generate improved astrometric and photometric reference catalogs on
 the basis of Pan-STARRS observations.

\end{enumerate}

Figure~\ref{stages} shows the flow of data between the various IPP
software subsystems and the different analysis stages.  The thick
lines represent the flow of pixel data, the thin lines represent the
flow of metadata and object data, and the grey lines represent the
flow of commands.  The hatched systems represent external Pan-STARRS
systems (OTIS, the Sky Server, the PSPS Object Database, the
Moving/Transient Object Pipeline, and other Client Science Pipelines.

The individual analysis stages are implemented as UNIX command-line
programs.  Each command performs the action of the stage on a single
quantum of data.  These analysis stages are built of lower-level
C-functions which may be wrapped in a higher-level programming
language such as Perl or Python.

As discussed above (section~\ref{IPP:IPPscheduler}), the decision to
execute a specific analysis stage for a specific dataset is made by
the IPP Scheduler, which sends the infomation to the IPP Controller.
The IPP Controller executes the analysis tasks constructed by the IPP
Scheduler for that stage on an appropriate machine and monitors the
success or failure of the job.

An important design decision which must be addressed by the IfA IPP
Team is the question of how the analysis stages determine
configuration and reference data needed by the analysis.  In one
scenario, it is always the responsibility of the low-level function to
perform the necessary query of the reference databases.  In an
alternative scenario, it is the responsibility of the scheduler to
extract that information and send it as part of the processing
command.

\paragraph{Science Image Analysis}

The Science Image analysis stages together represent the primary data
analysis performed by the IPP.  The science image analysis which is
performed by the IPP consists of:

\begin{itemize} 
\item detrending the images to remove the instrumental signature

\item astrometric and photometric calibration of the individual images

\item merging a collection of several images of the same portion of
the sky obtained over a short period of time to remove image defects
and gaps

\item subtracting the appropriate reference static-sky image

\item cleaning the image of any transients

\item adding the cleaned image to the static sky

\item object detection of images at specific stages
\end{itemize}

We have divided the analysis steps into four analysis stages, which we
call Phases 1 - 4.  Each of these analysis stages deals with a single
data unit.  

\paragraph{Phase 1 : image processing preparation}

The Phase 1 analysis stage is performed on each science exposure (each
complete FPA image) to calculate basic astrometric data needed by the
later stages.  Phase 1 uses the static (pre-determined) telescope
distortion model and a table of nominal OTA positions and rotations,
combined with the guide star pixel and celestial coordinates, to
determine the correct telescope bore-sight, field rotation and
magnification.  The guide star coordinates are loaded from the
Metadata database.  These calculations are performed by comparing the
observed guide star detector coodinates with the known astrometic
positions of these same stars as reported by an external astrometric
reference.  The accuracy of the resulting astrometric solution is
expected to be \tbr{1 arcsec} across the field, sufficient in later
stages to match the vast majority of astrometric reference stars with
their detections with minimal effort.

In some circumstances, science images may have no guide stars.  This
may occur in the Pan-STARRS system if the detectors are not run in OTA
mode, for example for short snapshot images.  This may also be the
case if the IPP is being run on non-Pan-STARRS data.  In such a
circumstance, the Phase 1 stage uses the provided boresight
coordinates, exposure time, and camera zero-point to predict the pixel
coordinates of known bright stars expected to be found on the
detectors.  It then extracts a large box ($\sim$ 30 $\times$
30\arcsec) around these locations and performs extremely basic object
detection to determine the detector coordinates of those bright stars
which are not saturated but which are significantly above the
background level.  By targetting known locations in the image files,
only a small amount of data will have to be read.

If the image has invalid coordinates or no detectable bright stars,
Phase 1 fails and reports a descriptive error.

Given the above astrometric solution, the Phase 1 analysis stage
constructs a table of the overlaps between the science image to be
processed and the static sky images that must be constructed.  This
table will be used to guide the processing of the static sky in Phase
4.  The overlaps should be generously calculated so that small errors
in astrometry at Phase 1 will not cause any valid static sky / science
image pairs to be missed because of the astrometric error at this
phase.  It is acceptable for a small number of invalid overlaps to be
identified as these will be excluded in Phase 4.  Static Sky cells
which do not have sufficient science image overlap \tbr{$< 5\%$} need
not be processed because the few new measured pixels do not add
significantly to the Static Sky.

Phase 1 is the image processing preparation stage.  The analysis is
performed on a complete FPA.  At the end of this analysis, the FPA is
ready to be analysed in detail in Phase 2.  The Phase 1 tasks and
functions are:

\begin{itemize}

\item Extract FPA guide stars to determine astrometry across the full FPA

\item If no guide stars are available, phase 1 must measure the pixel
  coordinates of known bright stars expected in the field from the
  image data.

\item The total number of stars and size of the bright-star
  acquisition box shall be a user-configurable parameter in the range
  20 - 250.

\item Calculate the Image cell / Sky cell overlaps for each image.
  Sky cells which do not have sufficient science image overlap $< 5\%$
  are excluded from the overlap table.

\end{itemize}

\paragraph{Phase 2 : image reduction}
\label{IPP:Phase2}

The Phase~2 analysis is the detrend stage, in which the images from
the detector are processed to remove instrumental signatures.  In
addition, basic object detection is performed along with improved
astrometric and photometric calibration.  In each step of the analysis
process, an image mask and noise map must be constructed and updated
when appropriate.  The following operations may be applied during the
Phase~2 processing:

\begin{enumerate}
\item Convolve detrend images with the OT kernel, if appropriate
\item Flag bad and saturated pixels
\item Bias correction via overscan subtraction
\item Dark
\item Trim object image to remove overscan and edges corrupted by OT
\item Correct for non-linearity
\item Cross-talk
\item Flat-field correction
\item Sky \& Fringe subtraction
\item Identify CRs
\item Find objects in the image
\item Model the PSF variations across the image
\item Make postage stamps of bright objects.
\end{enumerate}

Of the calibration steps, some may be skipped if they do not
contribute to an improved image.  The decision to apply or skip a
particular step is determined by the Phase 2 recipe, which may specify
exposure time or flux limit cutoffs for some of the steps.

\subparagraph{Convolve detrend images with the OT kernel}

Certain detrend images are convolved by the OT kernel, so that they
accurately represent the detrend images appropriate for the object
images, which have been shifted using OT.  The detrend images which
must be convolved include: the flat-field and the
high-spatial-frequency fringe images. The appropriate kernel for each
cell of an OTA must be determined from the guide star history,
extracted from the IPP Metadata Database\footnote{or image header}.
If the OT kernel is not available, the convolution is skipped, with a
warning.

\subparagraph{Flag bad and saturated pixels}

A static bad pixel mask is used to identify pixels which are known to
be bad in the camera.  This mask is then processed with the science
image. Bad pixels which are charge traps are grown by the extent of
the OT convolution kernel.  Bad pixels above a charge trap (i.e.\ bad
colums) must not be grown, since they were not affected by pixel
shifting, but only became bad at read-out.

Pixels which are saturated in the A/D converter, or with a signal
level at which the response is excessively non-linear, must also be
masked, and this area must be grown by an additional pixel to mask
excess charge spillover.

The bad pixel mask must be carried with the science images.  Different
bits must be set to identify different reasons for masking the pixel.

\subparagraph{Bias correction via overscan subtraction}

The image bias must be subtracted. Since different detectors behave in
different ways, several options for modelling the bias are available.
The bias is measured from the image overscan region.  The bias
subtraction method must be capable of subtracting a single constant
from the complete image, or to subtract a 1-D bias which varies as a
function along the overscan.  The function used to represent the
overscan region may be a spline or a chebychev polynomial derived from
the data values along the overscan.  The values used to determine both
the single constant or the inputs to the spline and polynomial fits
are derived from groups of pixels on the basis of one of several
statistics, including the sample and robust mean, median, and modes.
In the case of a single constant, all of the overscan pixel values are
used in the calculation of this statistic.  In the case of the 1-D
functional representation, the input values to the fit must represent
the coordinate along the overscan, with the statistic derived from the
pixels in the perpendicular direction at each location.  Sigma-clipping
on the input data values must be an option.

\subparagraph{Trim object image}

The image is trimmed to remove the non-imaging pixels, such as the
overscan and any pre-scan pixels, along with those pixels near the
edges that have been compromised due to OT operation.  The definition
of the imaging area of the detector is determined from the camera
configuration data or from the metadata associated with the image,
with the choice a user-configurable option.

\subparagraph{Correct for non-linearity}

If required, the object image (after bias correction) must be
corrected for the effects of non-linearity through a provided
polynomial fit to the pixel data values or a numeric lookup table as a
function of pixel flux.  The choice to apply the correction must be
set by the user.

\subparagraph{Flat-field correction}

The object image (after bias correction and non-linearity correction)
must be corrected for sensitivity variations as a function of
position, dividing by a flat-field image.  

\subparagraph{Sky \& Fringe subtraction}

After the science image has been flat-fielded, the flux contribution
of the sky (from both continuum emission and the line emission that
causes fringing) must be subtracted from the image.  The subtraction
needs to remove background (technically, foreground) variations which
are not celestial but generated in the atmosphere or by more localized
scattering.  This background should include the contribution from the
zodiacal light.  This background subtraction does not address the
artifacts generated by bright stars: bleeding columns, ghosts, or
other localized reflection effects.  This process also produces a
superbinned image of the background map which may be used as a
debugging diagnostic.

\subparagraph{Identify `cosmic rays'}

Charged particles in the detector frequently cause features which do
not have the morphology of astronomical objects.  In a well-sampled
image, these may be easily identified by the sharpness of the image.
In a near critically-sampled image, these `cosmic rays' may be
indistinguishable from stellar objects.  The IPP must have the
capability of making the morphological identification of cosmic rays
if the imaging data is sufficiently well sampled.  The identified
cosmic rays should be masked with a configurable growth factor so that
additional pixels beyond the detected pixels in the feature are also
masked.

\subparagraph{Find objects in the image}

After the image have been processed by the preceeding steps, the Phase
2 analysis performs a basic object detection analysis.  Objects on the
flat-fielded object image are found, and general parameters are
measured.  Object detection is performed at several stages by the IPP,
with different object parameters measured in each case.
Table~\ref{APdetections} gives a list of the different detection
stages and the object parameters measured for those stages.  For the
Phase 2 analysis, the object parameters are: the object centroid and
the position covarience matrix, the instrumental PSF magnitude and
error, local background level and error, a measurement of the
star-galaxy separation, and a measurement of the object shape
($\sigma_x, \sigma_y, \theta$).  The detection threshold must be
configurable, and be a function of the average background flux or the
image noise map.  Minimal object classification must be performed to
distinguish objects which are consistent with a single PSF, objects
which are inconsistent, and objects which are saturated.  The
resulting collection of detected objects are saved in the AP Database
along with the relevant image metadata (\ie filter, exposure time,
etc).  In addition, this process constructs a model of the
point-spread-function (PSF) as a function of position in the image.
This PSF model is saved as part of the image metadata.

\subparagraph{Astrometry}

The astrometric parameters determined in Phase 1 have an accuracy of 1
arcsec, sufficient to allow easy association between the newly
detected objects and many reference objects in the image.  In Phase 2,
the detected objects are matched with known astrometric reference
objects, using reference object coordinates which have been adjusted
for proper motion.  The matches are then used to improve the
astrometric parameters for the individual OTAs.  The OTA astrometric
parameters which are determined may include terms up to 3rd order in
position, though the terms which are actually fitted are
user-configurable.  The Cell astrometric parameters are not allowed to
vary at this stage.  The fit must be robust, rejecting outlier matches
(either stars with poorly determined proper motion or spurious
matches).  The resulting astrometric solution is consistent across the
OTA field to within \tbr{0.2 arcsec}.

\subparagraph{Postage Stamps}

The IPP must have the capability of extracting regions surrounding a
specified subset of objects from the flattened images.  These postage
stamp images must be saved for additional use by client science
pipelines.  The identification of these objects must be on the basis
of a set of rules applied to the object's magnitude and position.  The
postage stamps are not restricted in shape to simple rectangles, but
may represent more complex regions.  They are written the Image
Server.

%%

Phase 2 is the detrend stage, in which each detector is separately
processed to remove instrumental signatures.  The result of Phase 2 is
an image with high-quality astrometric and photometric calibrations, a
collection of objects detected in the image and characterized in a
rudimentary way (star / non-stellar), and a measurement of the PSF
across the detector.  

The tasks and functions of Phase 2 are as follows:

\begin{itemize}

\item Convolve the flat-field and high-spatial-frequency fringe images
  with the OT kernel.

\item Mask ghosts of bright stars which introduce residual feature
  more significant than 1\% of the background.

\item Bias subtract the image.

\item Correct each chip independently for non-linearity.

\item Flat-field correct the image.

\item Subtract a fit to the detector-dependent fringing pattern.

\item Subtract a fit to the low-spatial frequency sky background.

\item Identify `cosmic rays' on the basis of morphology.

\item Perform (positive) object detection on the processed images,
  down to a user-configured threshold, likely to be $\sim 20\sigma$.
  The detection threshold may optionally be a function of the average
  background flux or the local noise level.

\item Measure the following object parameters:

  \begin{itemize}
  \item object centroid and position errors.
  \item an extended object position ($x_g, y_g$).
  \item instrumental PSF magnitude and error.
  \item local background level and error.
  \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) of the object
    and their covariance matrix.
  \end{itemize}

\item Perform minimal object classification to distinguish objects
  which are consistent with a single PSF, objects which are
  inconsistently large, objects which are inconsistently small, and
  objects which are saturated.

\item Match the detected objects with known astrometric reference
  objects, including proper-motion compensation.

\item Fit the reference and detected object coordinates to determine
  astrometric parameters for the individual OTAs, including
  polynomials of the coordinates up to 3rd order (user-specified
  parameter).  The Cell astrometric parameters are not allowed to vary
  in the fit, which uses outlier rejection to determine a robust
  solution. 

\item Extract subrasters (`postage stamps') surrounding a
  user-specified list of coordinates from the flattened
  images, to be saved in the IPP Image Server.

\item measure the PSF variation as a function of detector position.

\end{itemize}

\paragraph{Phase 3 : exposure analysis}

The Phase 3 analysis stage works with the results from a complete FPA
obtained during Phase 2 to improve the photometric and astrometric
calibrations.  

Phase 3 uses the objects detected in Phase 2, matched with an
appropriate reference catalog, to determine the image photometric zero
point and zero-point variations across the field.  If zero-point
variations are significant, the zero-point variations are modeled with
a Chebychev polynomial correction of order 3 or less.  The complete
FPA image must be categorized as photometric or not on the basis of
the zero-point consistency, comparisons between the zero-point of the
image and recent longer-term (week or month long) measurements of the
zero-point, and the external indicators of photometricity.  In
addition, statistics of the transparency are measured and saved as
part of the related Metadata.

Phase 3 also uses the objects detected in Phase 2, matched with an
appropriate reference catalog, to determine improvements to the
astrometric solutions.  The improved solution is determined by fitting
a new distortion model appropriate to this image.  The resulting
astrometric accuracy is limited by the astrometric reference
catalog. (see Table~\ref{AstroRefs} below).

Phase 3 also uses the individual measurements of the background and
the superbinned background maps to generate an improved background map
over the entire FPA.  The large-scale background correction is
determined on the basis that the background should be smoothly varying
between different chips (OTAs).

Phase 3 also uses the individual chip models of the PSF variations to
model the global PSF variations across the field.  There will be
discontinuities at the chip boundaries due to charge diffusion and
chip displacements along the optical axis, but there will also be an
overlying trend due to the local coherence of atmospheric seeing
variations. 

%%%

The Phase 3 analysis uses the objects detected in Phase 2 and external
reference catalogs to determine improved photometric and astrometric
calibrations for the FPA as a whole, and to improve the measurement of
the PSF and sky variations across the field.  The Phase 3 tasks and
functions are as follows:

\begin{itemize}

\item Phase 3 uses the objects detected in Phase 2, matched with a
  user-specified reference photometry catalog, to determine the image
  photometric zero point and zero-point variations across the field.

\item If zero-point variations are significant ($> 0.01$ mag
  peak-to-peak), the zero-point variations are modeled with a
  polynomial correction of order 3 or less.

\item The photometric nature of the FPA image is categorized on the
  basis of the zero-point consistency, the transparency compared with
  recent long-term measurements in the filter, and the external
  indicators of photometricity.

\item Phase 3 uses the objects detected in Phase 2, matched with an
  appropriate astrometric reference catalog, to improve the distortion
  model used for the image.  The resulting astrometric accuracy is
  consistent across the field to 30 mas, and is limited by the
  astrometric reference catalog, (eg, 100 - 250 mas for
  USNO-B1.0).

\item The Phase 3 analysis modifies the background correction of Phase
  2 based on the full-field statistics to achieve an accuracy of 1\%
  of the background.

\end{itemize}

\paragraph{Phase 4 : image combination}
\label{IPP:Phase4}

Phase 4 is the image combination stage, in which multiple images of
the same portion of the sky are merged and confronted with the static
sky image.  Phase 4 operates on the smallest data unit of the static
sky, the sky cell, along with the associated pixels from a collection
of images which have been processed through phases 1--3.  The size and
exact representation of a static sky cell are yet to be determined.
The working concept is that the static sky cells contain roughly the
same number of pixels as an OTA (4k x 4k) and represent a portion of a
local tangent plane projection.  As mentioned above
(Section~\ref{IPP:ImageServer}), the pixel scale of the static sky is
planned to be 0.2\arcsec, somewhat smaller than the 0.3\arcsec\ raw
image pixel scale.

For each sky cell, the corresponding pixels are extracted from the
exposures being processed and mapped to the projection of the sky
cell. The pixels from the multiple input processed images are combined
into a single, cleaned image.  Outlier pixels may be optionally
rejected at this stage (optionally, since moving objects will be
rejected in images obtained over a wide range of times).  This image
is then confronted with the static sky cell data to produce a
difference image.  Residual objects in the difference image above a
threshold are detected and excised from the original cleaned image.
The remaining pixels are added to the existing static sky image.
Object detection must be performed on the difference and cleaned
images.

Objects in the difference image are detected and a specific set of
object parameters are measured from these detections.
Table~\ref{APdetections} gives a list of the different detection
stages and the object parameters measured for those stages.  For the
Phase 4 difference image (P4$\Delta$), the measured object parameters
consist of: the object centroid and the position covarience matrix,
the instrumental PSF magnitude and error, local background level and
error, a measurement of the star-galaxy separation, and a measurement
of the object shape ($\sigma_x, \sigma_y, \theta$).  The detection
threshold must be configurable, and be a function of the average
background flux or the image noise map.  Minimal object classification
must be performed to distinguish objects which are consistent with a
single PSF, objects which are inconsistent, and objects which are
saturated.  The resulting collection of detected objects are saved
along with the relevant image metadata (\ie filter, exposure time,
etc).

Objects in the cleaned, summed image are detected and a specific set
of object parameters are measured from these detections.
Table~\ref{APdetections} gives a list of the different detection
stages and the object parameters measured for those stages.  For the
Phase 4 summed image (P4$\Sigma$), the measured object parameters
consist of: the object centroid and the position covarience matrix,
the instrumental PSF magnitude and error, local background level and
error, a measurement of the star-galaxy separation, a measurement of
the object shape ($\sigma_x, \sigma_y, \theta$), the Petrosian radius,
magnitude, axis ratio, and angle; and the S\'ersic radius, magnitude, axis
ratio, angle, and parameter $\nu$.  The detection threshold must be
configurable, and be a function of the average background flux or the
image noise map.  Minimal object classification must be performed to
distinguish objects which are consistent with a single PSF, objects
which are inconsistent, and objects which are saturated.  The
resulting collection of detected objects are saved along with the
relevant image metadata (\ie filter, exposure time, etc).  In this
measurement, objects at known positions will also be measured even if
they have not been detected.

Objects which are detected in both of the Phase 4$\Sigma$ and Phase
4$\Delta$ images are saved to the AP Database, along with the relevant
image metadata (\ie filter, exposure time, etc).  In the process of
adding these objects to the database, the transients which are
correlated with previous detections of an object (and those which are
not) will automatically be determined.  An independent process will
query the AP Database for such transient objects of interest which are
to be sent, along with their associated metadata, to the MOPS and
other science client pipelines.  This step must be performed at least
once per night.

It is essential that the static sky image (which may have been
painstakingly accumulated over many months) not be corrupted by adding
in transient sources, or data that is of suspect quality (due, e.g.,
to an error upstream in the processing).

Object analysis of the static sky images is {\em not} a part of the
Phase 4 analysis.  This processing is envisioned to take place
relatively infrequently (perhaps weekly or even monthly) and is
scheduled as a separate analysis task, probably run during the day at
a time when the computing infrastructure is not under significant load.

%% 

Phase 4 is the image combination stage, in which multiple images of
the same portion of the sky are merged and confronted with the static
sky image.  The Phase 4 tasks and functions are as follows:

\begin{itemize}

\item The Phase 4 analysis determines the corresponding set of image
  pixels for a given sky cell.

\item These pixels are extracted from the input images, using the
  astrometric information for each OTA and Cell to determine the exact
  overlaps.

\item The Phase 4 analysis skips any sky cells with fewer than 5\% of
  their pixels overlapping the input images.

\item Pixels which have been extracted from the input images are
  geometrically warped to match the corresponding pixels in the sky
  image.  This transformation is based on the measured astrometric
  solution for the input images relative to the reference catalog used
  to generate the static sky image.  The warping may use a
  locally-linear astrometric solution to speed the processing.
  
\item Phase 4 determines the appropriate photometry scaling factors
  needed to combine the images photometrically.

\item When multiple images are combined, the group of input pixels
  which contribute to an output pixel are examined and pixels from the
  group of images which are inconsistent with the ensemble (by an
  amount defined by the user-configurable parameters) are identified
  and flagged, though this outlier rejection shall be performed
  optionally.

\item The resulting collection of pixels is used to construct a single
  output image, cleaned of the outliers.

\item The cleaned, combined image is PSF matched with the static sky
  image.

\item The static sky image is subtracted from the stacked, cleaned
  image, resulting in the difference image (P4$\Delta$ image)

\item The Phase 4 analysis performs object detection on the difference
  images.  All objects in the difference image above a user-configured
  signficance threshold are detected, including both positive and
  negative flux objects.  The detection threshold may optionally be a
  function of the average background flux or the local noise
  level.  The likely significance threshold is $\sim 3\sigma$.

\item P4$\Delta$ objects have the following object parameters
  measured:
  \begin{itemize}
  \item object centroid and position errors.
  \item instrumental PSF magnitude and error.
  \item local background level and error.
  \item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$.
  \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix.
  \end{itemize}

\item Minimal object classification is performed to distinguish
  objects which are consistent with a single PSF, objects which are
  inconsistent, and objects which are saturated.

\item The pixels belonging to variable sources are masked in the
  input image.

\item A new, cleaned image is constructed from the masked input images
  (P4$\Sigma$ image)

\item Object detection is performed on the cleaned, summed image to a
  user-configured significance threshold ($\sim 7\sigma$).  Only
  positive flux object are considered.  The detection threshold may
  optionally be a function of the average background flux or the local
  noise level.

\item P4$\Sigma$ objects have the following object parameters
  measured:
  \begin{itemize}
  \item object centroid and position errors.
  \item an extended object position ($x_g, y_g$).
  \item instrumental PSF magnitude and error.
  \item local background level and error.
  \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their
    covariance matrix.
  \item the Petrosian radius, magnitude, axis ratio, and angle.
  \item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$.
  \end{itemize}

\item Minimal object classification is performed to distinguish
  objects which are consistent with a single PSF, objects which are
  inconsistent, and objects which are saturated.

\item Before the image is added to the static sky, it must pass Q/A
  tests:
  \begin{itemize}
  \item the measured photometry scatter for the image must be less
      than \tbr{1\%}.

  \item the measured astrometry scatter for the image must be less
  than \tbr{30 mas}.
  \end{itemize}

\item The final, cleaned input image is added to the static sky so
  that an incrementally-deeper static sky image may be
  made.
\end{itemize}

\paragraph{Static Sky Analysis}
\label{IPP:StaticSky} 

The IPP is responsible for performing object detection and analysis on
the static sky.  This analysis is performed continuously (every day or
week) on those portions of the sky within 15\degree\ of the sun.  In
this analysis, the object measurement is much more detailed than those
performed in the real-time analysis.  The currently envisioned
parameters to be measured for every object are listed in
Table~\ref{APdetections}.  The parameters include the object centroid
and the position covarience matrix, the instrumental PSF magnitude and
error, local background level and error, a measurement of the
star-galaxy separation, a measurement of the object shape ($\sigma_x,
\sigma_y, \theta$), the Petrosian radius, magnitude, axis ratio, and
angle; the S\'ersic radius, magnitude, axis ratio, angle, and
parameter $\nu$, and a collection of annular aperture flux
measurements, all of which are also measured for the P4$\Sigma$
images.  In addition, the galaxy-shape parameters $Gamma_1 \&
\Gamma_2$, along with the complete `polarization' terms are measured,
as well as a collection of annular aperture flux and variance
measurements.  Another unique feature of the static sky analysis is
that the object detection may be performed simultaneously on all
filters, in which case the locations and other parameters may be more
strongly constrained by simultaneously fitting between all bands.  The
analysis to be performed may be substantially more complex than the
real-time analysis because of the relatively low data rate.  Instead
of needed to process thousands of images per night ($\sim 350$Mpix per
second), it is only necessary to process the complete sky in a year,
or an average rate of $\sim$2 Mpix per second, or $< 1$\% of the
object analysis in the other analysis stages.

\paragraph{Calibration Stages}
\label{IPP:mkcal}

The Calibration Analysis Stages construct calibrations from the
relevant input data.  Some of these combine multiple raw input images
together, after processing, to create a high-quality high-signal
master calibration image.  Some of the calibrations are used to
correct other calibrations.  Each of the calibration stages must also
provide the tools to test the quality of the input data against
existing master calibration data and to test the consistency of
multiple measurements of the calibration.
 
The Calibration analysis stages may be performed on whatever
timescales are appropriate and necessary to maintain the quality and
relevance of the calibration images.  Below, we list the specific
calibration data which must be constructed in the calibration analysis
stages.  

The IPP must generate basic calibration images using the raw bias,
dark, and flat-field (dome or twilight) images obtained by the
telescope as the input.  The analysis of these images requires
relatively simple stacking of the input set of images.  Outlier
rejection, both of complete input images as well as pixels within the
input stack, must be performed.  In addition, each type of image
requires an appropriate normalization which may depend on the data
levels in other detectors in the input set.  Each of these calibration
stages must be able to determine from the input stack if the relevant
calibration image needs to be updated and perform an initial test to
see which input images are consistent and valid.

\subparagraph{Bias Images}

Bias images may be needed to correct for structure in the bias.  The
IPP must have the capability of constructing a master bias image from
a stack of raw bias frames.  The input bias images, representing
offsets from the overscan level, are processed by subtracting the
overscan, including 1D structure if needed.  

The master bias frame construction uses outlier image and outlier
pixel rejection to construct a single high-quality bias frame.  The
statistic used to determine pixel values from the input stack can be
set by the user to be one of the following: the sample mean, median,
and mode, robust mean, median, and mode, and the clipped mean and
median.  Testing of the input images consists of constructing residual
images, in which the master bias is applied to the input images.
These images may be included or excluded from an additional iteration
of the stack on the basis of their pixel-to-pixel statistics.

\subparagraph{Dark Images}

Dark images may be needed to correct for structure in the dark
current.  The IPP must have the capability of constructing a master
dark image from a stack of raw dark frames.  The input dark images are
first corrected for the bias using whatever method is appropriate for
the science images.  Master dark frames depend on their exposure time.
As such, the input dark frames must have a limited range of exposure
times, and the output dark frame includes the exposure time as part of
its associated metadata.  

The master dark frame construction uses outlier image and outlier
pixel rejection to construct a single high-quality dark frame.  The
statistic used to determine pixel values from the input stack can be
set by the user to be one of the following: the sample mean, median,
and mode, robust mean, median, and mode, and the clipped mean and
median.  Testing of the input images consists of constructing residual
images, in which the master dark image is applied to the input images.
These images may be included or excluded from an additional iteration
of the stack on the basis of their pixel-to-pixel statistics.  A
collection of master dark frames with a range of exposure times are
used to determine the scaling of the dark frame as a function of
exposure time.

\subparagraph{On-Off Dark Images for Light Leaks}

A type of image which may be necessary for calibrations will be pairs
of images taken at night with the shutter closed with and without the
dome shutter closed.  Such a pair of images can be used to determine
any light-leak in the camera which may contribute additional flux
across the mosaic.

\subparagraph{Flat-Field Images}

Master flat-field images must be constructed from a collection of
input flat-field images.  The input flat-field images may be obtained
from any of the standard sources: the dome, the twilight sky, and the
night-time sky.  The choice of flat-field input image must be
determined experimentally from observations during the commissioning
phase of the telescope.  The IPP flat-field construction system must
be capable of handling any of these sources.  

An appropriate set of input images is selected on the basis of their
flux levels, time of observations, and the observing conditions.  The
input flat-field images are processed (bias and dark corrected if
needed) and the resulting images are stacked.  The master flat-field
construction uses image and pixel outlier rejection to construct a
single high-quality master flat-field frame.  The statistic used to
determine pixel values from the input stack can be set by the user to
be one of the following: the sample mean, median, and mode, robust
mean, median, and mode, and the clipped mean and median.  Testing of
the input images consists of constructing residual images, in which
the master flat-field image is applied to the input images.  These
images may be included or excluded from an additional iteration of the
stack on the basis of their pixel-to-pixel statistics.

\subparagraph{Mask Images}

Preliminary bad-pixel mask images are generated on the basis of
comparison between raw flat-field images and a cleaned, stacked
master.  The mask creation system accepts a collection of flat-field
images and identifies pixels which are consistently poorly flattened.
Pixels which are under-responsive are also identified as pixels to be
masked.  

\subparagraph{Sky \& Fringe Frames}

Fringe-correction frames must be generated to remove the fringe
pattern caused by thin-film interference in the top layers of CCDs,
particularly in the redder passbands.  Fringe correction frames may be
constructed on the basis of observations of the night-sky in the
appropriate filters or on the basis of dome fringe lamp observations.
The choice of the appropriate source will be determined experimentally
on the basis of data obtained during the commissioning phase.  The IPP
must be capable of handing either source.  The images are first
flattened to remove the pixel-to-pixel sensitivity variations of the
detector.  The combination of multiple input fringe frames may not be
simply stacked since the amplitude of the fringe pattern varies
independently of other variations in the image.  The amplitude of the
fringe pattern in the input frames is measured and the images scaled
to normalize the fringe amplitude to a consistent range (-1 to +1) for
all input images before they are combined with one of the standard
combination statistics (mean, median, mode, etc).  The quality of the
input frames is tested by flattening the input image and applying the
master fringe-frame.  The resulting residual image statistics are used
to select or exclude specific input images.

\subparagraph{Shutter Correction Map}

Shutter correction map images may be generated based on the timing
measurements of the shutter itself, or on the basis of dome-flat
images of decreasing exposure times down to the shortest available
exposures.

\subparagraph{Low-k Sky Models}

Large-scale background structure in images which is not caused by
thin-film interference must also be detected and corrected.  Models of
this background structure may be a necessary input to the correction
proceedure.  The IPP must have the capability of generating image
models of the large-scale structure patterns observed with the
telescope

\subparagraph{Flat-Field Correction Frame}

Flat-field images, whether constructed from the dome, twilight, or
night-sky images, do not perfectly correct the detector response in a
consistent fashion across the full field of the camera.  The IPP must
have the capability of generating flat-field photometric correction
frames on the basis of the measured photometry of objects which are
moved to a variety of locations on the detector in a sequence of
images.  The flat-field correction frames analysis stage makes use of
targetted observations following a specified dither pattern, and
extracts the photometered objects from the AP Database to determine
the necessary photometric corrections.  The resulting image is applied
to the master flat-field image.  Testing of the correction is
performed by applying the correction to the basic master flat-field
image, applying that flat-field image to the dithered photometry
observations, and performing the object detections.  Comparion of the
photometry of individual stars at different locations on the mosaic
will demonstrate the consistency of the flat-field image.

\subparagraph{Non-Linearity Correction}

The IPP must have the capability of constructing a correction for
non-linearity in the detectors.  These frames are constructed from
exposures of a uniform source with a range of exposure times.  The
non-linearity correction frames provide polynomial correction
coefficients or a lookup table describing the correction.  There is
likely to be a single non-linear correction for each OTA detector, or
potentially for each Cell.  The IPP must handle these two cases.

\paragraph{Reference Catalog Creation}

One of the primary goals inital goals of Pan-STARRS is the creation of
photometric and astrometric reference catalogs for the general
community and for additional Pan-STARRS calibration.  This
internally-generated reference catalog is necessary to achieve the
photometry and astrometry goals set for the project.  The generation
of these catalogs is inherently a research project, and will require
human control and intervention.  The IPP will be required to provide
the data access, manipulation and visualization tools needed to
construct these reference catalogs and to assess their quality.  In
this section, we discuss the tools needed for this effort.

\subsubsection{Bias Image Creation}

The Bias calibration stage constructs a master bias image from a
collection of raw bias images.  The tasks and functions include:

\begin{itemize}

\item The Bias calibration stage corrects the input images based on
  the overscan region, determined from either the header or from
  metadata.

\item The Bias calibration stage combines the input images using the
  statistic specified by the user, selected from one of the following:
  sample mean, median, and mode, robust mean, median, and mode, and
  the clipped mean and median.

\item The Bias calibration stage construct residual images, in which
  the master bias is applied to the input images.

\item Outlier residual images, those for which the residual bias and
  variance in the bias image are excessive, are excluded from the
  input image stack and the bias image reconstructed.
\end{itemize}

\subsubsection{Dark Image Creation}

The Dark calibration stage shall construct a master dark image from a
  collection of raw dark images.  The tasks and functions include:

\begin{itemize}

\item The Dark calibration stage raises an error if the input images
  have exposure times which deviate by more than 2\%.

\item The Dark calibration stage corrects the input dark images for
  the bias.

\item The Dark calibration stage combines the input images using the
  statistic specified by the user, selected from one of the following:
  sample mean, median, and mode, robust mean, median, and mode, and
  the clipped mean and median.

\item The Dark calibration stage constructs residual images, in which
  the master dark is applied to the input images.

\item Outlier residual images, those for which the residual level and
  variance are excessive, are excluded from the input image stack and
  the dark image reconstructed.
\end{itemize}

\subsubsection{Flat-field Image Creation}

The Flat-field calibration stage constructs a master flat-field image
from a collection of raw flat-field images.  The tasks and functions
include:

\begin{itemize}

\item The Flat-field calibration stage accepts a group of images from
  one of the following flat-field sources: dome, twilight,
  night-sky.

\item The flat-field calibration stage raises an error if the
  input images in a single stack used more than one of the above
  flat-field sources or multiple filters.

\item The Flat-field calibration stage corrects the input flat-field
  images for the bias and dark.

\item The Flat-field calibration stage combines the input images using
  the statistic specified by the user, selected from one of the
  following: sample mean, median, and mode, robust mean, median, and
  mode, and the clipped mean and median.

\item The Flat-field calibration stage constructs residual images, in
  which the master flat-field is applied to the input images.

\item Outlier residual images, those for which the residual level and
  variance are excessive ($> 0.1$\%, or 1.02 times the Poisson limit
  of the flat-field image), are excluded from the input image stack
  and the flat-field image reconstructed.
\end{itemize}

\subsubsection{Mask Image Creation}

The Mask calibration stage constructs a bad-pixel mask from a stack of
raw flat-field images and a master flat-field image.  The tasks and
functions include:

\begin{itemize}

\item The Mask calibration stage masks the pixels which are
  inconsistent in the input flats by more than 1\%, given sufficient
  signal-to-noise in the input flats.

\item The Mask calibration stage mask the pixels which are
  consistently low or high in the input flats by more than a factor of
  3 beyond the typical pixel.

\item The Mask calibration stage masks the pixels identified in a
  table of bad pixels generated externally to the calibration stage.

\item The Mask calibration stage uses multiple bit values to identify
  the different types of masked pixels.

\item The Mask calibration stage raises an error if the input images
  generate too many bad pixels in the mask.
\end{itemize}

\subsubsection{Fringe-frame Creation}

The Fringe-frame Creation calibration stage constructs a master fringe
frame from a stack of raw night-time sky images or from a stack of
dome fringe frames.  The tasks and functions include:

\begin{itemize}

\item The Fringe-frame Creation calibration stage raises an error if
  the input stack consists is images generated with more than one type
  of fringe source or with multiple filters.

\item The Fringe-frame Creation calibration stage flattens the input
  images to remove the pixel-to-pixel sensitivity variations of the
  detector.

\item The Fringe-frame Creation calibration stage measures the fringe
  amplitude on the input fringe images.

\item The Fringe-frame Creation calibration stage scales the input
  fringe images based on the fringe amplitude.

\item The Fringe-frame Creation calibration stage combines the scaled
  input images using the statistic specified by the user, selected
  from one of the following: sample mean, median, and mode, robust
  mean, median, and mode, and the clipped mean and median.

\item The Fringe-frame Creation calibration stage constructs residual
  images, in which the master fringe image is applied to the input
  images, along with all necessary preceding calibration images.

\item The Fringe-frame Creation calibration stage measures the
  residual fringe amplitude on the residual images.
\end{itemize}

\subsubsection{Low-spatial-frequency Sky Models}

The Sky Model calibration stage constructs a sky model image set from
a stack of raw night-time sky images.

\subsubsection{Flat-field correction Frame Creation}

The Flat-field correction calibration stage constructs a flat-field
correction image from dithered observations of a stellar field.  The
tasks and functions include:

\begin{itemize}

\item The Flat-field correction calibration stage constructs a
  flat-field correction image from dithered observations of a stellar
  field.

\item The Flat-field correction calibration stage constructs a
  flat-field correction image for each filter and camera
  independently.

\item The Flat-field correction calibration stage constructs a
  correction image which makes the photometry of multiple observations
  of the same stellar source consistent at different locations in the
  focal plane.

\item The Flat-field correction calibration stage constructs corrected
  flat-field images using the measured correction.

\item The Flat-field correction calibration stage determines the
  consistency of the corrected flat-field images using the dithered
  stellar field observations flattened with the corrected flat-field
  image.
\end{itemize}

\subsubsection{Non-linearity correction}

The Non-linear correction calibration stage constructs a correction
model for low-level non-linearity effects in the detector.  The tasks
and functions include:

\begin{itemize}

\item The Non-linear correction calibration stage constructs a
  non-linear correction from a collection of images of a constant
  source with varying exposure times.

\item The Non-linear correction calibration stage construct a
  non-linear correction which linearizes the detector fluxes
  $<0.5\%$.

\item The Non-linear correction calibration stage determines the
  saturation regime, in which the non-linear correction is no longer
  consistent to $<0.5\%$.
\end{itemize}

\subsubsection{Telescope Astrometry Parameters}

\begin{itemize}
\item The IPP Calibration system constructs static models of the
  telescope astrometry parameters (e.g., distortion, detector warps)
  once per week.

\item The IPP Calibration system constructs static models of the
  telescope astrometry parameters (e.g., distortion, detector warps)
  with an accuracy to produce astrometry consistent to 30
  milliarcsec.

\item The IPP Calibration system monitors changes in the telescope
  astrometry parameters and issue a warning if the parameters change
  by more than 2\%.
\end{itemize}

\subsubsection{Zero-Point Monitoring}

The IPP Calibration system determines telescope filter and camera
zero-points on a nightly basis with an accuracy sufficient to
determine photometry in the native filter systems to 5 millimags.

\subparagraph{Astrometry Reference Creation}

\begin{table}
\begin{center}
\caption{Astrometric Reference Catalogs\label{AstroRefs}}
\begin{tabular}{lrrrrl}
\hline
\hline
Name       & scatter limit   & proper   & depth      & Nstars     & filters \\
           & (milliarcsec)   & motion   &(mag)       & (millions) &         \\
\hline
Hipparcos  &   1             & 2        &  7.3       &    0.1     & {\em V}       \\ 
Tycho2	   &  10             & 1        & 11.5       &    2.5     & {\em B,V}     \\ 
UCAC-2     &  20             & 1        & 16.0       &   48.0     & {\em R}       \\ 
USNO-A2.0  & 250             & N/A      & 19.0       &  526.2     & {\em B,R}     \\ 
USNO-B1.0  & 200             & \tbr{20} & 21.0       & 1042.6     & {\em B,R}     \\ 
2MASS	   &  70             & N/A      & \tbr{15.0} &  470.0     & {\em J,H,K}   \\ 
\hline
\end{tabular}
\end{center}
\end{table}

The existing astrometric reference catalogs are known to have
limitations in accuracy as noted in Table~\ref{AstroRefs}.  The
internal accuracy of the Pan-STARRS dataset can potentially be much
higher than the external reference catalogs.  The IPP must have the
capability of generating an astrometric reference on the basis of the
observations obtained by the AP survey.  The IPP must provide the
analysis tools needed to generate the master astrometric reference
catalog.  Much of the required functionality is covered by the AP
Database.

The two basic, necessary ingredients for the construction of the
Astrometric Reference Catalog are: the observed coordinates of stars
and the existing astrometric reference catalogs.
Table~\ref{AstroRefs} lists a subset of the reference catalogs which
we will use at different stages in the analysis process, along with
notes about their accuracy.

These catalogs must be available and accessible to the AP Database.
It is necessary to have the tools to convert the reference catalog
object coordinates to all of the possible coordinate frames of
relevance in the telescope and camera system, including:
\begin{itemize}
\item Catalog to mean positions
\item Mean to apparent positions
\item Apparent positions + pointing to arbitrary tangent plane coordinates
\item Apparent positions + pointing to focal plane coordinates
\item focal plane to specific detector (OTA)
\item specific detector to detector cell
\end{itemize}

In addition to the reference catalogs, it will be necessary to
determine and have available for the analysis system a variety of
approximate calibration data, including the telescope and camera
optical distortion models and the variation of the image PSF across
the camera field, as a function of color.

The other necessary ingredient in the astrometry reference creation is
the observation of stars by Pan-STARRS.  The object detections are
produced by several of the analysis stages discussed in the Science Image
Analysis section.  The likely measurement of relevance to the
astrometric reference catalog is the object extraction for the
individual, detrended images (section~\ref{IPP:Phase2}).  The detected
objects must be matched against the reference catalogs, and it must be
possible to determine fit coefficients as a function of position
alone, or with combinations of magnitude or color.  The fitting method
must include robust outlier rejection.  It is also necessary to have
information about the objects which are detected in the catalog, but
not the science image or vice-versa, as well as an assessment of the
centroiding errors for each object.  It must be possible to plot the
fit residuals against a wide variety of parameters, including the
object positions, magnitudes, colors, etc, and to make subset
selections of the objects on the basis of these parameters.  .

An alternative measurement of the stellar positions is derived from
the guide stars, which are much brighter than the typical saturated
stars.  It must be possible to compare the coordinates of the guide
stars with the coordinates of the other stars in the image.  It must
also be possible to perform the various fitting steps for the guide
stars rather than for the normal image data.

\subparagraph{Photometry Reference Creation}

\begin{table}
\begin{center}
\caption{Photometric Reference Catalogs\label{PhotoRefs}}
\begin{tabular}{lrrr}
\hline
\hline
Name       & scatter  & depth & filters \\
           & mmag     & mag   &         \\
\hline
SDSS       & 15       & 16    & {\em u,g,r,i,z} \\
CFHT-LS    & \tbr{10} & 18    & {\em u,g,r,i,z} \\
Landolt    & 10-20    & 15    & {\em U,B,V,R,I} \\
\hline
\end{tabular}
\end{center}
\end{table}

The IPP must provide the analysis tools needed to generate a master
photometric reference catalog.  The tools needed for generation of the
photometric reference catalogs are similar in essence to those used
for the astrometric reference catalog.  It is necessary to confront
the observed objects against the existing reference catalogs to
determine the necessary calibrations.  Again, much of the required
functionality is covered by the AP Database.  

The necessary ingredients for the construction of the Photometric
Reference Catalog are: the observed magnitudes of stars and the
existing photometric reference catalogs.  An internally consistent
magnitude system will be generated as the primary reference catalog.
In addition, comparison of these magnitudes with the reference
magnitudes will allow for the determination of color transformations
and calibrated magnitudes in the reference system.
Table~\ref{PhotoRefs} lists a variety of reference catalogs which may
be used in the process.  These catalogs must be available and
accessible to the AP Database.

The other necessary ingredient in the photometry solution is the
observation of stars with Pan-STARRS.  The photometry is determined by
several of the analysis stages discussed in the Science Image Analysis
section.  The likely measurement of relevance to the photometric
reference catalog is the object extraction for the individual,
detrended images (section~\ref{IPP:Phase2}).  It is necessary to have
the tools to convert between different photometric systems, including:
\begin{itemize}
\item instrumental to nominal detector magnitude
\item nominal detector magnitude to average filter system
\item average filter system to reference photometry system
\end{itemize}
These transformations are based on a set of measured coefficients for
the color and airmass dependency of the measurement.  In addition to
these types of transformations, it is necessary to have the ability to
measure and apply relative photometry corrections.  

The detected objects must be matched against the reference catalogs,
and it must be possible to determine fit coefficients as a function of
airmass, magnitude, color and detector coordinates, or with
combinations of the above.  The fitting method must include robust
outlier rejection.  It is also necessary to perform exclusions on the
basis of object locations, instrumental magnitudes, observed and
reference errors, and time of the observations. It must be possible to
plot the fit residuals against a wide variety of parameters, including
the object positions, magnitudes, colors, etc, and to select a subset
of the objects on the basis of these parameters.  It will likely be
necessary to maintain individual color transformations for each
detector and filter combination to a single internal system for each
filter.

An alternative measurement of the stellar photometry is derived from
the guide stars, which are much brighter than the typical saturated
stars.  It must be possible to relate the magnitudes of the guide
stars with the magnitudes of the other stars in the image.  It must
also be possible to perform the above fitting steps for the guide
stars rather than for the normal image data.

\subsubsection{Modules}

In order to encapsulate functionality, the analysis stages are
constructed of a sequence of steps.  The analysis stages consist of
scripts in a high-level language, likely to be either Python or Perl,
which executes a sequence of C-level functions.  The C-level functions
called by the script are called {\em modules} and represent basic data
analysis operations.

\subsubsection{Pan-STARRS IPP Library}

In order to facilitate testing and development, and to encourage
flexibility, the IPP is built in a layered fashion.  The lowest level
functions are written in C and collected together into a Pan-STARRS
library, \code{PSLib}.

The Pan-STARRS Data Library consists of C structures describing the
basic data types needed by the IPP and C functions which perform the
basic data manipulation operations.  The library is organized into
four topics: System Utilities, Basic Data Collections, Data
Manipulation, and Astronomy-Specific Functions.  The Modules are
constructed using these low-level Library functions as needed.

\subsection{Summary of Derived Requirements}

\begin{enumerate}

\item The IPP Image Server shall accept raw images from the summit at
 a sustained rate of 1 exposure (4 FPAs or 8~GB) per \tbr{40 seconds}.
\label{IPP:DeReq:1}

\item The IPP Metadata Database shall accept metadata from the summit
 at a sustained rate of \tbr{1 MB per 40 second}.
\label{IPP:DeReq:2}

\item The IPP Calibration Analysis shall produce master calibration
 images from the raw calibration images in less \tbr{2 hours}.
\label{IPP:DeReq:3}

\item Master calibration images shall not introduce systematic
 uncertainties in the photometry greater than \tbr{0.2\%}.
\label{IPP:DeReq:4}

\item The IPP Science Analysis shall pre-process the science images
  with the master calibration images at a sustained rate of 1 exposure
  per \tbr{40 seconds}.
\label{IPP:DeReq:5}

\item The IPP Science Analysis shall merge multiple pre-processed
 science images into stacked images with corresponding signal-to-noise
 maps at a sustained rate of 1 exposure per \tbr{40 seconds}.
\label{IPP:DeReq:6}

\item The IPP Science Analysis shall excise pixels from the input
 images which are outliers for the ensemble of corresponding pixels
 with an efficiency of $> 99$\%.
\label{IPP:DeReq:7}

\item The IPP Science Analysis shall merge the cleaned images into the
 static sky image, and update the corresponding exposure (S/N) maps,
 at a sustained rate of 1 exposure per \tbr{40 seconds}.
\label{IPP:DeReq:8}

\item The IPP Science Analysis shall detect and measure parameters of
objects on the pre-processed science images.
\label{IPP:DeReq:9}

\item The IPP Science Analysis shall detect and measure parameters of
objects on the stacked science images.
\label{IPP:DeReq:10}

\item The IPP Science Analysis shall detect and measure parameters of
objects on the difference images.
\label{IPP:DeReq:11}

\item The IPP Science Analysis shall detect and measure parameters of
objects on the static sky images.
\label{IPP:DeReq:12}

\item The IPP Science Analysis shall determine astrometry of the
 detected objects relative to an external astrometric reference with
 an accuracy of \tbr{750 mas} (for bright objects) in the
 Commissioning phase of the telescope.
\label{IPP:DeReq:13}

\item The IPP Science Analysis shall determine astrometry of the
 detected objects relative to an external astrometric reference with
 an accuracy of \tbr{250 mas} (for bright objects) during the
 construction of the Pan-STARRS Astrometric Reference Catalog.
\label{IPP:DeReq:14}

\item The IPP Science Analysis shall determine astrometry of the
 detected objects relative to the Pan-STARRS Astrometric Reference
 with an accuracy of \tbr{100 mas} (for bright objects) during normal
 operations.
\label{IPP:DeReq:15}

\item The IPP Science Analysis shall determine photometry of the
 detected objects within an internal photometric system with scatter
 less than \tbr{25 millimags} (for bright objects) during the
 Commissioning phase of the telescope in photometric weather.
\label{IPP:DeReq:16}

\item The IPP Science Analysis shall determine photometry of the
 detected objects within an internal photometric system with scatter
 less than \tbr{10 millimags} (for bright objects) during the
 construction of the Pan-STARRS Photometric Reference Catalog in
 photometric weather.
\label{IPP:DeReq:17}

\item The IPP Science Analysis shall determine photometry of the
 detected objects within an internal photometric system with scatter
 less than \tbr{5 millimags} (for bright objects) during normal
 operations in photometric weather.
\label{IPP:DeReq:18}

\item The IPP Science Analysis shall determine photometry of the
 detected objects in an external photometric system with scatter less
 than \tbr{10 millimags} (for bright objects) during normal operations
 in photometric weather.
\label{IPP:DeReq:19}

\item The IPP Reference Creation System shall produce an astrometric
  reference catalog from the extracted objects within 6 months of the
  end of the AP Survey.
\label{IPP:DeReq:20}

\item The IPP Reference Creation System shall produce an astrometric
  reference catalog with an absolute accuracy of \tbr{100 mas} and a
  local relative accuracy of \tbr{30 mas} for bright objects.
\label{IPP:DeReq:21}

\item The IPP Reference Creation System shall produce an astrometric
  reference catalog with proper motions measurements for
  non-solar-system objects with an accuracy of \tbr{20 mas / year} for
  unsaturated, bright stars.
\label{IPP:DeReq:22}

\item The IPP Reference Creation System shall produce a photometric
  reference catalog from the extracted point-source objects within 6
  months of the end of the AP Survey.
\label{IPP:DeReq:23}

\item The IPP Reference Creation System shall produce a photometric
  reference catalog with a consistency across the sky of \tbr{5
  millimag}.
\label{IPP:DeReq:24}

\item The IPP Reference Creation System shall produce a photometric
  reference catalog with an absolute calibration to the external
  system (defined by \tbr{SDSS} and the CFHT Legacy Survey Standards)
  with an accuracy of \tbr{10 millimag} (for bright objects).
\label{IPP:DeReq:25}

\item The IPP shall publish the static sky images to the Pan-STARRS
PSPS on a time-scale of \tbr{6 month}.
\label{IPP:DeReq:26}

\item The IPP shall publish the detected objects to the Pan-STARRS
PSPS on a time-scale of \tbr{1 month}.
\label{IPP:DeReq:27}

\item The IPP shall publish the IPP and OTIS metadata to the
Pan-STARRS PSPS on a time-scale of \tbr{1 week}.
\label{IPP:DeReq:28}

\item The IPP shall provide to the MOPS subsystem the detected
 single-occurrence transient objects \tbr{by the end of every night}.
\label{IPP:DeReq:29a}

\item The IPP shall provide to the MOPS subsystem the metadata
appropriate to the images from which single-occurrence transient
objects were detected \tbr{by the end of every night}.
\label{IPP:DeReq:29b}

\item The IPP shall provide to external Pan-STARRS clients the
  detected objects within \tbr{5 minute} after the image is obtained.
\label{IPP:DeReq:29c}

\item The IPP shall store the raw images for a period of \tbr{1 month}.
\label{IPP:DeReq:30}

\item The IPP shall store the detected objects for a minimum of \tbr{1 year}.
\label{IPP:DeReq:31}

\item The IPP shall store the metadata for the lifetime of the project.
\label{IPP:DeReq:32}
\end{enumerate}

\subsection{Internal Interfaces}

The IPP has internal interfaces between several of the architectural
components and between the architectural components and the analysis
stages.

\begin{enumerate}

\item IPP Scheduler - IPP Controller.  The IPP Scheduler must
send to the IPP Controller information about the tasks to be
performed and must receive from the IPP Controller descriptions of the
success or failure of these tasks.

\item IPP Scheduler - Metadata DB.  The IPP Scheduler must query the
Metadata DB to determine an appropriate course of action.  The IPP
Scheduler must send result and status information to the Metadata DB.

\item IPP Controller - Analysis Tasks.  The IPP Controller must
initiate the Analysis Tasks and monitor their output and exit status.

\item Analysis Tasks - Metadata DB.  The Analysis Tasks must be able
to query the Metadata DB as needed to extract metadata needed for a
given task.  The Analysis Tasks must be able to send results and
updates to the Metadata DB.

\item Analysis Tasks - Image Server.  The Analysis Tasks must be able
to extract relevant images from the Image Server.  The Analysis Tasks
must be able to send output images to the Image Server.

\item Analysis Tasks - AP DB.  The Analysis Tasks must be able
to extract information related to specific objects from the
Astrometric and Photometric Database.  The Analysis Tasks must be able
to send result detections to the AP Database. 
\end{enumerate}

\subsection{Requirements Trace Matrix}
\begin{center}
\footnotesize
\begin{tabular}{|l|l|l|l|} 
\hline 
\multicolumn{2}{|c|}{\bf Subsystem Requirements} &  
\multicolumn{2}{|c|}{\bf System/Subsystem Requirements} \\ \hline 
\FRN{\bf Number} & 
\FRS{\bf Caption} & 
\FRN{\bf Number} & 
\FRS{\bf Caption} \\ \hline

\ref{IPP:Req:1}  & \FRS{Photometrically consistent images to 1\%} & \ref{OpsReq:D} & \FRS{Photometric precision of 0.01 mag} \\ \hline 
\ref{IPP:Req:2}  & \FRS{Photometrically calibrated images to 1\%} & \ref{OpsReq:D} & \FRS{Photometric precision of 0.01 mag} \\ \hline 
\ref{IPP:Req:3}  & \FRS{Absolute astrometry to 100 mas} & \ref{OpsReq:E} & \FRS{Absolute Astrometric precision of 100 mas} \\ \hline 
\ref{IPP:Req:4}  & \FRS{Relative astrometry to 30 mas} & \ref{OpsReq:F} & \FRS{Relative Astrometric precision of 100 mas} \\ \hline 
\ref{IPP:Req:5}  & \FRS{Background correction to 1\%} & \ref{OpsReq:G} & \FRS{remove foregrounds to $<$ 1\% of background} \\ \hline 
\ref{IPP:Req:6}  & \FRS{Construct Static Sky for $g$ filter images} & \ref{OpsReq:J} & \FRS{construct static sky images} \\ \hline 
\ref{IPP:Req:7}  & \FRS{Construct Static Sky for $r$ filter images} & \ref{OpsReq:J} & \FRS{construct static sky images} \\ \hline 
\ref{IPP:Req:8}  & \FRS{Construct Static Sky for $i$ filter images} & \ref{OpsReq:J} & \FRS{construct static sky images} \\ \hline 
\ref{IPP:Req:9}  & \FRS{Construct Static Sky for $z$ filter images} & \ref{OpsReq:J} & \FRS{construct static sky images} \\ \hline 
\ref{IPP:Req:10} & \FRS{Construct Static Sky for $y$ filter images} & \ref{OpsReq:J} & \FRS{construct static sky images} \\ \hline 
\ref{IPP:Req:11} & \FRS{Construct Static Sky for $w$ filter images} & \ref{OpsReq:J} & \FRS{construct static sky images} \\ \hline 
\ref{IPP:Req:12} & \FRS{Detect \& classify objects on science images} & \ref{OpsReq:P} & \FRS{classify detected objects } \\ \hline 
\ref{IPP:Req:13} & \FRS{Detect \& classify objects on science image stacks} & \ref{OpsReq:P} & \FRS{classify detected objects } \\ \hline 
\ref{IPP:Req:14} & \FRS{Detect \& classify objects on static sky images} & \ref{OpsReq:P} & \FRS{classify detected objects } \\ \hline 
\ref{IPP:Req:15} & \FRS{Detect \& classify transients} & \ref{OpsReq:P} & \FRS{classify detected objects} \\ \hline 
\ref{IPP:Req:16} & \FRS{Degrade image size by $<$ 10 mas} & \ref{SysDeReq:2}, alloc. & \FRS{Degrade image size by \PSiqdeg\ of median seeing} \\ \hline 
\ref{IPP:Req:17} & \FRS{Process images arriving at cadence of 40 seconds} & \ref{OpsReq:B} & \FRS{processing for transients within 5 min} \\ \hline 
\ref{IPP:Req:18} & \FRS{Limit false alarm rate for transients} & \ref{OpsReq:M}   & \FRS{false alarm rate of $<1\%$} \\ \hline 
\ref{IPP:Req:19} & \FRS{Publish static sky images to PSPS} & \ref{OpsReq:Q} & \FRS{Data Products shall be made available} \\ \hline 
\ref{IPP:Req:20} & \FRS{Publish detected objects to PSPS} & \ref{OpsReq:Q} & \FRS{Data Products shall be made available} \\ \hline 
\ref{IPP:Req:21} & \FRS{Publish metadata to PSPS} & \ref{OpsReq:Q} & \FRS{Data Products shall be made available} \\ \hline 
\ref{IPP:Req:22} & \FRS{Provide access to preferred science clients}       & \ref{SysDeReq:9} & \FRS{allow interface to preferred science clients} \\ \hline 
\ref{IPP:Req:23} & \FRS{Store raw images for 1 month} & allocated 	   & \FRS{} \\ \hline 
\ref{IPP:Req:24} & \FRS{Store detected objects for 1 year} & allocated 	   & \FRS{} \\ \hline 
\ref{IPP:Req:25} & \FRS{Store metadata for project lifetime} & allocated   & \FRS{} \\ \hline 
\end{tabular}
\end{center}

\begin{center}
\footnotesize
\begin{tabular}{|l|l|l|l|} 
\hline 
\multicolumn{2}{|c|}{\bf Derived Subsystem Requirements} &  
\multicolumn{2}{|c|}{\bf Top-level Subsystem Requirements} \\ \hline 
\FRN{\bf Number} & 
\FRS{\bf Caption} & 
\FRN{\bf Number} & 
\FRS{\bf Caption} \\ \hline

\ref{IPP:DeReq:1}  & \FRS{Accept images from summit at rate...} & \ref{IPP:Req:17} & \FRS{Process images arriving at cadence of 40 seconds} \\ \hline 
\ref{IPP:DeReq:2}  & \FRS{Accept metadata from summit at rate...} & \ref{IPP:Req:17} & \FRS{Process images arriving at cadence of 40 seconds} \\ \hline 
\ref{IPP:DeReq:3}  & \FRS{Produce master calibration images in...} & allocated & \FRS{} \\ \hline 
\ref{IPP:DeReq:4}  & \FRS{Master cal. image introduce less than $1\%$} & \ref{IPP:Req:1} & \FRS{Photometrically consistent images to 1\%} \\ \hline 
\ref{IPP:DeReq:5}  & \FRS{Pre-process science image at rate...} & \ref{IPP:Req:17} & \FRS{Process images arriving at cadence of 40 seconds} \\ \hline 
\ref{IPP:DeReq:6}  & \FRS{Merge images into stacked images at rate...} & \ref{IPP:Req:17} & \FRS{Process images arriving at cadence of 40 seconds} \\ \hline 
\ref{IPP:DeReq:7}  & \FRS{Excise $>99\%$ of outlier pixels from stack} & \ref{IPP:Req:18} & \FRS{Limit false alarm rate for transients} \\ \hline 
\ref{IPP:DeReq:8}  & \FRS{Merge cleaned images into static sky at rate...} & \ref{IPP:Req:17} & \FRS{Process images arriving at cadence of 40 seconds} \\ \hline 
\ref{IPP:DeReq:9}  & \FRS{Measure objects on pre-processed images} & \ref{IPP:Req:12} & \FRS{Detect \& classify objects on science images} \\ \hline 
\ref{IPP:DeReq:10} & \FRS{Measure objects on stacked images} & \ref{IPP:Req:13} & \FRS{Detect \& classify objects on science image stacks} \\ \hline 
\ref{IPP:DeReq:11} & \FRS{Measure objects on difference images} & \ref{IPP:Req:15} & \FRS{Detect \& classify transients} \\ \hline 
\ref{IPP:DeReq:12} & \FRS{Measure objects on static sky images} & \ref{IPP:Req:14} & \FRS{Detect \& classify objects on static sky images} \\ \hline 
\ref{IPP:DeReq:13} & \FRS{astrometric accuracy for commissioning phase} & allocated & \FRS{} \\ \hline 
\ref{IPP:DeReq:14} & \FRS{astrometric accuracy for reference catalog phase} & allocated & \FRS{} \\ \hline 
\ref{IPP:DeReq:15} & \FRS{astrometric accuracy for normal operations} & \ref{IPP:Req:3} & \FRS{Absolute astrometry to 100 mas} \\ \hline 
\ref{IPP:DeReq:16} & \FRS{photometric accuracy for commissioning phase} & allocated & \FRS{} \\ \hline 
\ref{IPP:DeReq:17} & \FRS{photometric accuracy for reference catalog phase} & allocated & \FRS{} \\ \hline 
\ref{IPP:DeReq:18} & \FRS{relative photometric accuracy for normal operations} & \ref{IPP:Req:2} & \FRS{Photometrically calibrated images to 1\%} \\ \hline 
\ref{IPP:DeReq:19} & \FRS{absolute photometric accuracy for normal operations} & \ref{IPP:Req:2} & \FRS{Photometrically calibrated images to 1\%} \\ \hline 
\ref{IPP:DeReq:20} & \FRS{astrometric reference within 6 mo} & \ref{IPP:Req:3}, allocated & \FRS{Absolute astrometry to 100 mas} \\ \hline 
\ref{IPP:DeReq:21} & \FRS{astrometric reference astrometry accuracy} & \ref{IPP:Req:3} & \FRS{Absolute astrometry to 100 mas} \\ \hline 
\ref{IPP:DeReq:22} & \FRS{astrometric reference proper motion accuracy} & \ref{IPP:Req:3} & \FRS{Absolute astrometry to 100 mas} \\ \hline 
\ref{IPP:DeReq:23} & \FRS{photometric reference within 6 mo} & \ref{IPP:Req:2}, allocated & \FRS{Photometrically calibrated images to 1\%} \\ \hline 
\ref{IPP:DeReq:24} & \FRS{photometric reference global consistency} & \ref{IPP:Req:2} & \FRS{Photometrically calibrated images to 1\%} \\ \hline 
\ref{IPP:DeReq:25} & \FRS{photometric reference absolute accuracy} & \ref{IPP:Req:2} & \FRS{}Photometrically calibrated images to 1\% \\ \hline 
\ref{IPP:DeReq:26} & \FRS{publish static sky images every 6 mo.} & \ref{IPP:Req:19} & \FRS{Publish static sky images to PSPS} \\ \hline 
\ref{IPP:DeReq:27} & \FRS{publish detected objects every 1 mo.} & \ref{IPP:Req:20} & \FRS{Publish detected objects to PSPS} \\ \hline 
\ref{IPP:DeReq:28} & \FRS{publish metadata every 1 week} & \ref{IPP:Req:21} & \FRS{Publish metadata to PSPS} \\ \hline 
\ref{IPP:DeReq:29a} & \FRS{provide transients to MOPS rate...} & \ref{IPP:Req:22} & \FRS{Provide access to preferred science clients} \\ \hline 
\ref{IPP:DeReq:29b} & \FRS{provide metadat to MOPS at rate...} & \ref{IPP:Req:22} & \FRS{Provide access to preferred science clients} \\ \hline 
\ref{IPP:DeReq:29c} & \FRS{provide transients to other clients at rate...} & \ref{IPP:Req:22} & \FRS{Provide access to preferred science clients} \\ \hline 
\ref{IPP:DeReq:30} & \FRS{store raw images for 1 month} & \ref{IPP:Req:23} & \FRS{Store raw images for 1 month} \\ \hline 
\ref{IPP:DeReq:31} & \FRS{store detected objects for 1 year} & \ref{IPP:Req:24} & \FRS{Store detected objects for 1 year} \\ \hline 
\ref{IPP:DeReq:32} & \FRS{store metadata for project lifetime} & \ref{IPP:Req:25} & \FRS{Store metadata for project lifetime} \\ \hline 
\end{tabular}
\end{center}

\endnotesection

\ifwhole\else

\subsection{Notes}{

There are several system concepts and goals which must be defined
before the IPP section:

\begin{enumerate}
\item relationship between FPA image, OTA, and cell
\item concept of a `major frame' (observationally defined group of
  images). 
\item distinction of a images grouped together for a science goal from
  images grouped together for processing purposes.

\item timing budget: need to specify total delay allowed from shutter
  close to specific science result, and break down for the stages:
  readout, OTIS-IPP communication, OTIS-IPP data transfer, IPP
  processing Phase 2 and Phase 4, IPP-client science pipeline
  communication.

\item{discuss the astrometry and photometry errors introduced at each
  stage of the analysis}

\item mention future pre-processing at the summmit?

\item does the Top-Level Description need a discussion of the basic goals of the image analysis?

\end{enumerate}

\subsection{Definitions}

\subsection{External Interfaces}

\end{document}
\fi 
