\documentclass[panstarrs]{panstarrs}

% basic document variables
\title{Pan-STARRS PS-1 Image Processing Pipeline}
\subtitle{System/Subsystem Design Description}
\shorttitle{IPP SSDD}
\author{Eugene A. Magnier, Paul A. Price, Josh Hoblitt}
\audience{Pan-STARRS PMO}
\group{Pan-STARRS Algorithm Group}
\project{Pan-STARRS Image Processing Pipeline}
\organization{Institute for Astronomy}
\version{00}
\docnumber{PSDC-430-014}

% allow paragraphs to be listed in TOC for now 
\setcounter{tocdepth}{3} 

\begin{document}
\maketitle

% -- Revision History --
\RevisionsStart
% version     Date         Description
DR.01     & 2006.01.19 & First draft release to CDR committee \\ \hline
DR.02     & 2006.01.22 & Second draft release to CDR committee \\ \hline
00        & 2006.04.26 & First version release after CDR report \\ \hline
\RevisionsEnd

\inserttbd
\inserttbr
\pagebreak 

\tableofcontents
\pagebreak

\listoffigures
\pagebreak
\pagenumbering{arabic}

\section{Scope}

\subsection{Identification}

This document establishes Software Design Requirements for the
Panoramic Survey Telescope and Rapid Response System (Pan-STARRS)
Image Processing Pipeline (IPP) for the prototype telescope PS-1, and
is a System-level controlled specification/design description document
in the official Pan-STARRS engineering specification tree.

\subsection{System Overview}

The Institute for Astronomy at the University of Hawaii is developing
a large optical synoptic survey telescope system, the Panoramic Survey
Telescope and Rapid Response System (Pan-STARRS). The science goals,
priorities, top-level concept of operations with associated
operational requirements, and system performance drivers with
associated system performance requirements are described in the
Pan-STARRS Science Goals Statement (SGS).  As described in this
document, The system conceptual design for Pan-STARRS utilizes an
array of four 1.8m telescopes each with a 7 degree$^2$ field of view,
giving the system an \'etendue larger than all existing survey
instruments combined (defined as the product of the collecting area
$A$ multiplied by the field-of-view solid angle $\Omega$).  Each
telescope will be equipped with a 1 billion pixel CCD camera with low
noise and rapid read-out, and the data will be reduced in near real
time to produce both cumulative static sky and difference images from
which transient, moving, and variable objects can be
detected. Pan-STARRS will be able to survey up to $\approx 6,000$
degree$^{2}$ per night to a detection limit of approximately 24$^{th}$
magnitude.  This unique combination of sensitivity and sky coverage
will open up many new possibilities in time domain astronomy including
a major goal of surveying the Potentially Hazardous Object (PHO)
population down to a diameter of $\approx 300$ meters.  In addition,
the Pan-STARRS data will be used to investigate a broad range of
astronomical problems of extreme current interest concerning the Solar
System, the Galaxy, and the Cosmos at large.  A prototype single
telescope system, PS-1, is being developed as a preliminary step
before construction of the complete four telescope system.

\begin{tabular}{ll}
Project sponsor:&	AFRL, United States Air Force \\
Acquirer:       &	University of Hawaii Institute for Astronomy \\
User: 		&	Astronomical community \\
Developer:      &	University of Hawaii Institute for Astronomy, participating \\
                &       institutions, and associated subcontractors	
\end{tabular}

\subsection{Document Overview}

The Pan-STARRS IPP System/Subsystem Design Description (SSDD) contains
the complete design description of the Pan-STARRS PS-1 IPP in order to
achieve the requirements specified by the Pan-STARRS PS-1 IPP Software
Requirements Specification (SRS; PSDC-430-005).  The requirements flow
which began in the SGS and Pan-STARRS system SCD, and continued in the
IPP SRS and the IPP SCD, is used to guide the design presented here.

\subsection{Requirements Definitions}

The Pan-STARRS document naming scheme is PSDC-NNN-MMM-VV, where the VV
entry specifies the document version number.  Where documents are
identified without the version number, the latest official version in
that series is implied.  

Open issues (TBDs) in this document are marked {\bf \color{red} in
bold red}.

Quantities which should be reviewed (TBRs) are marked {\bf
\color{blue} in bold blue}.

\subsubsection{``Shall''}  When used in this specification, the word
``shall'' refers to an explicit requirement of a system component or
the complete system.  

\subsubsection{``Should''}  When used in this specification, the word
``should'' refers to a desired characteristic of a system component or
the complete system.

\subsubsection{``Will''}  When used in this specification, the word
``will'' provides information about a characteristic of a related
system component or a complete related system.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\DocumentsInternalSection
PSDC-230-001  &   PS-1 Design Reference Mission \\ \hline
PSDC-230-002  &   PS-1 System Concept Definition \\ \hline
PSDC-400-006  &   The Pan-STARRS IPP Computational Challenge \\ \hline
PSDC-430-004  &   Pan-STARRS IPP C Code Conventions \\ \hline
PSDC-430-005  &   Pan-STARRS IPP PS-1 Software Requirements Specification \\ \hline
PSDC-430-006  &   Pan-STARRS IPP Algorithm Design Document \\ \hline
PSDC-430-007  &   Pan-STARRS IPP PSLib Supplementary Design Requirements Specification \\ \hline
PSDC-430-010  &   Pan-STARRS IPP Perl Code Conventions \\ \hline
PSDC-430-011  &   Pan-STARRS IPP System Concept Definition \\ \hline
PSDC-430-012  &   Pan-STARRS IPP Modules Supplementary Design Requirements Specification \\ \hline
PSDC-430-014  &   Pan-STARRS IPP PS-1 Cluster Support \\ \hline
\DocumentsExternalSection
Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\
\DocumentsEnd

\section{Subsystem Overview}

The Pan-STARRS Image Processing Pipeline (IPP) performs the image
processing and data analysis tasks needed to enable the scientific use
of the images obtained by the Pan-STARRS telescopes.  The primary
goals of the IPP are to process the science images from the Pan-STARRS
telescopes and make the results available to other systems within
Pan-STARRS.  It also is responsible for combining all of the science
images in a given filter into a single representation of the
non-variable component of the night sky defined as the ``Static Sky''.
To achieve these goals, the IPP also performs other analysis functions
to generate the calibrations needed in the science image processing
and to occasionally use the derived data to generate improved
astrometric and photometric reference catalogs.  It also provides the
infrastructure needed to store the incoming data and the resulting
data products.

The IPP inherits lessons learned, and in some cases code and prototype
code, from several other astronomy image analysis systems, including
Imcat (Kaiser), the Sloan Digital Sky Survey (REF), the Elixir system
(Magnier \& Cuillandre), and Vista (Tonry).  Imcat and Vista have a
large number of robust image processing functions.  SDSS has
demonstrated a working analysis pipeline and large-scale database
system for a dedicated project.  The Elixir system has demonstrated an
automatic image processing system and an object database system for
operational usage.

The users of the IPP output are all systems internal to the Pan-STARRS
project.  They consist of: 1) the Preferred Science Clients, which
receive specified data products on short timescales.  2) the Moving
Object Processing System (MOPS), which is one of the Preferred Science
Clients, but has the distinction of being a component funded by
Pan-STARRS.  It will receive the detections of non-stationary
transient objects.  3) the Published Science Products Subsystem
(PSPS), which will receive all data products of interest to the
community external to the Pan-STARRS data processing systems, and will
act as the long-term archive and publishing clearinghouse.

The IPP receives data from two Pan-STARRS subsystems: the Camera, from
which it receives the large volume of image data, and OTIS
(Observatory, Telescope and Infrastructure Subsystem), from which it
receives metadata describing the images and the environmental
conditions.  The location of the IPP computing hardware will evolve
over the course of the first year of the PS-1 project.  Initially,
during the start of the commissioning phase, a subset of hardware will
be located at the summit facility; when the network connection is
established and the new facility is available, the IPP hardware will
be relocated to the MHPCC computer room in Kihei.  A subset of
processing tasks may eventually be re-assigned to machines at the
summit if justified by the savings in data transfer time and cost, but
this implementation is considered a possible future evolutionary
path.  The details of the hardware deployment plan are discussed in
the section on the IPP hardware.  

The Pan-STARRS camera produces images consisting of multiple chips
(Orthogonal Transfer Arrays or OTAs), each consisting of multiple
cells (continuous set of pixels).  The baseline design for the
Pan-STARRS camera contains 64 chips each with 64 cells.

This document defines the design requirements of the IPP for the PS-1
prototype telescope.  Much of the IPP design for PS-4 will be
identical to or closely based on the PS-1 implementation.  The
software organization and the infrastructure systems are expected to
be identical, with minor improvements in details.  The type of
analysis steps to be performed will be nearly identical, with some
additional details added for PS-4 to improve the accuracy.

Although generally very similar, in terms of the IPP PS-1 differs from
the complete PS-4 system in several specific ways.  First, with only
one telescope and camera, the data throughput rate is substantially
reduced to a maximum of 1 64-OTA image per 40 seconds rather than 4.
Since PS-1 is a prototype for testing the Pan-STARRS hardware and
software subsystems, the observing strategy is not a fixed quantity.
The PS-1 Design Reference Mission (PSDC-230-001) provides some
guidelines for the types of observing tests which will probably be
performed, including possibly starting an Astrometric and Photometric
Survey which will eventually cover the entire $3\pi$ steradians of the
sky accessible to PS-4.  As a prototype, it is expected that much of
the data collected by PS-1 will be processed multiple times to test
and tune the analysis steps.  Compare with PS-4, this difference in
approach has implications for the storage required by PS-1: rather
than delete images soon after they have been used, raw images from
demonstration observations must be stored for at least the first two
years of PS-1 operations.  The PS-1 Design Reference Mission is used
as an upper limit for these storage requirements to drive the hardware
design.

\subsection{System Design Decisions}

Since Pan-STARRS is a survey project, all data from the telescopes
will be uniformly analyzed by the Pan-STARRS Image Processing Pipeline
(IPP), and the appropriate resulting data products made available to
internal and external science analysis systems as they become
available.  The processing performed by the IPP on the science images
will consist of detrending and object detection for the individual
images, combination of multiple overlapping images and further object
detection, subtraction of a reference (static-sky) image and detection
of residual objects, update of the static sky images, and detailed
object analysis of the static sky images.  In addition, the IPP will
produce improved astrometric and photometric reference catalogs on an
as-needed basis.  The output data products from the IPP consist of the
calibration images, reduced images from the individual telescopes,
combined images, difference images, the static sky image, object
photometry, and reference astrometry and photometry.

The requirements for the IPP, as identified in the PS-1 IPP SRS
(PSDC-430-005) fall into several broad categories: data analysis
precision, throughput, system reliability, flexibility, testability,
and traceability.  The details of the analysis tasks are specified in
order to achieve the precision.  The architectural design as discussed
below is motivated by the need for reliability and flexibility.  The
hardware organization and the distributed/parallel processing model is
motivated by the throughput requirements.  The need for flexibility
and testability drives the software organization.  The need for simple
testing procedures drives both the software organization and the
separation of the system architecture into different infrastructure
elements.

\subsection{Analysis Stages, Programs, and Libraries} 

The processing steps listed above can be divided into well-defined
analysis stages, each of which operates on a particular unit of data,
such as a single OTA image or a collection of astronomical objects.
The different analysis stages defined by the IPP are discussed in
Sections~\ref{sec:AnalysisStages} - \ref{sec:MiscStages}.  The
analysis defined by these analysis stages is further divided into
specific programs which perform the particular operation.  Some
analysis stages are primarily defined by a single programs; others
require multiple programs and substages.  It is also possible for a
single program to be involved in multiple analysis stages.  The major
IPP analysis programs are discussed in some detail in
Section~\ref{sec:AnalysisPrograms}.  Other important programs in the
collection called \code{ippTools} are used to define the analysis
stages and their interrelationships (Section~\ref{sec:ipptools}).  The
IPP software hierarchy is further divided into \code{psModules}
(representing complex analysis operations, often astronomical in
nature) and \code{psLib}, the data and analysis foundation library
(Section~\ref{sec:SoftwareHierarchy}).

Analysis tasks representing the different analysis stages are
performed on the IPP computer cluster.  Note the distinction between
the generic analysis {\em stage} and a specific analysis {\em task}.
An analysis stage represents a type of analysis which is performed,
such as the basic image calibration and object detection analysis.  An
analysis task is a particular realization of an analysis stage, e.g.,
the analysis of OTA number 61 from exposure 654321 to produce a
specific set of output data products.  The analysis stages are
discussed in detail in Section~\ref{sec:AnalysisStages}.

A particular stage may process individual images, collections of
images, or derived data products.  Because of the nature of the image
data, many of the analysis stages can be run in parallel if needed to
increase the processing throughput.  For example, the analysis of a
chip in one image does not depend on the results from another chip.

\subsection{Architectural Components}

\begin{figure}
\begin{center}
\resizebox{6in}{!}{\includegraphics{pics/IPPoverview}}
\caption{ \label{fig:overview} IPP System Overview}
\end{center}
\end{figure}

In order to achieve the required functionality, the IPP provides an
infrastructure within which the Analysis Stages described above are
executed.  In order to facilitate the subsystem testing, the IPP
software infrastructure has been divided into a number of
clearly-defined architectural software units.  In the following
summary, these elements are described in terms of the concept they are
addressing; in later sections, this document will discuss the
specifics of the implementations used by the IPP for PS-1.

\begin{itemize}

\item {\bf Image/File Server:} This component is a large data store
  for all images and large used by the IPP, including the raw images
  from the telescope, the master calibration images, the reference
  static-sky images, and any temporary image data products produced by
  the IPP.  The Image/File Server accepts data products and stores
  them until they are no longer needed by other portions of the IPP.
  It allows other IPP subsystems to refer to the data files with an
  abstract identification without needing to worry about the details
  of the physical location.  Conversely, it allows other entities to
  determine or specify the physical locations if needed.  The
  Image/File Server is capable of storing any large data files which
  are not well-suited for inclusion in a more structured relational
  database, and for which access needs to be widely available beyond
  the individual process which created the file.  The IPP has
  developed the software system called 'Nebulous' to perform this
  function.

\item {\bf Metadata Database:} This component stores the data which is
  not directly related to images or astronomical objects, but which is
  needed to perform the IPP analyses.  The metadata may include the
  summary weather information for each night, or details about the
  filters, camera, telescopes, etc.  Note that the IPP Metadata
  Database is not required to retain all archival engineering data
  from all of Pan-STARRS; other Pan-STARRS subsystems use their own
  internal databases to store engineering metadata and only the
  necessary subset is transferred to the IPP Metadata Database.  The
  IPP uses the MySQL database engine, together with stand-alone
  software tools to define, manage and examine the Metadata Database
  tables.  These tools and other access functions are built using
  autogenerated APIs supplied by the software called 'glueforge'

\item {\bf Astrometry \& Photometry Database:} This component stores
  and manipulates astronomical objects detected in various images, as
  identified above, including individual measurements of objects on
  the images, the summary information about those objects, and
  reference object data.  It also provides mechanisms for users to
  query and manipulate the objects and detections.  For this
  component, the IPP has adopted the tool called 'DVO' from the Elixir
  system and has made / is making necessary upgrades to meet the
  requiements of the PS-1 system.

\item {\bf IPP Controller:} In order to achieve the required
  processing throughput for the IPP analysis stages, it is necessary
  to use distributed computing processes on a large number of
  computers.  The IPP Controller manages the collection of analysis
  tasks performed on these machines.  The actual implementation of the
  controller is called 'pcontrol', and is strongly coupled to the
  Scheduler discussed below.

\item {\bf IPP Scheduler:} This component is a decision-making
  mechanism which guides the operation of the IPP.  It evaluates the
  currently available collection of data, identifies the necessary
  analysis, and assigns the analysis tasks to the IPP Controller.  The
  actual implementation of the scheduler is built on top of the
  controller in a system called 'PanTasks'.  

\end{itemize}

The relationship between these software units is shown in
Figure~\ref{fig:overview}.  This figure also shows the interactions
between the IPP and other Pan-STARRS systems.  The implementation of
the architectural components is discussed in detail in
Section~\ref{sec:ArchComponents}.

\begin{figure}
\begin{center}
\resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}}
\caption{ \label{fig:hardware} IPP Hardware Organization}
\end{center}
\end{figure}

\subsection{IPP Hardware Organization}

The IPP will utilize substantial computer resources, both in terms of
computational power and in terms of data storage and network
bandwidth.  The IPP requires relatively large amounts of data storage
space, primarily for the image data.  Image data is organized in two
categories.  First, there is the per-OTA data -- data associated with
specific OTAs, including the raw images, the calibration images, and
temporary processed images at various stages.  Second, there is the
data associated with the static sky imagery, which is in turn
organized into smaller sky-cell units.  In addition to image data,
there are the somewhat smaller data entities of the Metadata Database
and AP Database.

The computer hardware is organized into nodes which provide both data
storage and computational resources.  The data storage nodes are
divided into three classes: those which deal with the per-OTA image
data, those that provide the storage for the static sky images, and
those that provide the storage for the other data systems, the
Metadata Database and the AP Database.  In addition, the computational
tasks related to the individual images take place on the per-OTA
storage nodes and the processing of stacks of images takes place on
the static sky storage nodes.

Figure~\ref{fig:hardware} presents the basic concept for the hardware
organization for the IPP.  This diagram shows the two types of compute
nodes: (1) OTA-level processing and storage nodes and (2) Static Sky
processing and storage nodes.  Also shown are two switches which
divide the network into OTA and Static-Sky portions.  In such an
organization, the inter-switch communication must meet the throughput
needs between these network portions (though a single switch may also
be used if its backplane capacity is sufficient).  The additional data
systems (Metadata Database and AP Database) are also shown.

\tbd{re-draw to use a single switch}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{System Design : Architectural Components}
\label{sec:ArchComponents}

\subsection{Nebulous : the IPP Image/File Server}

\subsubsection{Corresponding Requirements}

The IPP Image/File Server must meet the requirements specified in
Section 3.4.1 of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005).  The
specified design is chosen to meet requirements 3.4.1.3, and 3.4.1.5.
The other three requirements (3.4.1.1, 3.4.1.2, and 3.4.1.4) depend on
the volume and capabilities of the hardware, and are addressed in
Section~\ref{sec:Hardware}.  

\subsubsection{Image/File Server Overview}

The IPP Image/File Server is a repository for all images and other
large data files required by the IPP.  Along with the storage
hardware, it provides tools for managing the distribution of these
large data files and for accessing the files.  Data files stored by
the IPP Image/File Server include the raw images, the calibration
images, intermediate processing stage images as needed, final
processed images, difference images, image subsections, and any large
non-imaging data files needed by the IPP.  The IPP Image/File Server
must retain the files for as long as they are needed by the IPP.

The IPP team has developed the software system 'Nebulous' to meet the
requirements of the Image/File Server.  Nebulous uses a MySql database
engine to track the identities and locations of the files under its
management.  Nebulous currently requires that all files it manages be
available from a locally mounted file system; however, it does not
place specific requirements upon the choice of this file system.
Currently, the IPP is being implemented use NFS as the distributed
file system, though other systems such as GFS or GSFS could be used in
an equivalent fashion.  

The decision to make use of a locally mounted file system is driven by
two IPP-wide design decisions.  First, as a practical and technical
consideration, in several locations throughout the IPP analysis, the
efficiency can be increased significantly if processes have the
ability to seek randomly in a file.  For example, when combining
multiple partially-overlapping images, rather the be forced to read
through an entire image data, the proccess can seek through a
multi-component file to the pixels of interest.  The other driving
motivation is more philosophical in nature: the IPP is designed to use
components which operate as simple user commands in the UNIX
environment.  By requiring Nebulous to work with files in the locally
mounted file system, all UNIX programs will be able to interact with
those files if needed.  This allows the IPP to include essentially any
normal analysis program as part of the IPP system without difficulty.
This philosophical choice is also present in the design of the IPP
Scheduler / Controller system and in the design implementation of the
data analysis programs. 

Nebulous is a parallel storage system.  It stores data across a
collection of computer nodes, each with their own data storage
resources.  Any single file is stored on only a single computer and
storage device.  In order to achieve the data throughput requirements,
Nebulous will be used to distribute the images across the processor
nodes in an organized fashion, i.e., associating specific machines
with specific detectors.  It is not the responsibility of Nebulous to
determine which computer should be associated with a specific data
concept (Chip / region of sky), but it instead provides the hooks to
enable the association of a particular file with a particular machine.

There are three data concepts relevant to Nebulous:
\begin{itemize}
\item {\bf Storage object:} This represents a single, unique data
  entity managed by Nebulous.

\item {\bf Storage ID:} This is the identifier of a particular storage
  object in Nebulous.  The Storage ID is the key used by any Nebulous
  users to retrieve a specific file of interest.  The file ID is
  simply a unique string, equivalent to the filename in a UNIX file
  system.

\item {\bf Instance (or Storage Object Instance):} A single copy of
  the storage object in Nebulous.  In general, a given storage object
  may have several instances in Nebulous, normally on different
  computer nodes.

\end{itemize}
Upon request of a specific Storage ID, Nebulous provides file pointers
(in C), handles (in Perl), or file names corresponding to the
instances of the storage objects.  

Nebulous provides the storage and access mechanisms, but it does not
include any logic or information about the data.  It does not, e.g.,
monitor the age of images and delete them on some schedule.  This
functionality currently resides in the IPP Scheduler
(Section~\ref{sec:pantasks}).

As shown in Figure~\ref{fig:Nebulous}, Nebulous consists of the
following principal components:
\begin{itemize}
\item Nebulous client(s)
\item Nebulous server
\item Nebulous database 
\item Storage hardware 
\end{itemize}

\begin{figure}
\begin{center}
\resizebox{4.5in}{!}{\includegraphics{pics/ImageServer}}
\caption{The components of Nebulous, the IPP Image/File Server.}
\label{fig:Nebulous}
\end{center}
\end{figure}

\subsubsection{Nebulous Client APIs}

Clients interact with the IPP Image Server via a small number of C
APIs.  Bindings are also provided for Perl \tbd{and Python} and UNIX
shell commands in some cases.  This document only gives an overview of
the commands; for details on usage, please see the Nebulous user's
guide.  The client commands are:

\begin{itemize}
\item {\tt create} : create a new storage object in the Image Server.
  This function takes as input the requested Storage ID and returns a
  C-style file pointer, Perl file handle, or file name corresponding
  to the new instance of the storage object.  The arguments to the
  function include an optional node name on which the new storage
  object must be located.  If this target is not given, the Image
  Server places the new storage object on an appropriate machine from
  the pool.

\item {\tt replicate} : a new instance of the given storage object is
  created.  The target node may be optionally specified, otherwise an
  appropriate node is selected.

\item {\tt cull} : removes one of the instances of the storage object.
  The input parameters may optionally specify the target machine from
  which to delete the object.

\item {\tt delete} : deletes all instances of the storage object and
  sets the storage object status to {\tt deleted}.

\item {\tt open} : open an instance of an existing storage object, as
  identified by the storage ID.  This function may also specify the
  node on which the object should be opened (if an instance of the
  object is not stored on that node, the function returns an error).
  On success, the function returns a file pointer.  If the object is
  opened for 'write' access, all but one instance is deleted to ensure
  consistency of the data.  

\item {\tt find} : return a list of filenames in the UNIX name space
  associated with the storage object identified by the given file ID.
  Since there are in general multiple instances for a given storage
  object, this function returns the collection of all available
  instances.  These may be freely opened by the client server using
  the standard \code{fopen} functions. 

\item {\tt lock} : attempt to acquire a Nebulous lock on the storage object.

\item {\tt unlock} : release a Nebulous lock from the storage object.

\item {\tt stat} : returns status information about the specified
  storage object, including the number of instances of the object.

\item {\tt copy} : create a new storage object with one instance of
  the corresponding object.

\item {\tt move} : rename a storage object

\item {\tt import} : copy an existing file object into Nebulous.  

\end{itemize}

\subsubsection{Nebulous Server}

The Nebulous client requests are mediated via the Nebulous server.
Communication between the clients and the server is via SOAP
implementing the commands above.  The identity of the machine on which
the Nebulous server runs is part of the Nebulous configuration
information.

The server is responsible for keeping track of storage objects, all
instances of that object, and enforcing locking semantics.  Extensive
logging and tracing support is provided for debug and to allow for
statics generation and possible {\em hotspot} optimization.

Nebulous uses a centralized server model.  This model was choosen
because it allows efficient {\em pattern matching} of storage object
names.  The current 'best' technique for a distributed metadata store
is with distributed hash tables.  Unfortunately, no widely available
DHT implementation allows efficient {\em pattern matching} of key
names.

\paragraph{House keeping}

\subparagraph{Lock sweeping} 
In the event that a Storage Object operation fails to complete
successfully stale locks will have to be identified and removed from
the IPP Pixel Data Server Database.  This should be done periodically
by comparing the entries in the Lock table to the list of active nodes
maintained by the IPP Controller.  It should also happen as soon as
possible after a node goes offline (triggered by the IPP Controller
marking a node as offline?).  A sweep must be /completed/ before an
offline node can be marked on-line.

Once a node is determined to be offline all entries in the Lock table
set by that node should be identified.  The locks on the Storage
Object Instances pointed to by those entries should then be rolled
back and Lock Record entries themselves must be removed from the lock
table.

\subparagraph{Consistency sweeping} 
Periodically the IPP Pixel Data Server meta-data and Storage Object
will need to be checked for sanity.  This would be similar to running
fsck on a modern filesystem.  Consistency sweeping should include Lock
sweeping and should be considered a super-set.

\subsubsection{Nebulous Database}

The Nebulous Server uses a database to store the information about the
data storage objects, their instances, and the available hardware
resources.  A {\tt mysql} database engine is used to manage the
database table.  The database tables defined for the Image Server are
listed in Table~\ref{tab:ImageServerTables}, and their contents are
listed in Appendix~\ref{sec:ImageServerTableContents}.  This database
engine is not in general the same one used for other IPP subsystems;
the full IPP hardware configuration will include independent machines
for each of the major databasing systems (Nebulous, Metadata, DVO).
In the earlier incarnations, the same hardware and database engine may
be used.
%
\begin{table}[ht]
\begin{center}
\caption{Nebulous Database Tables\label{tab:ImageServerTables}}
\begin{tabular}{ll}
\hline
\hline
{\bf Table Name} & {\bf Description} \\
\hline
\code{storage_object}  & all storage objects known to Image Server \\
\code{instance}        & all instances of all storage objects \\
\code{volume}          & data storage devices known to Image Server \\
\hline
\end{tabular}
\end{center}
\end{table}

\subsubsection{Nebulous Storage Hardware}

Nebulous manages data across a collection of computers and as needed
on multiple storage devices on those computer nodes.  Nebulous
maintains a table of the available data volumes.  It tracks
information about each volume such as the total capacity, the current
capacity, the association between computer and data volume.  \tbd{Is
Nebulous responsible for detecting unavailable hardware?  it is
reponsible for changing allocations?  or is this a pantasks
responsibility?}

\subsubsection{Requirements Demonstrations}

\tbd{summary of throughput tests : create / copy / delete objects per
  second, etc}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Metadata Database}
\label{sec:Metadata}

\subsubsection{Corresponding Requirements}

The Metadata Database must meet the requirements specified in Section
3.4.2 of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005).  The specified
design is chosen to meet requirements 3.4.2.1, 3.4.2.2, 3.4.2.3,
3.4.2.4, 3.4.2.5.

\subsubsection{Overview}

The IPP Metadata Database acts as a repository for most non-pixel data
needed by the IPP subsystems.  This includes the image metadata, the
environmental data, system configuration data and system reference
data.  The Metadata Database is required to save the non-ephemeral
data for the lifetime of the project for future reference and
additional analysis.  The Metadata Database is also used in close
coupling with the analysis pipelines to track the state of elements as
they move through the processing system.  Metadata which is large in
volume or poorly structured is stored in an appropriate container file
(FITS Table, FITS Header, XML File) in Nebulous, with the Metadata DB
maintaining the corresponding Nebulous storage IDs of these files.

The IPP Metadata Database is a simple database system, consisting of a
number of simple tables without extensive inter-table links.  The IPP
uses the MySQL database engine for the the database.  To simplify the
coding and management of the database, the IPP uses autocoded APIs
constructed with the system called 'glueforge' to define and
manipulate the Metadata Database tables.

\begin{table}[hb]
\begin{center}
\caption{Metadata Database Tables\label{tab:MetadataDBTables}}
\begin{tabular}{ll}
\hline
\hline
{\bf Table Name}           & {\bf Description} \\
\hline
Weather                    & Details on the weather, including internal temperatures. \\
SkyProbe Transparency      & Analysis of SkyProbe B \& V data. \\
SkyProbe Absorption        & Analysis of SkyProbe A data. \\
SkyProbe Emission          & Analysis of SkyProbe E data. \\
DIMM                       & Summary of DIMM data analysis. \\
NIR                        & Summary statistics from NIR camera. \\
Dome Status                & The time history of the dome status. \\
Telescope Status           & The time history of the telescope status. \\
Raw FPAs                   & Information about the raw FPA exposures. \\
Pending Science Chips      & Science images to be processed and status. \\
Processed Science Chips    & Science images which have been migrated to the processed state. \\
Observation Group          & Details about a group of associated observations. \\
Observation Frame          & Major frame information. \\
Science Processing stats   & Details on processed cells. \\
Chip / Sky overlaps        & List of overlaps between sky cells and detectors. \\
Processed Sky-Cell stats   & Details of the sky cell processing. \\
\hline
\end{tabular}
\end{center}
\end{table}

\subsubsection{Metadata Tables}

Table~\ref{tab:MetadataDBTables} lists the Metadata tables identified to
date for the Metadata Database.  The contents of these tables are
outlined in Appendix~\ref{sec:MetadataTableContents}, with examples for
the data entries and their data types in many cases.  Additional
tables will be added as necessary as the data analysis scripts are
fleshed out in detail.  The Metadata Database, with a flat data
organization, is flexible enough to add additional information as it
is identified.

\subsubsection{Autocoded Metadata Queries}

The IPP provides standardized interfaces to the Metadata tables using
the 'glueforge' system.  Glueforge uses a standard table description
file to construct a collection of standard interface functions with
easily predicted names.  Given the description of a table (say Foo),
Glueforge provides a C-struct which represents the elements of single
row of the table (\code{FooRow}).  It also provides APIs to create a
new Foo table (\code{FooCreateTable()}), to insert a row either
supplying the elements of Foo (to \code{FooInsert()}) or by supplying
a pointer to data of type Foo (to \code{FooInsertObject()}).  Simple
queries may be constructed to select rows from the table
(\code{FooSelectRow()}).  The same mechanism generates data I/O
functions for writing FITS tables from a collection of the data
elements.  

This autocoding system for interacting with the Metadata database
makes the software very flexible for changes to the structures of the
Metadata database tables.  The programmer does not need to know
anything about the details of a given table to interact with it,
except when a specific element is needed.  Thus, new columns can be
added to the tables, and only require a re-compilation for most
portions of the IPP code.  Even migration to a new table schema for
existing data becomes fairly trivial.  Such a migration only would
require the definition of a conversion function from the old structure
to the new structure.  The more general features of glueforge are
discussed in the glueforge manual pages.  

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{DVO : the AP Database}

\subsubsection{Corresponding Requirements}

The AP Database must meet the requirements specified in Section 3.4.3
of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005).  The specified design
is chosen to meet requirements 3.4.3.1 and 3.4.3.2.  The IPP is
modifying the Elixir program 'DVO' to perform the role of the IPP AP
Database.  

\subsubsection{Overview}

DVO, the Desktop Virtual Observatory, is a software system which
stores data related to astronomical objects derived from various
sources, and provides mechanisms to related multiple detections
together as astronomical objects.  DVO deals with two related
concepts: {\em objects} and {\em detections}.  The {\em objects} are
descriptions of astronomical objects while the {\em detections} are
the specific measurements of those objects, typically measured from
astronomical images.  A collection of {\em detections} may be used to
derive average quantities which describe a particular {\em object}.  A
third class of measurement to be considered are those supplied by
external references.  Such measurements may be treated as {\em
detections}, with the caveat that access to the raw measurements and
metadata are usually unavailable: the reported measurements and errors
must be accepted as they are reported.

DVO stores the collections of detections which were derived from
specific images.  It provides a mechanism to determine the image from
which a specific detection was derived, and in conjunction with the
Image Server locate the corresponding data file.  DVO also makes it
possible to extract all detections derived from a specific image and
to determine quantities such as the pixel coordinates of the detection
on the image.

DVO also has the capability to associate multiple detections of a
specific object.  Several major classes of objects will be present,
each of which must be handled correctly.  DVO distinguished the
following types of objects.

{\bf Stars, compact galaxies, and QSOs} will have nearly fixed
locations relative to other distant stars, with only small deviations
for individual measurements.  The association between multiple
detections of such objects is made on the basis of their coincident
positions.  DVO determines the average position of the object and the
deviations of the individual detections from that average on the basis
of the ensemble of individual detection.

{\bf Solar System Objects} do not have a fixed location.  Detections
of such objects are linked by their orbits, and depend on both the
position and the time of the image.  DVO does not attempt to make this
link; this is the role of the MOPS system.  However, it has the
ability to accept identifications made externally with specified
detections and to return the identifier of the moving object
associated with the specific detections.  These associations also
include descriptive information such as the offset of the detection
from the predicted location of the detection based on the orbit.  This
functionality is required to allow DVO to ignore known moving object
detections from other types of queries.

{\bf High-proper-motion objects} in the general vicinity of the solar
system fall in between these first two classes of objects.  Their
proper motion and parallax response is significant enough ($>0.2$
arcsec in 1 year) that they are not well-described by an average
location and a collection of offsets.  These objects are better
described by a distance and a proper motion vector.  DVO provides the
association between the specific detections and an average object
which includes finite parallax and proper motion.

{\bf Orphaned detections} are not associated with a specific
astronomical object of any of the above classes.  Most of these will
be spurious (not representing real objects), some will be from solar
system objects for which orbits are not yet determined, some will be
from faint stars near the detection limits, and some will be from
short-term transients which have only been detected once.  DVO
maintains these detections until they have been associated with one of
the objects above.  DVO provides mechanisms by which individual
detections may be migrated back and forth between the orphan state and
association with an astronomical object.

DVO stores the information about the detection, the related objects,
and the images which provided the measurements.  For every detection,
DVO provides the mechanisms to link the detection back to the image
which supplied it.  DVO also provides the capability to determine the
images containing a specific location but for which no detection was
made.  The minimum set of information which must be carried for these
non-detections is the image and the associated object or orphan.

DVO also stores the relationships between various
photometric systems and the evolution of that relationship.  It
provides mechanisms to convert between the measured instrumental
magnitude of a detection with a specific filter, detector, and
telescope, and at a particular time and the implied magnitude in the
average Pan-STARRS photometry system, given a determined set of
calibrations.  It also provides the capability to convert magnitudes
in one system to the magnitudes in another system; an example of such
a conversion is between the average Pan-STARRS filter systems and the
various reference systems appropriate for those filters.

\subsubsection{Photometric systems and the DVO Photcodes}

One of the major roles of DVO is to relate different photometric
measurements made with different instruments and detectors together.
We may have observations made with the same basic filters, but using a
number of different detectors.  We may have observations from
different telescopes in similar filters.  We may have reference data
related to some filter, but obtained and published by other observers.
We would like to related these measurements together in optimal ways,
making use of whatever information we have available.  DVO provides
several mechanisms to enable these relationships.

We identify three distinct types of photometry measurements within
DVO:
\begin{itemize}
\item {\bf reference photometry}  These measurements are provided by
  external observers.  For reference photometry, we do not have access
  to very must information used to determine the magnitudes of the
  objects of interest.  We have the reference magnitudes corresponding
  to a type of filter, and presumably some information of the error on
  the measurement.  We might possibly know the epoch of the
  observations, but not necessarily.  
\item {\bf detection photometry} This is our primary measurement of
  interest: the photometry of objects measured from images which we
  have processed.  More specifically, the detection photometry is an
  instantaneous measurement from a specific image with well-known
  properties, such as exposure time, airmass, instrument source, etc.  
\item {\bf internal photometry} With the application of an appropriate
  zero point and other calibration terms, any detection photometry can
  be calibrated to represent a measurement in a well-known photometric
  system.  The internal photometry measurements are calibrated to be
  on a photometric system which represents a consistent system for a
  particular telescope or collection of data, minimizing the
  calibration transformations necsessary.
\end{itemize}

Defining the relationships between the different types of measurements
is part of the process of photometric calibration.  DVO uses the
concept of the 'photcode' to identify the source of the photometry,
and to define the relationships between different photometry sources.
A photcode identifies a photometric system: for the detection
photometry measurments, each combination of telescope, camera, filter,
and detector is associated with a unique photcode; there are also
unique photcodes for the internal photometry systems and any distinct
external reference source.  

As a concrete example, consider the Pan-STARRS PS-1 system.  There
will be three different cameras in use at different times: GPC-1,
TC-3, and the SkyProbe camera.  There are at least 6 filter systems:
{\it grizy} and {\it w}.  The SkyProbe camera has a single CCD, TC-3
has 16 different detectors, and GPC-1 has up to 64 different devices.
Each of these combinations is potentially a different photometric
system, so a different photcode is defined for each combination.
These photcodes would have names such as: GPC1.02.r (r filter with the
GPC1 camera and OTA 02) or SP1.00.g (SkyProbe 1, g filter).  These
($64 \times 6 + 16 \times 6 + 5 = 485$) photcodes are all identified
as 'detection' photcodes, specifying that detection photometry is
associated with them

There are also 6 different internal photometric systems of interest,
namely those associated with the 6 named filters, {\it grizy} and {\it
w}. Each of these 6 systems is identified with an internal photcode.
The internal photcodes are further distinguished as 'primary' or
'secondary', which specifies how the DVO system stores average
quantities related to these types of photcodes (see the discussion of
the tables below).  

Finally, there may be multiple external photometric systems of
interest, some of which are related to the major internal photometry
systems, some of which are not.  For example, the Pan-STARRS project
may refer to photometry from the SDSS secondary standards, the SDSS
data releases, Johnson photometry from Landolt (1992), observations
from 2MASS in $JHK$, USNO-B observations, and so forth.  Each of these
photometric systems is assoiciated with a different photcode; only
some of these are relevant to the detection or internal photometry
system.

Within DVO, the detection and internal photcodes each define a
relationships as well as a specific photometric system.  Associated
with each of these photcodes are the parameters of the photometry
transformation from the photometric system of the photcode to another
photometric system.  For the detection photcodes, the parameters
define the transformation to the equivalent internal photcode system.
The currently-defined transformation parameters consist of the
following photometry equation:
%
\[ 
M_\lambda = m_\lambda + C_\lambda + K_\lambda (\mbox{airmass} - 1) + \sum_{i = 1}^{i < N}
A_{\lambda,i} (\mbox{color}_\lambda - \mbox{color}_{o,\lambda})^i 
\] 
%
where $C_r$ represents the zero-point of the transformation, $K_r$
represents the slope of the airmass trend, $\mbox{airmass}$ is the
airmass for a given measurement, $\mbox{color}$ is the color of the
source of interest (as identified below), $\mbox{color}_r$ is the
reference color for sources in this photometry system, and $A_{r,i}$
is the coefficient of the $i$ power of the color difference.  Up to
fourth order color terms are currently allowed.  For any photcode, the
color is defined as the difference of the measurements in two other
photcodes, usually two 'internal' photcodes.  The photcode information
also specified the equivalent photcode to which the transformation corresponds.

For the detection photcodes, the target of the transformation must be
an internal photcode.  For the internal photcodes, the target of the
transformation is an external reference photcode system.  This
restriction implies that the internal photometry may only be
transformed (and thus compared with) a single external reference.
This is in fact the best practice as far as photometric calibration is
concerned: the 'standard' observations from different references
should always be treated as different photometric systems.  To allow
for the relationship of the internal photometry to multiple sources of
reference photometry, an additional set of photcodes are defined which
identify 'alternative' transformations for the internal photcodes.

It is important to note that not all of the photometry transformation
parameters identified above are relevant for each of the three major
types of photcode.  The detection photcodes will in general make use
of all of these elements, though the order of the color transformation
will hopefully be limited if the different devices are sufficiently
similar.  For the transformation from the internal photcodes, which
are derivative in some way of the detection photcodes, the airmass
component is invalid: for a single measurement, the
detection-to-internal transformation has already removed the airmass
trend; for an averaged internal photometric measurement, no single
airmass corresponds to the observations.  Finally, no transformation
parameters are defined for the reference photcodes at this time.

DVO provides methods by which these photometry transforamtions are
automatically applied.  The specific measurements (detection
photometry) are stored in the database tables as instrumental
magnitudes, and any operation which examines these measurements must
make use of the APIs to convert to an appropriate common system.  A
further complication to note is that the photcodes defined above are
static; they do not include any information about changes to the
system sensitivity.  This information is carried externally to the
photcode calibration information; the transformations defined by the
photcodes must be considered the {\em starting point} for any
photometric analysis.  An additional adjusment can be applied.  

The detections from a specific image may all have a 'calibration'
offset applied which bring the measured photometry into a common
relative system.  This calibration offset is associated with the image
and may be a function of position on the detector.  The tables which
carry the individual measurements also include the calibration
magnitude appropriate for each measurement to speed up the application
of this offset.  In a well-calibrated collection of photometry, all of
the detection measurements will have a measured calibration magnitude,
yielding a collection of internal photometry measurements which are
all consistent.  An additional piece of information is the zero-point
history, which tracks the system-wide variations in the average
sensitivity.  The zero-point history can be used to predict the
calibration magnitudes for any observation which is not tied directly
via relative photometry to the rest of the photometric observations.

Putting all of these pieces together, the photometry APIs in DVO can
be used to return any of the following types of photometric
measurements:
\begin{itemize}
\item raw instrumental magnitudes for any detection

\item 'catalog' magnitudes, applying only the airmass and static
  zero-point calibrations to a detection magnitude; this is useful to
  test the detector-color transformation.

\item 'system' measurements, applying the complete static
  transformation for a detection magnitude to the internal photometry
  system; for photometric weather and no zero-point variations, this
  would be a measurement in the internal photometry system.

\item 'relative' magnitudes, applying the measured calibration offset
  to the calibrated detection magnitude determined above; in a
  well-calibrated system, this represents a consistent internal
  photometry measurement.

\item 'calibrated' magnitudes, correcting the measure detection
  photometry by applying the transformation from the internal
  magnitude system to the external reference magntiude system.

\item 'average' magntiudes, the raw internal photometry magnitudes
  (note the distinction between the 'average' quantities, which are
  derived from a collection of detections an the 'relative' quantities
  which represent an instantenous measurement in the same system).

\item 'reference' magnitudes, in which the 'average' internal
  photometry values are transformed to the refernce magnitude system.  
\end{itemize}
The complexity of these transformations is necessary to allow the
examination of the trends of actual measurements with external
parameters.

\subsubsection{DVO Database Tables}

\begin{figure}
\begin{center}
\resizebox{4.5in}{!}{\includegraphics{pics/dvo.01.ps}}
\caption{\label{fig:DVOtables} \small Data types managed by DVO}
\end{center}
\end{figure}

Figure~\ref{fig:DVOtables} illustrates the data managed by DVO, and
Table~\ref{tab:DVOtables} provides a complete listing.  The contents
of these tables are outlined in Appendix~\ref{sec:DVOTableContents}.
Below, the use of these tables by DVO software is discussed below.
Several of the tables are not just simple tables in the database but
are instead table groups divided into many subtables, each of which
represents a portion of the sky (a {\tt region}).  These subtables may
also be distributed across different computers to distribute the
processing load.

\paragraph{Sky Regions Table}

The {\tt Regions} table is used to subdivide the tables of images,
objects, and detections, etc, as discussed above.  DVO
divides the sky into a hierarchy of regions (portions of the sky) each
of which is in turn subdivided into smaller portions.  Since nearly
all interactions with DVO performed by the IPP are limited
in spatial coverage, subdividing the tables allows a specific
interaction to search only a small subset of the data.  The table of
images is the smallest of the three; the table of detections is likely
to be the largest.  As a result, the {\tt Images} table group will be
subdivided at a shallow hierarchical level, while the {\tt Objects}
and {\tt Detections} are subdivided on deeper (more finely sampled)
levels.  The {\tt Regions} table defines the boundaries of the sky
regions and specifies if the region corresponds to an {\tt Images}
table, an {\tt Objects} table, and/or a {\tt Detections} table.  It
also specifies which regions in the next level of the hierarchy are
contained by the region, and which parent region it belongs to.  In
addition to improving the spatial access to the image, object, and
detection data, the {\tt Regions} table allows for multiple computers
to serve the database tables.  The region file specifies the machine
which stores the specific table.  Figure~\ref{fig:DVOskyregions}
illustrates schematically the subdivision of the sky and the
association between different levels of the hierarchy with different
subtables.

\begin{figure}
\begin{center}
\resizebox{6in}{!}{\includegraphics{pics/dvo.02.ps}}
\caption{DVO Regions and Image / Object tables}
\label{fig:DVOskyregions}
\end{center}
\end{figure}

\paragraph{Images Table Group}

The {\tt Images} table group lists all of the images which provided
the data in DVO.  These tables are subdivided by region on
the sky.  In general, the images listed in this table correspond to
the Chips.  This group of tables includes sufficient astrometric
parameters to represent the coordinates of the detections to a
sufficient accuracy.  Parallel to the Images table is the Mosaic
table.  This table is very similar to the Images table, but defines
the Mosaic which corresponds to a group of Images.  The parameters
include the astrometric information needed to define the camera
distortion.

\paragraph{Image Overlaps Table Group}

The specific subtable of {\tt Images} which contains a given image is
the one which contains the center pixel of that image.  An additional
table group, {\tt Image Overlaps} (with the same subtable organization
as the {\tt Images} subtables), lists images which overlap that
specific subtable.  Thus, given a particular coordinate, in order to
find that images which overlap that coordinate, it is necessary to
search the images in the {\tt Images} subtable which includes that
coordinate, and all images in the {\tt ImageOverlaps} subtable for
that coordinate.

\begin{table}[hb]
\begin{center}
\caption{DVO Database Tables\label{tab:DVOtables}}
\begin{tabular}{ll}
\hline
\hline
{\bf Table Name} & {\bf Description} \\
\hline
Images               & The images that have objects in the DB. \\
Image Overlaps       & Image regions which are touched by specific images. \\
Objects              & The objects --- average properties of multiple detections of the same object. \\
Average Magnitudes   & Average photometry in multiple filters \\
Solar System Objects & Identification of solar system objects \\
Matched Detections   & Detections of sources in an image identified with an Object. \\
Orphaned Detections  & Detections of sources in an image not identified with an Object. \\
Non-detections       & Non-detections of objects in an image. \\
SkyRegions           & spatial distribution of tables \\
Filters              & Filters understood by the system. \\
Photcodes            & Transformations between different photometric systems \\
Zero Points          & History of Zero-point \& Airmass terms \\
Distortion Models    & History of Optical Distortion terms \\
Database Hosts       & computers used to store the tables \\
\hline
\end{tabular}
\end{center}
\end{table}

\subsection{Objects Table Group}

\begin{table}
\begin{center}
\caption{DBO Detection Classes \& Object Parameters\label{tab:APdetections}}
\begin{tabular}{lrrrr}
\hline
\hline
Object Parameter & P2 & P4S & P4D & SS \\ 
\hline
PSF x,y, covar, $\alpha,\delta$               & + & + & + & + \\
PSF mag, $\sigma_{\rm mag}$                   & + & + & + & + \\
star/gal sep                                  & + & + & + & + \\
$\sigma_x$, $\sigma_y$, $\theta$              & + & + & + & + \\
local sky data                                & + & + & + & + \\
Petrosian R, M, $R_{50}$, $R_{90}$            & - & + & - & + \\
S\'ersic R, M, AB, $\phi$, $\nu$              & - & + & - & + \\
W.L. $\gamma_1$, $\gamma_2$, pol. terms       & - & - & - & + \\
exp. spaced aps., Poisson noise, variance     & - & - & - & + \\
\hline
\end{tabular}
\end{center}
\end{table}

The {\tt Objects} table group (also divided by region) stores the
average parameters for each astronomical object.  Certain details of
this table have not yet been specified.  In particular, objects with
significant parallax and/or proper motion may potentially be stored in
a distinct table.  Solar system object identifications, to the extent
average properties are maintained in DVO, will certainly
be stored in a separate table.  

\paragraph{Average Magnitudes Table Group}

A related table, also divided into the same regions, is the {\tt
Average Magnitudes} table.  In this table, there are multiple rows per
object, one for each of the primary filters of interest for which
photometric averaging is performed.  This organization makes the
number of primary (averaged) filters a configurable value.

\paragraph{Matched Detections Table Group}

The {\tt Matched Detections} table stores all of the measurements of
astronomical objects on specific images.  This table includes all
detections associated with the average {\tt Objects}.  As discussed
below, bright objects (above a configuration-specified signal-to-noise
level) are defined object even if only one detection has been found at
that position.  Faint orphaned objects are not added to this list or
the list of objects.  The different types of detections (P2,
P4$\Delta$, P4$\Sigma$) are distinguished by their photometry codes.
(This is only valid if DVO does not store different
quantities for these types of detections.)

\paragraph{Orphaned Detections Table Group}

The {\tt Orphaned Detections} table stores the detections which have
not been correlated with an existing object.  This table is only
populated for objects below a configuration-specified signal-to-noise
limit (e.g., 5$\sigma$).  Bright orphaned detections are assigned an
object and added to the {\tt Matched Detections} table.

\paragraph{Non-detections Table Group}

The {\tt Non-detections} table stores information about detection
failures for each object.  If an image is added to the database which
overlaps an object but the object is not detected, an entry is made in
this table.  In practice, this table may store only the most recent
non-detection and the total number, or a similar reduced set of
non-detection statistics.

\paragraph{Other Reference Tables}

The {\tt Filters} table identifies all of the physical filters
(specific pieces of glass) known to the system.  A related table, {\tt
Photcodes}, defines relationships between photometry systems.  A
photometry system may consist of a detector, telescope, and specific
filter, or it may be a derived photometry system.  The {\tt Database
Machines} table identifies all of the computers available to DVO.

\subsubsection{Database Table I/O}

\begin{figure}
\begin{center}
\resizebox{4.5in}{!}{\includegraphics{pics/dvo.03.ps}}
\caption{\label{fig:DVOformats} \small DVO Table I/O }
\end{center}
\end{figure}

DVO allows for a flexible representation of its data on disk.  Data
may be written to disk in one four possible mode: RAW, FITS MEF, FITS
SPLIT, and MYSQL.  These modes define the overall organization of the
data on disk.  In the RAW mode, the data is written to disk in a
pseudo-FITS table format which consists of a simple FITS header
describing the layout followed by the binary data in a block.  This
storage mode is maintained for historical reasons.  There are also two
types of FITS modes in which the data tables are written as valid FITS
Binary Tables.  In the SPLIT format, every data table is written as a
separate file, while in the MEF format, the object and detection
tables are bundled together into a single FITS file with multiple
table extensions.  The MEF format has the advantage of minimize the
proliferation of files, while the SPLIT format is required to make use
of the fastest read/write capabilities of DVO.  DVO makes use of these
raw data formats as a throughput risk mitigation strategy.  As
discussed below, this strategy has proven very successful.

There are also multiple formats in which the data may be stored.  The
different formats define which specific database table columns are
stored and with what numerical format and precision.
Figure~\ref{fig:DVOformat} illustrates the conversion process which
DVO performs when loading in the data.  When DVO loads data from a
file-based table (FITS or RAW), it first loads from the disk file into
a data structure representing the external format in use.  The
external structure is then converted into the internal format. The
internal structure is always specified to be the superset of all
external data formats.  This capability allows DVO to maintain
backwards compatibility with data tables written with early versions.
As DVO is extended and new elements are added to the tables, it is
only necessary to define the methods to convert the new internal table
into the external table.  In addition, DVO makes use of autocoded
table manipulation and I/O APIs which are generated for each data
structure based on a descriptive table.  This makes it easy to add new
data types and input/output methods without significant re-coding.

\tbd{DVO mysql table storage is not yet implemented}

\subsubsection{addstar : Insert Image \& Detection Set}

\begin{figure}
\begin{center}
\resizebox{4.5in}{!}{\includegraphics{pics/dvo.04.ps}}
\caption{\label{catalog} \small a figure }
\end{center}
\end{figure}

One of the most basic operations needed by DVO is to insert a
collection of detections derived from a specific image, and add the
definition of that image to the database.  This operation is critical
in terms of the processing throughput.  After the detections have been
assigned to the appropriate regions, they are matched against all
objects in the {\tt Objects} table.  Matches are performed only on the
basis of positional coincidence, using a matching radius which may
depend on the image astrometry errors, or may be a fixed distance.
Any matched detections are added to the {\tt Matched Detections}
table.  Any unmatched detections brighter than the Faint Detection
cut-off are specified as a new {\tt Object} and also added to the {\tt
Matched Detections} table.  Any faint unmatched detections are added
to the {\tt Orphaned Detections} table.  This division is important
because it allows the automatic association of new detections with
existing bright objects while limiting the I/O volume required to make
the detections.  In general, there will be many fewer {\tt Objects}
than {\tt Detections}, and there will be fewer bright orphans than
faint orphans.

A wide range of options are available to addstar.  These can be used
to modify the object matching rules, to reduce the number of tables
which are updated, to specify the output data format, and so forth.  A
few options modify the behavoir in substantial ways, as discussed in
the two sections below.

\tbd{flesh out discussion of the options}

\paragraph{Insert Reference Objects} 

\code{addstar -ref (filename)}

This mode of addstar reads a text file and adds the listed objects to
the database as a reference photcode type.  A collection of reference
objects are added to the database as a collection of detections.  The
reference photometry should in general be given its own photometry
code.  The reference data is different from the image detection set
because the associated image information is not included.  Thus, no
corresponding images are added to the database.

\paragraph{Insert Catalog Objects} 

\code{addstar -cat (name) -region ra ra dec dec}

In this mode, any of several all-sky or large-scale reference catalogs
are used for the input sources.  The catalog objects are added to the
database as reference objects.  The valid catalogs consist of 2MASS,
USNO, GSC.  Tycho and USNO-B will be added shortly.  Specific
photcode names are defined for each of these catalogs, and must be
appropriately requested and defined in the photcode table.  The
optional region restriction limits the insert to a subset of the sky.
The user does not always want to add 50GB of 2MASS detections to any
DVO database... 

\paragraph{Addstar Client/Server Interactions}

DVO currently uses stand-alone programs which are run from the command
line (like addstar, or the programs listed below), or it works with
the interactive DVO shell, which allows the user to query portions of
the database.  These programs all interact with the database tables
directly, making use of file locking to prevent conflicts.  

Unlike the other DVO programs (currently), it is possible to run
addstar as a client/server system.  In this configuration, the program
\code{addstard} is launched to run in the background as a server.  It
monitors a socket waiting for clients to contact it.  The client
program, \code{addstarc} appears to the user identical to the
stand-alone addstar.  However, rather than directly insert data into
the database, \code{addstarc} contacts the addstar server and sends it
the detections and associated image data (along with the information
about the user options).  The daemons accepts the incoming data and
then loads this data into the database, just as the stand-alone
addstar does.

The purpose of the addstar client/server design is three-fold.  First,
the client can be used by processes to send data to the DVO database
and then immediately exit.  The addstar loading process is one of the
more time-critical functions within the IPP.  However, unlike the
other portions of the IPP, the addstar processes must operate in
serial, at least when they are updating the same portion of the sky
(or the image table).  If the IPP analysis routines all needed to run
the stand-alone addstar program, they would eventually block waiting
for each addstar to complete, preventing other processing from
continuing.  The addstar client / server model allows the processing
node to invoke the addstar client, sending the data to the addstar
server.  The addstar server will then be the entity that manages the
serialization of the incoming data stream.  The addstar server has two
threads which run in parallel.  One thread monitors the socket and
accepts new data sets from addstar clients, adding the data to an
internal queue.  The other thread pulls data off of the queue and
updates the database with the data.  

A second advantage of the client/server interaction is that only the
new detections need to be sent across the network.  To update the
database, addstar must load the average objects for the region from
the database tables.  In the stand-alone mode, the addstar program
loads this data via NFS across the network from whatever device stores
the addstar tables.  In the client/server model, the addstar server
always runs locally on the machine which holds the database tables.
Thus, for the server, all database access is local disk access.

The final advantage of the client/server model is that it enables the
parallel database model, which is not yet implemented as of Jan
2006. In this model, there are multiple addstar servers.  Each one has
a fraction of the sky in the local tables.  The identification of
which table is managed by this host/addstar server is stored in the
SkyRegion table.  The addstar server simply accepts incoming
detections from the addstar clients.  Any detections which it receives
which fall within the boundaries of tables that it manages are updated
as normal.  The server then identifies the other addstar servers which
are responsible for the other detections.  It then sends these
detections to those servers using the same socket communication used
by the addstar clients.  The addstar server must also be ready accept
detections from other addstar servers.  This relationship is
completely parallel, and any addstar client may send its data to any
addstar server, letting the servers hash out who owns what.  The only
difficulty with this model is in handling sources near the boundaries
of the tables.  Note that this issues exists whether those tables are
distributed across multiple machines or not.

Addstar uses the following strategy to handle detections on the table
boundaries.  Detections are first added to each table completely
ignoring the neighboring tables.  A detection which is close to the
boundary may either be associated with an average object contained
within the table, or not.  If it is, the detection is associated with
that average object.  If not, a new average object is created at the
location of the detection.  So far, this process is identical to the
behavoir in the middle of the table.  One a longer time-scale, a
process is run which mediates the table boundaries. In this analysis,
the two neighboring tables are simultaneously examined.  The border
region, in a strip wider than the correlation radius, is examined in
detail.  If two objects within the border region fall within 2x the
correlation radius of each other, their individual detections are
re-examined.  These detections are re-added to a temporary table which
encompases the overlap.  the resulting objects will in general have
detections from either side of the boundary.  The average objects are
kept within the table as normal, but the detections are allowed to
migrate between the tables to stay with their object.  \tbd{this
boundary cleanup process is not implemented to date}.

\subsubsection{Relphot : Relative Photometry Analysis}

This operation uses the overlaps of images and multiple observations
of the same objects to determine the relative photometry zero-points
for a collection of images.  This is a task that is run much more
infrequently than the object insertion tasks.

The relphot analysis is currently performed with a single Sky region
as the starting point.  All images (or all chips from all mosaic
iamges) which overlap the sky region are identified in the image
table.  This set of images are considered set A.  Next, all skyregions
which are overlapped by all of these images are selected.  Finally,
all additional images which overlapped the new regions only are
selected.  These are considered as image set B.  The image selections
are also restricted to images of a single, user-selected photcode.  

All of the objects and detections which are contributed by the images
in sample A are extracted from the average and measure tables.  Only a
subset of the detections for which the S/N is greater than a
user-selected limit are kept.  Other restrictions, such as time range
or instrumental magnitude ranges may also be specified.  The
collection of average objects, their detections, and the images from
which they were derived now define a system of photometry equations.
In this system, every image has a calibration offset magnitude
($M_{cal}$), every object has an average magnitude in a relative
system ($M_{rel}$), and every detection of that object has a magnitude
defined by the equation $M = M_{rel} + M_{cal}$.  The goal is to solve
for the values of $M_{ref}$ and $M_{cal}$.  

There are two points to note about this operation.  First, the system
of equations is generally much too large to solve directly; we must
use an iterative technique to converge on a solution. Second, it is
important in the analysis to use robust averaging and identify
detections, stars, or images which are deviant in some way.  These
should be marked and given set weight in the solution.  These cases
may represent poorly measured objects (perhaps detections on or near a
bad column), variable stars, and images obtained in poor weather
conditions.

Relphot can also be used to determine the mosaic grid used to generate
photometrically corrected flats (-grid option).

\tbd{fill out this discussion in the analysis section on the
astrometric and photometric reference catalog}.

\subsubsection{Uniphot : Zero Point Analysis}

This operation uses the time history of relative photometry zero
points for images and the spatial overlap information to determine a
best set of image zero points which have a specific time scale for the
atmospheric stability.  This analysis is used after relative
photometry has been determined for data in DVO.  This analysis
currently is defined to unify the zero points of a collection of
disjoint regions; additional modifications will be needed to
simultaneously determine consistent zero points and relative
photometry corrections for a collection of images distributed over a
large range in space and time, but still with significant
overlap. 

\tbd{fill out this discussion in the analysis section on the
astrometric and photometric reference catalog}.

\subsubsection{relastro : Global Astrometry Analysis}

This operation uses the reference and image detections to improve the
astrometric reference catalog.  It determines an improved optical
distortion model for the camera and static astrometry model
components, and then applies the improved astrometric solutions to the
observations to yield high-quality astrometry for the average object
positions.  The astrometry model includes: (1) field distortion
introduced by the telescope optics, which is a smoothly-varying
function of the field position relative to the center of the telescope
boresite coordinates.  (2) focal plane geometry, which includes the
chip positions and rotations in the focal relative to the boresite,
along with chip-dependent plate-scale modifications needed to
represent tilts or warps of the individual detectors relative to the
ideal flat focal plane.  

\tbd{fill out this discussion in the analysis section on the
astrometric and photometric reference catalog}.

\subsubsection{DVO shell}

The DVO Shell is a user-tool for examining the visualizing data stored
by DVO. The DVO Shell uses the Opihi shell language structure (see
also PanTasks, Section~\ref{pantasks}), which provides a rich data
analysis language.  The shell language provides the user with
capabilities to define new commands, set and manipulate scalar,
vector, and image structures, and plot 2D graphics, including
projections of the sky.  In addition, the DVO shell is aware of the
DVO data tables and provides access mechanisms to these tables.  The
following is a brief overview of these table access features.

DVO provides several ways to access the information stored in the
database. Several simple commands allow the user to extract 1-D
information directly from one of the primary database tables. The
fundamental such commands are:
\begin{itemize}
\item imextract
\item avextract
\item mextract
\end{itemize}

These commands allow the user to extract data from one of the columns
represented by the image table(s), the average object tables, or the
measurement tables.  The extraction places the resulting data into a
vector data elements, which may be used to make plots or perform
analysis.  The user may constrain the query with spatial selection, by
photcode, by time ranges, and so forth.  Some examples:
\begin{verbatim}
   avextract all ra : select ra for all objects in displayed region
   avextract all g  : select g magnitudes
   avextract all mag -photcode r : select r magnitudes 
   avextract all Xm -photcode r  : select chisq values for r average mags
\end{verbatim}

Beyond these basic vector extractions, the user may perform more
complex extract operations such as color-based selections.  For
example, color-color diagrams can be easily made by extracting the
colors from the average or measurement tables and plotting the
resulting vectors.  The \code{ccd} commands extracts a specified pair
of colors for all objects with that color pair from the specified data
region.  Similarly, the command \code{cmd} extracts a color and a
magnitude into a pair of vectors.  Both commands may specify any of
the different types of magnitudes (relative, calibrated, etc)
discussed above.

An additional class of DVO Shell commands perform more complex
graphical operation.  For example, the command \code{images}, plots
the images which the specified region on the plotting too.  Other
commands allow the user to extract the images or the database tables
which overlap specified locations.

\begin{figure}
%\resizebox{4.5in}{!}{\includegraphics{pics/polar}}
\caption{\label{polar} \small Map of
the sky in polar project, and images added to database. }
\end{figure}

\begin{figure}
%\resizebox{4.5in}{!}{\includegraphics{pics/fullsky}}
\caption{\label{allsky} \small Map of the entire sky, and images added to database. } 
\end{figure}

Some examples of using the DVO shell to perform visualization are
given in Figures~\ref{polor} and \ref{allsky}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{PanTasks : the IPP Scheduler}
\label{sec:pantasks}

PanTasks is the IPP tool which manages the sequencing of data analysis
steps and, with the related tool `PControl', distributes the data
processing across a cluster of computers. \tbd{describe the 'opihi'
shell-scripting system}

The purpose of PanTasks is to manage the automatic construction and
execution of inter-related (often repetative) operations.  PanTasks
uses a set of rules to define UNIX commands, and their corresponding
command-line arguments, to be performed on some regular, repeated
basis.  The utility of PanTasks is that it can easily define and
manage an analysis system which is completely state-based, as opposed
to an event-driven system.

Consider, for example, a telescope which obtains a collection of
images over the course of a night. Every minute or two, it takes an
image and writes the image to some disk. An event-driven analysis
system would involve having the telescope initiate a process at the
end of the exposure. This process would perform an analysis, write
some output, then send trigger another process. This type of operation
works very well for a simple set up with reliable hardware. Such a
system becomes more difficult to maintain when hardware failures occur
or when multiple systems need to interact with each other. When
failures occur, the triggering information (the events) is easily
lost, thus some mechanisms are needed to detect these failures and
either re-send the trigger or send an alternative failure-mode
trigger. Or, if two systems need to interact, one or the other system
must block for results from the first. Stopping and restarting such an
analysis system is very delicate since the appropriate triggers must
be set up some how, eg by noticing which images have not succeeded and
restarting them at the appropriate stage. All of these types of
methods of handling complexity and failures are essentially
state-based rules. PanTasks allows the easy definition of a totally
state-based analysis system.

In a state-based system, some mechanism examines the state of the
system and decides which actions to perform based on the current
state. In the illustration above, the mechanism could examine the
images available (either by examining the disk or by examining the
state of a data table) and decide to perform an operation based on
what images are available. This makes it very easy to handle
complexity and errors. If an analysis fails, the state either is not
successfully updated or the error state is recorded, both situations
being easy to detect and easy to handle. Restarting the system simply
involves starting the state-monitoring mechanism. Combining results
from multiple input sources simply involves watching for the multiple
inputs to be available. PanTasks provides a mechanism to define state
monitors, and to define the actions which are performed when those
states occur. PanTasks action consist of initiating UNIX commands, where
the arguments of those commands may depend on the results of the state
tests.  

\subsubsection{Tasks vs Jobs}

The two basic units of PanTasks operation are the 'task' and the
'job'.  A 'job' is simply a command the user would execute on a
command line; it consists of a command along with optional command
line arguments.  A 'task' is a generic description of a type of job in
which the details of any specific piece of data are omitted.  The task
defines the UNIX command which corresponds to the job, and it provides
rules for determining the identity of data for the job.  The task also
defines tests which are used to decide if the job may be executed.
Finally, it defines a polling frequency in which PanTasks should
attempt to construct a new job for the task.

For example, we may want to regularly copy files from one location to
another.  Perhaps the name of the next available file is available in
a database table, and can be retrieved with the command 'nextFile'.
Perhaps we wish to check for new files every 60 seconds.  Thus, the
task is to copy files, while a specific job of this task might be of
the form \code{cp newfile newpath}.  With PanTasks, we would define a
copy task somewhat like the following:

\begin{verbatim}
  task copyfile
    periods -exec 60.0

    task.exec
      $file = `nextFile`
      if ($file == "none")
        break
      end
      command cp $file $newpath
    end

    task.exit 0
      queuepush copied $file
    end

    task.exit 1
      queuepush failure $file
    end
  end
\end{verbatim}

In this simple example, the task is attempted every 60 seconds.  If
there is no new file (output of 'nextFile' is 'none'), then no job
results from this task.  If there is a new file, the copy command is
performed.  This example is deceptively simple because one could
easily imagine writing a stand-along program which performs the same
thing.  The important advantages of using PanTasks for this type of
operation are: 
\begin{itemize}
\item the output from one job can be used to spawn a number of other tasks.
\item the success or failure state of each job can be used to spawn
  other tasks.
\item the details of the job and data test are kept separated from the
  rules which connect tasks together.  
\item the relationships between tasks are kept together in a single
  location.
\end{itemize}

In addition to these organizational advantages, PanTasks also provides
a direct connection to the tool for monitoring parallel jobs,
pcontrol.  Thus the jobs spawned by PanTasks can be defined to run in
the background locally or on any of the computers in the parallel
processing cluster.

\subsubsection{Parallel vs Local Job Processing}

Jobs which are generated by PanTasks tasks may either be run locally
(forked in the background on the same machine as PanTasks) or run on
the IPP parallel process controller, pcontrol. The default is for the
job to be run locally. If a job should be run on the parallel
controller, this can be specified by including the command host
(hostname) in the definition of a task. If the value of (hostname) is
'anyhost', then pcontrol may select any of its host computers to run
the job according to its own rules. If the value of (hostname) is one
of the computers managed by pcontrol, then that machine will be
selected for the job, if it is available. This amounts to a preference
to use that machine, but pcontrol is allowed to substitute a different
machine if it chooses. If the host command is given the option
-required, then pcontrol is forced to use the named host, even if the
machine is down, unknown, or otherwise unavailable. If the machine is
not available, pcontrol will simply hold onto the job until the
machine is available or the job is deleted. Note that PanTasks may
delete jobs from pcontrol if they remain pending for too long.

It is possible to interact directly with the parallel processor to
examine the current status, halt the parallel processor, etc. Commands
to the parallel processor are defined under the controller
command. The following controller commands are available:

\begin{itemize}

\item controller host (command) (hostname): Manage the parallel
  controller collection of hosts. This command can be used to add a
  new host, the delete one of the existing hosts, to turn a host on or
  off, and to check the status of a host

  \begin{itemize}
  \item controller host add (hostname): add a new host.

  \item controller host delete (hostname): delete a host.

  \item controller host on (hostname): tell pcontrol that the host is on.

  \item controller host off (hostname): tell pcontrol that the host is off.

  \item controller host retry (hostname): tell pcontrol to retry the host connection.

  \item controller host check (hostname): check the current status of a host. 
  \end{itemize}

  \item controller exit: stop controller execution.

  \item controller status: report controller current status.

  \item controller check: check job or host status.

  \item controller output: print accumulated messages from the controller. 
\end{itemize}

It is also possible to specify a host for a task which has not been
identified to the controller. If such a host is required, the
controller will simply keep the associated jobs in the pending state
until such a machine exists. See the pcontrol section below for
further discussion of the controller manipuation of jobs and hosts.

\subsubsection{Task Restrictions}

Tasks may have restrictions on when they create jobs and how
frequently they create jobs. The task command trange is used to
specify a valid or invalid time range for a task. A valid time range
limits the task evaluation to that time period. An invalid time range
excludes task evaluation from the time period. Any number of time
range restrictions may be defined, and the union of all restrictions
will define if a job may be created. By default, the time range is an
inclusive time range: the task is evaluated only if the current time
falls within the specified time range. Alternatively, if the -exclude
flag is given, the time range is exclusive, in which case the task is
not evaluated if the current time falls within this range.

The time range may be given as a range of absolute dates as follows:

\begin{verbatim}
 trange YYYY/MM/DD,HH:MM:SS YYYY/MM/DD,HH:MM:SS 
\end{verbatim}

where the two dates specify the start and end of the time range. In
either of these date representations, the least-significant elements
of the date and time may be dropped, defaulting to 00 (in the case of
hours, minutes, and seconds) or 01 (in the case of day and
months). Rather than specifying an end date, it is also valid to
specify a time interval from the starting date. The time interval is
specified as a number followed by a unit indicated by a single letter:
d (days), h (hours), m (minutes), s (seconds).

The time range may also be specified as a repeated period of time,
either as a time of day or a day and time of week. In the first case,
the time range is specified as follows:

 
\begin{verbatim}
 trange HH:MM:SS HH:MM:SS
 \end{verbatim}


where again the least-significant elements may be dropped and default
to 00. This type of restriction defines a time range which is valid
every day. The alternative is to specify a time range within the week,
in the following form:

\begin{verbatim}
 trange DAY@HH:MM:SS DAY@HH:MM:SS
\end{verbatim}

where the value of DAY may take on any of the three letter day-of-week
names (Sun, Mon, Tue, etc). This restriction specifies a start and end
time within a week which is evaluated for each week.

Below are several examples of valid time range restrictions

\begin{verbatim}
 trange 2005/01/01 2005/12/31   (only run during 2005!)
 trange 18:00 00:00             (only run from 6pm until midnight)
 trange 00:00 06:00             (only run from midnight until 6am)
 trange Mon@08:00 Fri@17:00     (only run between Mon morning and Fri afternoon)
 trange -exclude 12:00 13:00    (skip 1 hour from noon)
\end{verbatim}

Note that the current definition of trange does not include time zone
information. This means that all times are relative to UT. This should
be addressed by adding a timezone environment variable to PanTasks and
by allowing the trange to define a timezone offset.

It is also possible to restrict the total number of jobs which are
spawned for a given task. This is done with the nmax command, which is
given as part of the task definition. Once a task has constructed nmax
jobs, it stops task evaluation. It is possible to redefine the value
of nmax at any time by redefining the task. Any time the task is
redefined, the new values for any task concept will override the
existing values for the task concept.

\subsubsection{Inter-Task and Inter-Job Communications}

There are several ways in which the results of jobs may be used to
influence other jobs. These include:

\begin{itemize}
\item external communications
\item job exit status
\item job stdout/stderr parsing 
\end{itemize}

It is always possible for the interprocess communication to be
performed externally: all jobs may simply write results to an external
data source which is queried as part of the task evaluation. PanTasks
may interact with UNIX programs using Opihi system interaction
functions. These interaction methods include: the backticks for
setting Opihi variables:

\begin{verbatim}
 $variable = `UNIX Command`
\end{verbatim}

The exec command (which executes a UNIX command) and the backticks
both receive the UNIX command exit status, setting the variable
\code{$STATUS}. It is also possible to set a variable list to the
output of a UNIX command:

\begin{verbatim}
 list var -x "UNIX Command"
\end{verbatim}

In this last case, the values \code{$var:0 - $var:N-1} are set to the
value of the stdout lines from the UNIX command, and the value
\code{$var:n} is set to the number of output lines.
% $

Fine-grained control over the job exit status is available with the
task.exit macro command. This allows a task to define an exit macro
which is performed for different exit status conditions. The argument
to the task.exit command is the exit status value which triggers the
macro. This may consist of any valid numeric exit status value
(0-255). It may also have the value crash, in which case the macro is
executed if the program exited as a result of a signal (ie,
segmentation fault, etc). Finally, if may have the value default, in
which case, the macro is run if no other macro describes the exit
status.

Jobs may transmit their results back to PanTasks for further
evaluation through the standard output and standard error
streams. Whenever a job exits, the complete stdout and stderr streams
from the job are pushed onto the PanTasks queues \code{stdout} and
\code{stderr}. The job exit macros may then parse these queues, moving
the results into other PanTasks / Opihi data containers (queues,
variables, vectors, whatever is appropriate). Note that currently, the
output data is simply pushed onto these output queues. It is currently
the responsibility of the PanTasks programmer to use or dispose of the
data in these queues. This may change in the future: the queues may be
flushed for each job completion.

\subsubsection{PanTasks Monitoring Loop}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85,angle=-90]{pics/pantasks.01.ps}
\caption{\label{MonitorLoop} PanTasks Monitor Loop}
\end{center}
\end{figure}

Figure~\ref{MonitorLoop} illustrates the operation of PanTasks.
Currently, PanTasks is run as a stand-alone foreground process, not a
server.  In this current operating mode, PanTasks accepts commands
from the user (left side) via a GNU readline interface.  Readline
provides a mechanism to schedule an background process which is
executed on a regular basis, or in the interval after each keystroke.
This background processing is represented as the loop on the right.
In this loop, PanTasks checks on the status of pcontrol and on any
locally spawned background jobs.  It also checks the list of tasks for
tasks which are ready to be executed.  The tasks and the background
jobs are kept in rotating queues.  Every time the loop is processed,
PanTasks runs through a number of the background jobs and tasks,
attempting to evaluate as many as possible, but stopping after a
limited number of milliseconds.  This test stage cycles through the
tasks and/or jobs in their queue, but stops if it examines all of the
entries in the queue.  If this testing process runs out of time before
the entire queue is searched, it leaves the sequence intact for the
next pass.  The timeout is set to keep the keyboard small enough to
keep the human interaction from being troublesome.

\begin{figure}
\begin{center}
\includegraphics[scale=0.85,angle=-90]{pics/pantasks.02.ps}
\caption{\label{TaskLoop} PanTasks Task Check Loop}
\end{center}
\end{figure}

Figure~\ref{TaskLoop} shows the interactions performed for each check
of the list of tasks.  The task timers are examined to see if the
specific task is ready to be executed.  If so, then the task exec
macro is executed.  If this macro exit status is false, the loop goes
to the next task.  If the task exec macro is true, the job command is
constructed and the job is submitted to the correct location, either
the controller (pcontrol) or to the queue of local, background jobs.

\begin{figure}
\begin{center}
\includegraphics[scale=0.85,angle=-90]{pics/pantasks.03.ps}
\caption{\label{JobLoop} PanTasks Job Check Loop}
\end{center}
\end{figure}

Figure~\ref{JobLoop} shows the interactions performed for each check
of a single background job.  The job timer is checked to see if the
job has timed out.  Next, the job run status is examined to see if the
job has finished or not.  If the job has finished, the appropriate
exit macro is executed and the job results are returned to the rest of
the PanTasks data structures (ie, stderr and stdout queues are filled
out).  Users should not define exit macros which perform long running
jobs, or PanTasks will become quite sluggish to keyboard strokes.

PanTasks also checks the current status of pcontrol on each loop.
In this test, PanTasks requests from pcontrol a list of the jobs which
have finished.  It then attempts to harvest the finished jobs one by
one and place the results in the approrpiate locations.  This loop is
also limited to a fixed amount of time; any pcontrol jobs which remain
unharvested are left for the next pass by PanTasks.

\subsubsection{Running the scheduler}

Once a set of tasks has been defined, the scheduler can be
started. The scheduler will run in the background, at regular
intervals examining the collection of tasks and jobs. In these
periods, the scheduler attempts to construct new jobs and checks on
the status of jobs which may have finished, either locally or on the
controller. To start the scheduler, give the command run. To stop the
scheduler, given the command stop. The current status of the
scheduler, controller, and any jobs which have been spawned are listed
with the status command.

It is also possible to kill or delete individual jobs by hand with the
commands kill (jobID) or delete (jobID).  Other features

\subsection{pcontrol : the PanTasks parallel controller}

The IPP uses a group of computers to store and process images and to
manipulate collections of detections. These computers perform any of a
large number of analysis stages or other processing tasks without
significant interprocess communication. It is necessary to have a
mechanism which initiates computing tasks on the different computers,
which monitors the tasks as they are executed, which handles the
output and the errors from these tasks, and which reacts to the
failure of any of the computing nodes. The system responsible for the
tasks in the IPP is pcontrol.

\subsubsection{Host States}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85,angle=-90]{pics/pantasks.04.ps}
\caption{\label{HostStates} PanTasks Host States}
\end{center}
\end{figure}

pcontrol maintains a table of available processing computers (hosts)
and tracks their status. Hosts managed by pcontrol are allowed to be
in one of several states: \code{off}, \code{down}, \code{idle},
\code{busy}, and \code{done} (see Figure~\ref{HostStates}). There are
also two virtual states: \code{new} and \code{delete}.  These states
have the following meanings:

If the host is \code{off}, it is known to pcontrol, but pcontrol does
not have an active connection to the machine. Hosts which are
\code{off} are not available for jobs, and pcontrol does not attempt to
initiate a connection to them.  A pcontrol user may force a host to
transition to the \code{off} state with the command host
\code{off~(hostname)}. (Note that this command will set only one of
the connections to the named host to \code{off}. If multiple
connections to a machine have been defined, multiple \code{off}
commands must be sent).

When pcontrol is told to consider a machine on, the machine is moved
from the \code{off} state to the \code{down} state. Pcontrol attempts
to initiate a connection to the host. Connections are made by running
a remote client on the host, using the specified connection
method. The connection method may be ssh, rsh, or an equivalent remote
shell connection. The choice is specified by the \code{COMMAND} Opihi
variable. The remote connection starts a dedicated remote client which
must accept the pcontrol client commands and respond
appropriately. The provided remote client is called {\em pclient},
though in principal other equivalent programs could be used by setting
the Opihi variable \code{SHELL}.  This feature more generally allows a
user to specify a path to the remote client, if it is not in the
user's path.

If the remote connection is successful, the connected host is moved by
pcontrol from the \code{down} state to the \code{idle} state. If the
connection is unsuccessful, pcontrol will try again after a certain
period of time. If the connection continues to be unsuccessful, the
retry period is doubled for each successiver connection attempt. If
the user wants to force pcontrol to retry the connection to a machine
(if, for example, the timeout is now very long, but the user knows the
machine's ethernet cable has been re-inserted...), this can be
achieved with the command host \code{retry (hostname)}. A host which
is \code{down} is in the limbo state between \code{off} and
\code{idle}.

Once pcontrol has made a successful connection to the host, the host
is in the \code{idle} state. At this point, it is ready to accept jobs
from pcontrol for execution. Pcontrol periodically queries the hosts
to check that they are still alive. If a host is discovered to be
unresponsive, and particularly if the remote pipe connection has
closed, then the machine is moved back to the \code{down} state.

Hosts which are \code{idle} may accept a job from pcontrol. A job
simply consists of a bare UNIX command, without redirection of
standard input or standard output. The host will initiate the job, and
pcontrol will place the host into the \code{busy} state. The remote
client, pclient, runs the job in the background and will continue to
accept input from pcontrol. pcontrol will continue to check the status
of the host, and now also the status of the specific job. As before,
if the connection breaks, pcontrol will migrate the host to the
\code{down} state. Any job already initiated on a host which goes down
will be returned to the pool of unexecuted jobs for later processing,
so the job will not be lost.

When the job exits, pclient tells pcontrol that the job is completed,
and specifies the exit status. At this point, pcontrol will move the
host from \code{busy} to \code{done} state. It will stay in this state
until pcontrol can determine the ending conditions and reset the
remote client. pcontrol requests the standard error and standard
output from the job from pclient. pcontrol stores this data with its
information about the completed job, and send a reset command to the
remote client. Once these cleanup tasks are successfully completed,
pcontrol will move the host to the \code{idle} state, ready for
further jobs.

Each physical computer may have multiple processors. pcontrol treats
each processor independently. It is up to the system configuration if
each computer needs to reserve one of its CPUs to manage background
tasks or if pcontrol should attempt to send one task per CPU and let
the operating system handle the I/O load. some of this behavior will
probably be eventually more intelligent. For example, the commands
which turn a host on or off should be able to do the same operation to
all host connections for the same machine name.

A machine may be completely removed from pcontrol's host tables with
the command host delete (hostname).

\subsubsection{Jobs}

The pcontrol accepts new jobs with the command \code{job ...}, in
which the ellipsis represents the command and arguments of a valid
UNIX command. The commands are run under \code{sh}, and are executed
in the user's home directory. Users should be wary of the conditions
under which the remote jobs are run. If the nodes in question all
cross-mount the same home directories, multiple jobs which interact
with the same named file may produce unexpected results. The
controller cannot enforce good behavior on the part of the remote
jobs; it is the responsibility of the user to ensure that conflicts do
not arise by, eg, always using unique output file names.

Other issues may arise from the fact that pcontrol may be choosing any
of the hosts to run the job. Typical failures arise if the user does
not realize that specific jobs do not behave the same on all machines,
or if a necessary resource (eg, some input data file) is only
available or accessible from some of the hosts. It is also the
responsibility of the task to wait for network lags (ie, NFS delays).

pcontrol gives each task a unique internal identifier (Job ID)
equivalent to the process ID used in UNIX. When a job is submitted to
pcontrol, the command echoes back the Job ID. This ID may be used by
other pcontrol commands to obtain information about or interact with
the job.  For example, PanTasks uses the Job ID to examine specific
jobs.

A job may specify a specific host for the task execution. The host
specified for a job may be required, or desired. In the first case,
pcontrol, will only run the job on the specified host, waiting until
it is available before attempting the job. In the second case,
pcontrol will attempt to send the job to the specified host, but if
the host is unavailable (how long? what conditions?), pcontrol will
allow the job to be sent to an alternative host. pcontrol attempts to
honor the requests for required and desired hosts, giving priority
first to required-host jobs, then to the desired-host jobs, and
finally to all other jobs. To specify a host for a job, the following
commands are used:

\begin{verbatim}
 job -host (command and arguments...)
 job +host (command and arguments...)
\end{verbatim}


The first case specifies a desired host, while the second specifies a
required host. It is also possible to specify the special host name
anyhost, which is equivalent to not specifying a host at all.

Job priority / urgency levels are not implemented at this time.

I/O vs CPU tasks are not currently distinguished by pcontrol

pcontrol stores the stdout and stderr for each completed job. To
retrieve these data from these streams, the user issues the commands
stdout (JobID) and stderr (JobID). The result is a single line
specifying the number of bytes to expect, followed by a dump of the
buffers, followed by the prompt. It is the user's responsibility to
relieve pcontrol of this data load by deleting jobs once they are no
longer needed. Job deletion is performed with the command delete
(JobID).

\begin{figure}
\begin{center}
\includegraphics[scale=0.85,angle=-90]{pics/pantasks.05.ps}
\caption{\label{JobStates} pcontrol job states.  Transitions labeled Ux
  are issued by the pcontrol user (including PanTasks).  Transitions
  labeled Px are initiated by pcontrol.  Transitions labeled Tx are
  the result of the job completion}
\end{center}
\end{figure}

Jobs are moved between the following states by pcontrol (see
Figure~\ref{JobStates}):
\begin{itemize}
\item pending: the job has not yet been executed.
\item busy: the job is currently being executed.
\item done: the job has completed, but the stdout/stderr has not been processed by pcontrol.
\item exit: the job has completed with a valid exit status
\item crash: the job has completed with a crash status (exit on signal). 
\end{itemize}

\subsubsection{Miscellaneous Commands}

It is possible to check the status of a single host or job with the
user command \code{check}.

pcontrol continuously examines the stack of jobs, adjusting their
state as needed and extracting their output when it is ready. These
checks are performed in the background, with pcontrol ready to accept
further commands from the user in the foreground. These checks are
performed after every keystroke, and also after an inactivity
timeout. The interrupt interval defaults to 1 second, but may be
adjusted with the pulse command, which takes as an argument, the
number of microseconds for the timeout.

The pcontrol system status may be examined with the command
status. This provides a dump of the job stacks and the host stacks.

It is possible to list the jobs currently in a specific stack,
corresponding to the list of jobs with a given state. This is done
with the command jobstack (stackname). The valid stack names are
pending, busy, exit, crash, and done. The result is a list of all jobs
on the specified stack. This is useful to determine quickly which jobs
have exited or crashed.

A specific job may be killed with the command \code{kill
(JobID)}. This command is only valid for a job in the busy state. Any
job in the pending, exit, or crash state may be deleted with the
\code{delete (JobID)} command. This is necessary to free the memory
associated with the job and its output streams.  PanTasks
automatically performs these job harvesting functions.

The command \code{verbose (mode)} turns the verbosity of the pcontrol
operations on or off.

The pcontrol and Nebulous have related needs for information from the
combined storage-and-processing nodes regarding which nodes are
available. Currently, the two systems independently examine the
hardware to judge the availability.  It is not yet clear if this
information is best stored in a single location (either pcontrol or
Nebulous), which provides the information to other systems on demand.
The current implementation allows the IPP system as a whole to
distinguih nodes which are available for processing from those that
are available to serve data as part of Nebulous.

\begin{figure}
\begin{center}
\includegraphics[scale=0.85,angle=-90]{pics/pantasks.06.ps}
\caption{\label{PControlLoop} PControl Job Monitor Loop}
\end{center}
\end{figure}

Figure~\ref{PControlLoop} illustrated the pcontrol job monitor loop.
This is very similar to the loop in PanTasks.  A background process
launched by readline during idle periods cycles through the list of
hosts (children) and migrates jobs to and from them as needed.

\subsubsection{pclient}

pclient is the remote process monitor for pcontrol, the parallel
process controller.

The program pclient is used to support the remote jobs which are run
on the remote hosts by pcontrol. The concept of pclient is to act as a
buffer between the job running on the remote host and pcontrol. The
pcontrol design uses (by default) ssh connections initiated by
pcontrol to the remote hosts. These connections execute the remote
program of pclient. The use of a remote login process lets the UNIX
system take care of the user authentication issues. In this case, the
recommended practice is to set up ssh to allow the connection to the
remote host without additional authentication using the appropriate
authorized keys.  The security is maintained by the security of ssh
logins to the computers used by PanTask and pcontrol.  

It is convenient to keep a continuous connection to the remote
hosts. This avoids incurring the overhead of authentication for each
command which is executed, while keeping a high-quality user
authentication process in place.

pclient acts as a buffer between pcontrol and the remote background
process, allowing the continuous connection to remain viable without
samping pcontrol with output from the jobs.  

\begin{figure}
\begin{center}
\includegraphics[scale=0.85,angle=-90]{pics/pantasks.07.ps}
\caption{\label{Pclient} Pclient monitor loop.}
\end{center}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{System Design : Science Analysis Tasks and Stages}
\label{sec:AnalysisStages}

This section describes the design of the science analysis stages which
perform the fundamental image analysis steps of the IPP.  The IPP
science image processing stages perform analyses on the night-sky
science images to extract the science data from these images.  These
consist of: 
\begin{itemize}
\item Phase 1, the image processing preparation stage,
\item Phase 2, the image reduction stage
\item Phase 3, the exposure analysis stage
\item Phase 4, the image combination stage.  
\end{itemize}
These analysis tasks must process the images in a timely manner so
that the incoming data stream will not overload the IPP Image Server.
The decision to execute a specific pipeline for a specific dataset is
made by the Scheduler, which sends the information to the Controller.
The Controller executes the pipeline for the data on an appropriate
machine and monitors the success or failure of the processing stage.

The analysis stages are written using programs executed as UNIX
commands.  These commands may be executed by the IPP Controller, or
may be executed individually by hand.  This option makes testing of
the complete analysis system much easier because the individual
analysis stages may be tested independently of each other and the rest
of the IPP infrastructure.  The analysis stages discussed in this
section use two somewhat different types of programs.  One set of
programs performs the heavy lifting of the data analysis: they examine
the pixels in images and perform some statistical analysis of the
pixel values or manipulate the data products in one way or another.
Another set of programs are used to tie the analysis stages together.
This latter set of programs examine the state of the Metadata database
and select images for processing.  They are used by PanTasks, the IPP
Scheduler to make decisions about which of the analysis programs to
run on which data.  In this section, we discuss the details of the
analysis stages in terms of the science analysis.  Below, in
Section~\ref{AnalysisPrograms}, we discuss the major analysis programs
and top-level analysis routines used by those program.  An important
distinction to be noted here is that the same analysis programs with
somewhat different options and configuration information can be
employed by different analysis stages.  We exclude detailed discussion
of the other connecting tools used to define the analysis stages.
These tools and the detailed construction of the pipelines which make
up the analysis stages are touched upon in Section~\ref{sec:PanTasks},
which discusses the IPP Scheduler program, PanTasks.  They are
discussed in more detail in Section~\ref{ippTools}.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Phase 1: image processing preparation}

The Phase 1 analysis stage is performed on each science exposure (each
complete FPA image) to calculate basic astrometric data needed by the
later stages.  Phase 1 uses the static (pre-determined) telescope
distortion model and a table of nominal OTA positions and rotations,
combined with the guide star pixel and celestial coordinates, to
determine the correct telescope bore-sight, field rotation and
magnification.  The guide star coordinates are loaded from the
Metadata database.  These calculations are performed by comparing the
observed guide star detector coordinates with the known astrometric
positions of these same stars as reported by an external astrometric
reference.  The accuracy of the resulting astrometric solution is
expected to be $\sim 1$ arcsec across the field, sufficient in later
stages to match the vast majority of astrometric reference stars with
their detections with minimal effort.

In some circumstances, science images may have no guide stars.  This
may occur in the Pan-STARRS system if the detectors are not run in OTA
mode, for example for short snapshot images.  This may also be the
case if the IPP is being run on non-Pan-STARRS data.  In such a
circumstance, the Phase 1 stage uses the provided boresight
coordinates, exposure time, and camera zero-point to predict the pixel
coordinates of known bright stars expected to be found on the
detectors.  It then extracts a large box ($\sim$ 30 $\times$
30\arcsec) around these locations and performs extremely basic object
detection to determine the detector coordinates of those bright stars
which are not saturated but which are significantly above the
background level.  By targeting known locations in the image files,
only a small amount of data will have to be read.

If the image has invalid coordinates or no detectable bright stars,
Phase 1 fails and reports a descriptive error.

Given the above astrometric solution, the Phase 1 analysis stage
constructs a table of the overlaps between the science image to be
processed and the static sky images that must be constructed.  This
table will be used to guide the processing of the static sky in Phase
4.  The overlaps should be generously calculated so that small errors
in astrometry at Phase 1 will not cause any valid static sky / science
image pairs to be missed because of the astrometric error at this
phase.  It is acceptable for a small number of invalid overlaps to be
identified as these will be excluded in Phase 4.  Static Sky cells
which do not have sufficient science image overlap ($< 5\%$) need not
be processed because the few new measured pixels do not add
significantly to the Static Sky.

\subsubsection{Examples}

Examples of Phase 1 as called from the command line, with different
types of images:

\begin{verbatim}
Phase1 -file filename.fits [FPA is single fits file]
Phase1 -list filename.list [FPA is collection of files]
Phase1 -imdb ID            [FPA is single file in image server]
Phase1 -FPA  ID            [FPA identifier from metadata db]
\end{verbatim}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Phase 2 : image reduction}

\subsubsection{Overview}

Phase 2 processing within the Pan-STARRS image processing pipeline is
the detrend stage, where the images from the detector are processed to
remove instrumental signatures.  This analysis is performed on
individual chips, which can be identified as the data entity which has
a single, continuous astrometric solution.

Phase 2 consists of the following operations, some of which as noted
may be skipped by the recipe:
\begin{itemize}
\item Load science image
\item Identify appropriate detrend images
\item Load detrend images
\item Form OT kernel
\item Convolve detrend images with the OT kernel
\item Bias/dark/overscan subtraction
\item Mask bad pixels
\item Trim overscan
\item Non-linearity correction
\item Flat-field
\item Mask diffraction spikes and optical ghosts
\item Subtract sky
\item Find and photometer objects in the image
\item Identify CRs by morphology
\item Determine PSF model
\item Improve astrometry
\item Extract Bright object postage stamps
\end{itemize}
The steps are explained in detail below.

\subsubsection{Load Images}

The Phase 2 analysis must load the science image to be analyzed into
memory, as well as the corresponding metadata (either from the image
header and/or from the IPP Metadata Database).  It must use the
metadata for the image, along with information from the processing
recipe, to determine the appropriate detrend images to be used for
this analysis.  The Metadata Database stores the information necessary
to associate a specific science image with one of the registered
master detrend images for each type.  These images are also loaded by
the Phase 2 analysis (note that the design of Phase 2 may perform the
actual loading of pixels in blocks or groups to minimize the memory
impact).

\subsubsection{Form OT Kernel \& Convolve with Detrend Images}

Science images which have been obtained with Orthogonal-Transfer
Guiding have had their pixel response smoothed by the image correction
motion.  For these images, some of the detrend images need to be
convolved by the same OT kernel, so that they accurately represent the
effective pixel response.  The detrend images which must be convolved
include: the flat-field image, the high-spatial-frequency fringe
images, and dark images, if they are used. The appropriate kernel for
each cell of an OTA must be determined from the guide star history,
extracted from the IPP Metadata Database or from the image header.  If
the OT kernel is not available, but the image metadata notes that it
should be, the convolution is skipped, with a warning.

The convolution method depends on the size and structure of the OT
kernel.  If the kernel is small ($< 5x5$ pixels), direct convolution
may be employed.  If the kernel is large, but may be decomposed using
Gaussians, then it may be convolved using a decomposition method.

The module convolves each of the dark frame, flat-field, and the
fringe frame(s) by the OT convolution kernel.  Specific flags in the
static bad pixel mask are also grown by the outline of the OT
convolution kernel (see Section~\ref{sec:masks}).

\subsubsection{Bias Correction / Overscan Subtraction}

The image bias must be subtracted. Since different detectors behave in
different ways, several options for modeling the bias are available.
The bias is measured from the image overscan region.  The bias
subtraction method must be capable of subtracting a single constant
from the complete image, or to subtract a 1-D bias which varies as a
function along the overscan.  The function used to represent the
overscan region may be a spline or a Chebychev polynomial derived from
the data values along the overscan.  The values used to determine both
the single constant or the inputs to the spline and polynomial fits
are derived from groups of pixels on the basis of one of several
statistics, including the sample and robust mean, median, and modes.
In the case of a single constant, all of the overscan pixel values are
used in the calculation of this statistic.  In the case of the 1-D
functional representation, the input values to the fit must represent
the coordinate along the overscan, with the statistic derived from the
pixels in the perpendicular direction at each location.
Sigma-clipping on the input data values must be an option.

\subparagraph{Flag bad and saturated pixels}
\label{sec:masks}

A static bad pixel mask is used to identify pixels which are known to
be bad in the camera.  This mask is then processed with the science
image. Bad pixels which are charge traps are grown by the extent of
the OT convolution kernel.  Bad pixels above a charge trap (i.e.\ bad
columns) must not be grown, since they were not affected by pixel
shifting, but only became bad at read-out.

Pixels which are saturated in the A/D converter, or with a signal
level at which the response is excessively non-linear, must also be
masked, and this area must be grown by an additional pixel to mask
excess charge spillover.

The bad pixel mask is carried with the science images.  Different bits
are set to identify different reasons for masking the pixel.  Flags
are required for at least each of the following pixel attributes:

\begin{itemize}
\item The pixel is a charge trap;
\item The pixel is a bad column;
\item The pixel is saturated in the A/D converter;
\item The pixel is non-positive in the flat-field;
\item The pixel is part of a row that has excess noise; and
\item The pixel is determined to be a cosmic ray, based on its
morphology.
\end{itemize}

\subsubsection{Trim}

The image is trimmed to remove the non-imaging pixels, such as the
overscan and any pre-scan pixels, along with those pixels near the
edges that have been compromised due to OT operation.  The definition
of the imaging area of the detector is determined from the camera
configuration data or from the metadata associated with the image,
with the choice a user-configurable option.

The input science and mask frames are additionally trimmed by the
extent of the OT convolution kernel in each direction ($+x$, $-x$,
$+y$, $-y$).  Within the PSLib image handling functions, the trim
function is a virtual operation which simply marks the boundaries of
the trimmed image but does not remove the corresponding memory.  This
allows the later corrections to work with untrimmed correction images
and still apply the correct pixels.  At the end of Phase 2, the only
the trimmed portions of the output images are written out to disk.

\subsubsection{Non-Linearity Correction}

If required, the science image (after bias correction) must be
corrected for the effects of non-linearity through a provided
polynomial fit to the pixel data values or a numeric lookup table as a
function of pixel flux.  The choice to apply the correction must be
set by the analysis recipe.

\subsubsection{Flat field Correction}

The science image (after bias correction and non-linearity correction)
must be corrected for sensitivity variations as a function of
position, dividing by a flat-field image.  The mask is also modified
for zero-valued pixels in the flat-field image.

\subsubsection{Sky \& Fringe subtraction}

After the science image has been flat-fielded, the flux contribution
of the sky (from both continuum emission and the line emission that
causes fringing) must be subtracted from the image.  The subtraction
needs to remove background (technically, foreground) variations which
are not celestial but generated in the atmosphere or by more localized
scattering.  This background should include the contribution from the
zodiacal light.  This background subtraction does not address the
artifacts generated by bright stars: bleeding columns, ghosts, or
other localized reflection effects.  This process also produces a
super-binned image of the background map which may be used as a
debugging diagnostic.

This analysis step temporarily masks objects on the image using the
astrometric solution from the boresight and fits for the sky
background, consisting of a polynomial to model the continuum, and the
fringe frame(s) to model the fringes from sky emission lines.  If the
concentration of objects in the image is too high to reliably fit the
sky background, the background solution from an exposure close in time
and airmass to the current object image is used.  The output is the
sky-subtracted object image.

\subsubsection{Detect and Measure objects}

After the image have been processed by the preceding steps, the Phase
2 analysis performs a basic object detection analysis.  Objects on the
flat-fielded object image are found, and general parameters are
measured.  Object detection is performed at several stages by the IPP,
with different object parameters measured in each case.
Table~\ref{tab:APdetections} gives a list of the different detection
stages and the object parameters measured for those stages.  For the
Phase 2 analysis, the object parameters are: the object centroid and
the position covariance matrix, the instrumental PSF magnitude and
error, local background level and error, a measurement of the
star-galaxy separation, and a measurement of the object shape
($\sigma_x, \sigma_y, \theta$).  The detection threshold must be
configurable, and be a function of the average background flux or the
image noise map.  Minimal object classification must be performed to
distinguish objects which are consistent with a single PSF, objects
which are inconsistent, and objects which are saturated.  The
resulting collection of detected objects are saved in the AP Database
along with the relevant image metadata (\ie filter, exposure time,
etc).  In addition, this process constructs a model of the
point-spread-function (PSF) as a function of position in the image.
This PSF model is saved as part of the image metadata.

\subsubsection{Identify CRs by morphology}

Charged particles in the detector frequently cause features which do
not have the morphology of astronomical objects.  In a well-sampled
image, these may be easily identified by the sharpness of the image.
In a near critically-sampled image, these `cosmic rays' may be
indistinguishable from stellar objects.  This analysis step makes
morphological identification of cosmic rays if the imaging data is
sufficiently well sampled.  The identified cosmic rays are masked with
a configurable growth factor so that additional pixels beyond the
detected pixels in the feature are also masked.

\subsubsection{Perform Astrometry}

The detected objects are matched with known astrometric reference
objects, using reference object coordinates which have been adjusted
for proper motion.  The matches are then used to improve the
astrometric parameters for the individual OTAs.  The OTA astrometric
parameters which are determined may include terms up to 3rd order in
position, though the terms which are actually fitted are
user-configurable.  The Cell astrometric parameters are not allowed to
vary at this stage.  The fit must be robust, rejecting outlier matches
(either stars with poorly determined proper motion or spurious
matches).  The resulting astrometric solution is consistent across the
OTA field to within 1.0 arcsec.

\subsubsection{Perform Photometry}

If possible (if a local photometry reference exists), the Phase 2
analysis determines a photometry zero point for each image.  To do
this, it extracts the appropriate reference objects (from the AP
Database) and matches the stars between the two samples.  The
zero-point is derived on the basis of a static atmospheric absorption
model (eg, linear airmass slope).

\subsubsection{Bright object postage stamps}

The IPP must have the capability of extracting regions surrounding a
specified subset of objects from the flattened images.  These postage
stamp images must be saved for additional use by client science
pipelines.  The identification of these objects must be on the basis
of a set of rules applied to the object's magnitude and position.  The
postage stamps are not restricted in shape to simple rectangles, but
may represent more complex regions.  They are written to the Image
Server.  The outputs are these postage stamps and pixel masks, which
are sent to the IPP Pixel Server.

%\begin{figure}
%\begin{center}
%\resizebox{6in}{!}{\includegraphics{pics/phase2}}
%\caption{ \label{fig:phase2} Phase 2 dataflow - this diagram is old: update}
%\end{center}
%\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Phase 3 : exposure analysis}

The Phase 3 system operates on the combined Phase 2 results from an
FPA to determine improved solutions for the image calibrations and to
provide the parameters needed by Phase 4.  The Phase 3 output is saved
by the Metadata Database, and consists largely of improved values of
the calibrations already determined by Phase 2.  The analysis
performed by this pipeline consists of:

\begin{itemize}
\item improved astrometric solution based on comparison between
  objects in the images and the astrometric reference.
\item improved background model based on the full telescope field, or
  fields.
\item photometric solution based on comparison to photometric
  standards
\item FPA-wide PSF analysis
\end{itemize}

In the Phase 2 analysis, the astrometric solutions were determined
independently for each chip.  These solutions are limited by the
assumption of a static distortion and by the accuracy of the
astrometric reference.  In the phase 3 analysis, the astrometric
solutions of the complete FPA images are improved by combining the
astrometry for all chips.  The astrometry model consists of a
projection of the celestial coordinates to the telescope boresite
center, followed by a rotation to the average rotation of the FPA and
adjustment for the central plate scale.  The free parameters in this
stage are the boresite coordinates ($R_o, D_o$), the field rotation
($\theta_o$) and the plate scale ($\rho_o$), and are fitted in Phase
1.  These tangent plane coordinates are then distorted by the optical
distortion model, consisting of $N^{\rm th}$ order polynomials in two
dimensions.  Finally, the focal plane coordinates are mapped to the
individual chip coordinates, including the chip position and rotation,
as well as possible higher order terms representing warping of the
individual detectors.  A first pass at the chip positions is
calculated in Phase 2, while the complete set of parameters is fitted
as a whole during Phase 3.  The fitting process is determined in a
robust and stable way by fitting the gradient of the distortion as a
function of field position, removing the degeneracy of the distortion
coefficients on the chip position parameters.

In the Phase 2 analysis, the background is determined based only on
the available sky in a single chip.  However, the background
structures are normally correlated on the scale of the FPA, so an
improved background solution can be determined by combining the
information from many chips.  A high-order polynomial model of the
background may be used and subtracted from all chips.  

The Phase 3 photometric improvement is made by comparing the
photometered objects from Phase 2 with the corresponding objects in a
local reference catalog.  This analysis may only be performed if a
local reference is available.  Note that improved relative photometry
calculations may be performed in the absence of a reference catalog on
the basis of image overlaps in the AP Database {\em after} the
detections have been added to the Database.  Such a relative
photometry analysis is outside the scope of Phase 3 and will likely be
performed as an independent analysis process.  Given the presence of a
local photometry reference, the zero point variations across the field
may be measured, and possibly modeled.  If the zero-point variations
are excessive, then the image is marked as non-photometric by the
analysis.

In the Phase 4 analysis, the $N$ FPA images are optimally combined to
create a single image of the sky with bad-pixel and cosmic-ray
rejection.  This combination requires the calculation of a set of PSF
kernels to convert each of the input images to a single, common PSF.
These PSF kernels are determined from the per-chip PSFs measured in
Phase 2.

\subsection{Summary of the Phase 1-3 Data Products}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Phase 4 : image combination}

\subsubsection{Overview}

Phase 4 is the image combination stage, in which multiple images of
the same portion of the sky are merged and confronted with the static
sky image.  Phase 4 operates on the smallest data unit of the static
sky, the sky cell, along with the associated pixels from a collection
of images which have been processed through phases 1--3.  The size and
exact representation of a static sky cell are configurable parameters
which may vary between surveys.  In some cases, it may be feasible to
use a single large rectangular tangent plane image for the static sky
and accept the distortion near the edges.  In other cases, the static
sky cells may be more appropriate defined to represent a relatively
small patch over which the distortion is known to be small.  The
details of the static sky cell definition should not be a driver for
the Phase 4 analysis.  In order to meet the image degradation
requirements, the pixel scale of the static sky is planned to be
0.2\arcsec, somewhat smaller than the 0.25\arcsec\ raw image pixel
scale;  the choice of this scale must be determined with some caution.
If the pixels are too large, the degredation will be excessive; too
small, and the total volume of the static sky image data will be
excessively costly.

\begin{figure}
\begin{center}
\resizebox{6in}{!}{\includegraphics{pics/phase4}}
\caption{ \label{fig:phase2} Phase 2 dataflow - this diagram is old: update}
\end{center}
\end{figure}

In the basic concept of the Phase 4 analysis, each sky cell is
examined independently.  The corresponding pixels are extracted from
the exposures being processed and mapped to the projection of the sky
cell. The pixels from the multiple input processed images are combined
into a single, cleaned image.  Outlier pixels may be optionally
rejected at this stage (optionally, since moving objects will be
rejected in images obtained over a wide range of times).  This image
is then confronted with the static sky cell data to produce a
difference image.  Residual objects in the difference image above a
threshold are detected and excised from the original cleaned image.
The remaining pixels are added to the existing static sky image.
Object detection must be performed on the difference ($P4\Delta$) and
cleaned ($P4\Sigma$) images.  This process is illustrated graphically
in Figure~\ref{BasicP4}

\subsubsection{Image Warping and Matching}

The analysis first maps the detector images to the sky cell using the
specified linear transformations, then combines the images with strong
rejection criteria and uses the combined sky cell image to identify
artifacts in the original detector images.  It is desirable that the
artifacts are masked in the detector plane (i.e.\ before mapping to
the sky cell) so that they are not smeared out by the mapping;
alternatively, the CR mask needs to be grown by an additional pixel.
The mapped and masked detector images are then optimally combined
using the specified weighting.  Both sets of combinations use the
photometric calibration for the images to set the relative scales of
the input images.  The limiting magnitude for the combined sky cell
image should also be estimated.
%

\subsubsection{Static Sky Subtraction}

The corresponding static sky image is subtracted from the combined
image stack.  In this step, it is necessary to match the image kernel
between the input image and the static sky image.  This will be done
by solving for a best-fit image kernel which minimizes the difference
image using a technique equivalent to the Allard-Lupton method.  One
modification for the IPP is to represent the kernel as a combination
of independent pixels rather than represent the components of the
image difference kernel as a combination of Gaussians.  This method
also automatically determines a photometric match between the static
sky image and the input science image.

\subsubsection{Object Detection and Measurement}

Objects in the difference image are detected and a specific set of
object parameters are measured from these detections.
Table~\ref{tab:APdetections} gives a list of the different detection
stages and the object parameters measured for those stages.  For the
Phase 4 difference image (P4$\Delta$), the measured object parameters
consist of: the object centroid and the position covariance matrix,
the instrumental PSF magnitude and error, local background level and
error, a measurement of the star-galaxy separation, and a measurement
of the object shape ($\sigma_x, \sigma_y, \theta$).  The detection
threshold must be configurable, and be a function of the average
background flux or the image noise map.  Detections must be performed
for both positive and negative (static sky only) sources.  Minimal
object classification must be performed to distinguish objects which
are consistent with a single PSF, objects which are inconsistent, and
objects which are saturated.  The resulting collection of detected
objects are saved along with the relevant image metadata (\ie filter,
exposure time, etc).

Objects in the cleaned, summed image are detected and a specific set
of object parameters are measured from these detections.
Table~\ref{tab:APdetections} gives a list of the different detection
stages and the object parameters measured for those stages.  For the
Phase 4 summed image (P4$\Sigma$), the measured object parameters
consist of: the object centroid and the position covariance matrix,
the instrumental PSF magnitude and error, local background level and
error, a measurement of the star-galaxy separation, a measurement of
the object shape ($\sigma_x, \sigma_y, \theta$), the Petrosian radius,
magnitude, axis ratio, and angle; and the S\'ersic radius, magnitude, axis
ratio, angle, and parameter $\nu$.  The detection threshold must be
configurable, and be a function of the average background flux or the
image noise map.  Minimal object classification must be performed to
distinguish objects which are consistent with a single PSF, objects
which are inconsistent, and objects which are saturated.  The
resulting collection of detected objects are saved along with the
relevant image metadata (\ie filter, exposure time, etc).  In this
measurement, objects at known positions will also be measured even if
they have not been detected.

Objects which are detected in both of the Phase 4$\Sigma$ and Phase
4$\Delta$ images are saved to the AP Database, along with the relevant
image metadata (\ie filter, exposure time, etc).  In the process of
adding these objects to the database, the transients which are
correlated with previous detections of an object (and those which are
not) will automatically be determined.  A subset of these transient
objects are sent, along with their associated metadata, to the MOPS
and other preferred science client pipelines.  

\subsubsection{Static Sky Update}

It is essential that the static sky image (which may have been
painstakingly accumulated over many months) not be corrupted by adding
in transient sources, or data that is of suspect quality (due, e.g.,
to an error upstream in the processing).

\tbd{add discussion of the choices to be made in generating the
  static sky image stacks: interpolation methods, selection of input
  images by IQ, smoothing of input images by their PSF, weighting and
  clipping of input pixels}

Object analysis of the static sky images is {\em not} a part of the
Phase 4 analysis.  This processing is envisioned to take place
relatively infrequently (perhaps weekly or even monthly) and is
scheduled as a separate analysis task, probably run during the day at
a time when the computing infrastructure is not under significant load.

\tbd{add discussion of the multiple image analysis and object
  analysis without the static sky (ie, on all input images at once)}

\subsubsection{Magic and Phase 4 Modifications}

\begin{figure}
\begin{center}
\resizebox{6in}{!}{\includegraphics{pics/phase4a}}
\caption{ \label{fig:phase4a} Phase 4a}
\end{center}
\end{figure}

The Pan-STARRS relationship with the U.S. Air Force has some
implications for the data processing which place some interesting
constraints on the IPP implementation.  The U.S. Air Force is funding
the construction of PS-1, and we are thus subject to restrictions
under which the Air Force must operate telescopes.  The Air Force has
diplomatic and security concerns about publically releasing images in
which artificial satelites are detected, particularly if those
satelites are identified.  Historically, Air Force projects have been
restricted from releasing images with identified satelites,
considering the data to be of a sensitive nature.  

The language which governs such Air Force projects is in the process
of being modified so that images with identifable satelites can be
treated under the classification of 'For Official Use Only'.  Even
under such an arrangement, however, the Air Force requires that the
satelites which appear in the PS-1 images be excised in such a way
that their orbits cannot be reliably identified.  

Satelite detected in the Pan-STARRS images will appear as long ($>
100$ pixels) streaks.  The natural, and most effective, way to
identify such streaks is to search for them in the images after they
have been processed using the difference image processing in Phase 4.
A team from Boeing has been contracted to develop a software module to
idenfity streaks in these images.  They delivered the initial release
of their software, which is currently called {\em Magic}, in the
beginning of January 2006.  Further effort on that software will be
required to confront it with the real PS-1 image parameters.  

In order to mesh the operation of {\em Magic} with the IPP Phase 4
analysis for PS-1, some modifications must be made to the operation
sequence.  In the case of PS-1, unlike PS-4, the basic set of images
which confronted with the Static Sky image are obtained in sequence,
not essentially simultaneously.  This is important for Magic because
the satelite streak in one image will not appear in the other three.
Thus the Magic operation must be performed on intermediate difference
images for individual exposures.  To facilitate this, the basic Phase
4 described above is divided into three stages: Phase 4a, in which the
individual difference frames are constructed; Magic; and Phase 4b, in
which the results of Magic are used to filter the images before the
final stack and difference images are generated. 

Figure~\ref{phase4a} illustrates the operation of Phase 4a and Magic.
The individual warped image (P4$_W$) is constructed and differenced
against the corresponding Static Sky image.  In this analysis, poisub,
which performs the kernel matching and difference image construction,
is run with somewhat looser constraints on the image difference
kernel.  The lower-accuracy difference kernel will make the difference
image noisier in the vicinity of galaxies and bright stars.  Unlike
the science transients, however, most of the pixels involved in a
satelite streak are not in close proximity to another objects; for SNe
and GRBs, the objects are preferentially located near host galaxies
where it is most critical to have a high-quality subtraction.
Allowing poisub a more relaxed image difference kernel results in a
substantial improvement in the convolution speed.  The resulting
difference image is called P4$\Delta$'.  

{\em Magic} searches for streakes in the difference images by using a
heirarchy of Huff transforms.  A single run of {\em Magic} expects to
have access to all of the P4$\Delta$' images from a single FPA image.
It starts with independent Huff transforms of the individual cells,
which can be performed in parallel under the PanTasks paralleliztion
scheme.  The results of these Huff transforms are then merged to yield
the equivalent of a larger Huff-transformed image.  The heirarchy
searches for significant streaks from the individual cells, the
equivalent 2x2 cell sets, then 2x2 of those, etc, until the entire
array has been searched.  Significant peaks detected from the Huff
transforms are then used to restrict the pixels in the image space.
These pixels are examined and streaks detected by requiring a certain
fraction of the pixels along the purported streak to be lit.
Filtering using the cross-streak PSF can also be applied to enhance
the detection and minimize the false positives. Any streaks which are
detected are then excisized.  In order to obfuscate the idenification
of the satelite, this step masks the streak pixels, with a box
somewhat wider than the streak (10-20 pixel wide), displaced by an
unknown, random amount relative to the streak center-line.  This strip
is extended to the ends of the FPA array.  Analysis of the expected
density of streaks by the Boeing team, using known satelites as the
input to the simulation, shows that we should expect to lose less than
1\% of the pixels due to these long excised regions. 

The result of the Magic process is a set of masks for each raw PS-1
image.  Any images which are released beyond the IPP cluster must have
these masks applied, or with manual inspection for visible streaks.
\tbd{time-stamps? ok to release without the valid time? or only with }

The final portion of the analysis, Phase 4b, is illustrated in
Figure~\ref{phase4b}.  In this stage, the intermediate result images
from Phase 4a can be used.  The P4$_W$ images are masked with the
corresponding masks from Magic.  These images are then stacked as
normal, and the resulting summed image is differenced with poisub,
using a more stringent limit on the image difference kernel.

\begin{figure}
\begin{center}
\resizebox{6in}{!}{\includegraphics{pics/phase4b}}
\caption{ \label{fig:phase4b} Phase 4b}
\end{center}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{System Design : Calibration Image Processing}

The Calibration Analysis Stages construct calibrations from the
relevant input data.  Some of these combine multiple raw input images
together, after processing, to create a high-quality high-signal
master calibration image.  Some of the calibrations are used to
correct other calibrations.  Each of the calibration stages must also
provide the tools to test the quality of the input data against
existing master calibration data and to test the consistency of
multiple measurements of the calibration.
 
The Calibration analysis stages may be performed on whatever
timescales are appropriate and necessary to maintain the quality and
relevance of the calibration images.  The specific calibration data
which must be constructed in the calibration analysis stages is listed
below.

The IPP must generate basic calibration images using the raw bias,
dark, and flat-field (dome or twilight) images obtained by the
telescope as the input.  The analysis of these images requires
relatively simple stacking of the input set of images.  Outlier
rejection, both of complete input images as well as pixels within the
input stack, must be performed.  In addition, each type of image
requires an appropriate normalization which may depend on the data
levels in other detectors in the input set.  Each of these calibration
stages must be able to determine from the input stack if the relevant
calibration image needs to be updated and perform an initial test to
see which input images are consistent and valid.

\subsection{Bias Images}

Bias images may be needed to correct for structure in the bias.  The
IPP must have the capability of constructing a master bias image from
a stack of raw bias frames.  The input bias images, representing
offsets from the overscan level, are processed by subtracting the
overscan, including 1D structure if needed.  

The master bias frame construction uses outlier image and outlier
pixel rejection to construct a single high-quality bias frame.  The
statistic used to determine pixel values from the input stack can be
set by the user to be one of the following: the sample mean, median,
and mode, robust mean, median, and mode, and the clipped mean and
median.  Testing of the input images consists of constructing residual
images, in which the master bias is applied to the input images.
These images may be included or excluded from an additional iteration
of the stack on the basis of their pixel-to-pixel statistics.

\subsection{Dark Images}

Dark images may be needed to correct for structure in the dark
current.  The IPP must have the capability of constructing a master
dark image from a stack of raw dark frames.  The input dark images are
first corrected for the bias using whatever method is appropriate for
the science images.  Master dark frames depend on their exposure time.
As such, the input dark frames must have a limited range of exposure
times, and the output dark frame includes the exposure time as part of
its associated metadata.  

The master dark frame construction uses outlier image and outlier
pixel rejection to construct a single high-quality dark frame.  The
statistic used to determine pixel values from the input stack can be
set by the user to be one of the following: the sample mean, median,
and mode, robust mean, median, and mode, and the clipped mean and
median.  Testing of the input images consists of constructing residual
images, in which the master dark image is applied to the input images.
These images may be included or excluded from an additional iteration
of the stack on the basis of their pixel-to-pixel statistics.  A
collection of master dark frames with a range of exposure times are
used to determine the scaling of the dark frame as a function of
exposure time.

\subsection{On-Off Dark Images for Light Leaks}

A type of image which may be necessary for calibrations will be pairs
of images taken at night with the shutter closed with and without the
dome shutter closed.  Such a pair of images can be used to determine
any light-leak in the camera which may contribute additional flux
across the mosaic.

\subsection{Flat-Field Images}

Master flat-field images must be constructed from a collection of
input flat-field images.  The input flat-field images may be obtained
from any of the standard sources: the dome, the twilight sky, and the
night-time sky.  The choice of flat-field input image must be
determined experimentally from observations during the commissioning
phase of the telescope.  The IPP flat-field construction system must
be capable of handling any of these sources.  

An appropriate set of input images is selected on the basis of their
flux levels, time of observations, and the observing conditions.  The
input flat-field images are processed (bias and dark corrected if
needed) and the resulting images are stacked.  The master flat-field
construction uses image and pixel outlier rejection to construct a
single high-quality master flat-field frame.  The statistic used to
determine pixel values from the input stack can be set by the user to
be one of the following: the sample mean, median, and mode, robust
mean, median, and mode, and the clipped mean and median.  Testing of
the input images consists of constructing residual images, in which
the master flat-field image is applied to the input images.  These
images may be included or excluded from an additional iteration of the
stack on the basis of their pixel-to-pixel statistics.

\subsection{Mask Images}

Preliminary bad-pixel mask images are generated on the basis of
comparison between raw flat-field images and a cleaned, stacked
master.  The mask creation system accepts a collection of flat-field
images and identifies pixels which are consistently poorly flattened.
Pixels which are under-responsive are also identified as pixels to be
masked.  

\subsection{Sky \& Fringe Frames}

Fringe-correction frames must be generated to remove the fringe
pattern caused by thin-film interference in the top layers of CCDs,
particularly in the redder passbands.  Fringe correction frames may be
constructed on the basis of observations of the night-sky in the
appropriate filters or on the basis of dome fringe lamp observations.
The choice of the appropriate source will be determined experimentally
on the basis of data obtained during the commissioning phase.  The IPP
must be capable of handing either source.  The images are first
flattened to remove the pixel-to-pixel sensitivity variations of the
detector.  The combination of multiple input fringe frames may not be
simply stacked since the amplitude of the fringe pattern varies
independently of other variations in the image.  The amplitude of the
fringe pattern in the input frames is measured and the images scaled
to normalize the fringe amplitude to a consistent range (-1 to +1) for
all input images before they are combined with one of the standard
combination statistics (mean, median, mode, etc).  The quality of the
input frames is tested by flattening the input image and applying the
master fringe-frame.  The resulting residual image statistics are used
to select or exclude specific input images.

\subsection{Shutter Correction Map}

Shutter correction map images may be generated based on the timing
measurements of the shutter itself, or on the basis of dome-flat
images of decreasing exposure times down to the shortest available
exposures.

\subsection{Low-k Sky Models}

Large-scale background structure in images which is not caused by
thin-film interference must also be detected and corrected.  Models of
this background structure may be a necessary input to the correction
procedure.  The IPP must have the capability of generating image
models of the large-scale structure patterns observed with the
telescope

\subsection{Flat-Field Correction Frame}

Flat-field images, whether constructed from the dome, twilight, or
night-sky images, do not perfectly correct the detector response in a
consistent fashion across the full field of the camera.  The IPP must
have the capability of generating flat-field photometric correction
frames on the basis of the measured photometry of objects which are
moved to a variety of locations on the detector in a sequence of
images.  The flat-field correction frames analysis stage makes use of
targeted observations following a specified dither pattern, and
extracts the photometered objects from the AP Database to determine
the necessary photometric corrections.  The resulting image is applied
to the master flat-field image.  Testing of the correction is
performed by applying the correction to the basic master flat-field
image, applying that flat-field image to the dithered photometry
observations, and performing the object detections.  Comparison of the
photometry of individual stars at different locations on the mosaic
will demonstrate the consistency of the flat-field image.

\subsection{Non-Linearity Correction}

The IPP must have the capability of constructing a correction for
non-linearity in the detectors.  These frames are constructed from
exposures of a uniform source with a range of exposure times.  The
non-linearity correction frames provide polynomial correction
coefficients or a lookup table describing the correction.  There is
likely to be a single non-linear correction for each OTA detector, or
potentially for each Cell.  The IPP must handle these two cases.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{System Design : Miscellaneous Analysis Stages}
\label{sec:MiscStages}

This section discusses additional operations which are performed by
the IPP but which do not fall under the analysis of the science images
or the creation of the calibration images.

\subsection{Retrieval}

The retrieval stage simply retrieves images from an external source
(ordinarily OTIS at the Summit, but it could conceivably be some other
external source) and store it in the Image Server.  

\subsection{Static Sky Analysis}

The IPP is responsible for performing object detection and analysis on
the static sky.  This analysis is performed continuously (every day or
week) on those portions of the sky within 15\degree\ of the sun.  In
this analysis, the object measurement is much more detailed than those
performed in the real-time analysis.  The currently envisioned
parameters to be measured for every object are listed in
Table~\ref{tab:APdetections}.  The parameters include the object centroid
and the position covariance matrix, the instrumental PSF magnitude and
error, local background level and error, a measurement of the
star-galaxy separation, a measurement of the object shape ($\sigma_x,
\sigma_y, \theta$), the Petrosian radius, magnitude, axis ratio, and
angle; the S\'ersic radius, magnitude, axis ratio, angle, and
parameter $\nu$, and a collection of annular aperture flux
measurements, all of which are also measured for the P4$\Sigma$
images.  In addition, the galaxy-shape parameters $\Gamma_1 \&
\Gamma_2$, along with the complete `polarization' terms are measured,
as well as a collection of annular aperture flux and variance
measurements.  Another unique feature of the static sky analysis is
that the object detection may be performed simultaneously on all
filters, in which case the locations and other parameters may be more
strongly constrained by simultaneously fitting between all bands.  The
analysis to be performed may be substantially more complex than the
real-time analysis because of the relatively low data rate.  Instead
of needing to process thousands of images per night ($\sim 350$Mpix
per second), it is only necessary to process the complete sky in a
year, or an average rate of $\sim$2 Mpix per second, or $< 1$\% of the
object analysis in the other analysis stages.  These operations are
all functions which will be performed within the PSPhot program using
recipe options.

\subsection{Astrometric and Photometric Reference Catalog}

The IPP is responsible for generating the Astrometric and Photometric
(AP) Reference Catalog.  The IPP provides several tools for performing
this analysis.  The DVO programs \code{relphot}, \code{uniphot}, and
\code{relastro} perform most of the operations required to generate
the AP Reference Catalog.  These include the determination of the
image zero-points, the identification of objects with significant
variability, the detection of individual outlier measurements, the
detection of objects with substantial astrometric error, the analysis
of parallax and proper-motions, etc.  In addition, the DVO shell
program will be used to generate the color transformations from the
observed data and to perform other tests of the catalog quality.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Software Hierarchy}
\label{sec:SoftwareHierarchy}

In order to facilitate testing and development, and to encourage
flexibility, the IPP is built in a layered fashion.  The lowest level
functions will be written in C and collected together into a
Pan-STARRS library.  These library functions will be used to write
more complex modules.  The modules will be written in C but will make
use of SWIG to make their functionality available within other
languages.  In particular, the modules can be tied together with a
program in C or through the use of a high-level language such as Perl,
Python, or Tcl employing the SWIG interfaces.  For the high-level
functions in the operational system, the IPP will make use of Perl as
the scripting language to provide the required flow-control to tie the
modules together.

This approach satisfies the requirement that complicated low-level
analysis steps run fast, while preserving flexibility for coding the
high-level wrappers for which the speed requirements are not so
stringent.

\subsection{External Libraries}

Pan-STARRS will employ several external libraries to save duplicating
functionality that is already available.  These external libraries
will be wrapped by the Pan-STARRS Library, insulating the project from the
implementation details of the external libraries.  Examples of the
external libraries are FFTW and SLALib.

\subsection{Pan-STARRS Library}

The Pan-STARRS Library will consist of C structures describing the
basic data types needed by the IPP and C functions which perform the
basic data manipulation operations.  Note that a subset of the library
functions will be provided with SWIG interfaces as well to allow for
their use in the creation of the processing stages.  Examples of the
Pan-STARRS Library are Fourier transforms and transforming between
pixel and celestial coordinates.  The details of the Pan-STARRS
Library are specified in the document Pan-STARRS IPP PSLib
Supplementary Design Requirements Specification (PSDC-430-007), which
also addresses coding requirements detailed in the IPP PS-1 SRS
(PSDC-430-005), Section 3.3.

\subsection{IPP Modules}

The IPP analysis stages are broken down into modules which represent
specific functional operations.  The modules will be written in C
using the Pan-STARRS Library functions and will be grouped into a
Pan-STARRS Module Library.  The modules will be provided with SWIG
interfaces to all public APIs for their use in processing stages.
Examples of modules are overscan subtraction and image combination.
Some modules (e.g.\ find objects on an image) will be used by multiple
stages.  The details of the Pan-STARRS Modules are specified in the
document Pan-STARRS IPP Modules Supplementary Design Requirements
Specification (PSDC-430-012), which also addresses coding requirements
detailed in the IPP PS-1 SRS (PSDC-430-005), Section 3.3.

\subsection{IPP Analysis Programs}

The major IPP processing tasks are organized into stages, which
consist of multiple modules.  Each stage represents a collection of
complex operations performed on a single data entity.  Each stage
therefore represents the maximum amount of effort which can be
performed in serial without interaction between parallel threads.  The
stages will be written in Perl, linking the modules together.
Examples of stages are Phase 2 (detrend images) and Phase 4 (combine
images from multiple telescopes and search for transients).

\section{Top-Level IPP Analysis Programs}
\label{sec:AnalysisPrograms}

The IPP uses a handful of high-level analysis routines which perform
the bulk of the actual analysis effort.  These routines are used both
as stand-alone programs and in some cases as library functions called
by other stand-alone programs.  The 6 primary top-level analysis
routines are:
\begin{itemize}
\item {\bf ppImage} : the complete single-image analysis program.
\item {\bf ppMerge} : the basic image combinations program.
\item {\bf psphot} : the photometry analysis routine.
\item {\bf psastro} :  the astrometry analysis routine.
\item {\bf stac} : the science image combination program.
\item {\bf poisub} : the image difference program.
\end{itemize}

These programs are not mutually exclusive: psphot and psastro are both
used by ppImage, while psastro is used as a stand-alone tool within
the IPP, and psphot can be used as such.  

\subsection{Software Configuration and  Camera Definition Information}

Every camera which may be analysed by the IPP has differences in how
the data is represented.  The IPP is built with the flexibility to
handle data from many different cameras, not just the Pan-STARRS
Gigapix cameras.  This is partly to allow testing of the analysis
system on data from other telescopes, such as MegaPrime on CFHT and
Suprime on Subaru, but also to allow us to adapt to changes in the
design of the Gigapix cameras themselves.  It also means the IPP
software may be used by astronomers for other analysis projects beyond
the IPP.  

Most cameras provide extensive descriptive information in the FITS
image headers when the images are read out.  Typically, the location
and orientations of the individual detectors are defined by keywords
such as DATASEC and DETSEC.  Other variations on these words are used
for cameras which place the pixels from multiple amplifiers in the
same FITS data segment.  Other parameters, such as astrometric
information or exposure times, are stored in headers as well.  It is
possible to use these header keywords to guide the analysis software,
but there are two difficulties.  

First, it is very common for different keywords to be used by
different cameras, sometimes even the same camera may use different
keywords for the same information at different times (major readout
software upgrades, for example, can be accompanied by keyword
revisions).  In addition, within Pan-STARRS and the IPP, it is
necessary to have the capability to refer to the Metadata database as
the authoratative sources of some of these entries rather than the
image headers.  Given this circumstance, it is at least necessary to
define the appropriate source for a given data concept appropriate to
data from a specific camera.

The second problem arises when actually performing an analysis.  In
many circumstances, the software needs to know what data to expect
even when an appropriate camera image is not available.  This is
particularly true for a camera which is composed of multiple chips and
multiple amplifiers.  It is a frequent circumstance than some subset
of the chips or amplifiers will either be unavailable or are invalid
for one reason or another.  It is important for the software to have a
guide for what data should be available from a perfect readout of the
given camera so decisions can be made how to handle data which is not
complete.  This is also important to validate that a particular
dataset, which appears to be from a known camera, actually corresponds
to that camera and has all of the necessary information where
expected.

As part of the flexible design model, all of these analysis programs
use a common set of configuration files which define the details of
how their analysis should be performed.  The configuration files a
further divided into three main sets of configuration information:
\begin{itemize}
\item {\bf site configuration} this information defines the locations
  of data resources and the other available configuration information.
  For example, the site configuration would include the location of
  Nebulous, the metadata database access information, the list of
  available cameras, and so forth.
\item {\bf camera configuration} this information describes
  characteristics needed to interpret data from a specific camera.
  This includes such details as where to extract particular metadata,
  such as the filter name.  It also defines the camera layout and the
  expected organization of the data files.  
\item {\bf recipe} a particular analysis program may use one or
  multiple recipes.  The recipe defines the value of optional
  configuration information for that analysis program.  For example,
  by using different recipes, ppImage can be made to perform a
  complete Phase 2 image analysis, including detailed object detection
  and astrometric calibration, or with a different recipe, ppImage may
  be used to perform only bias subtraction (for example, as part of
  the detrend image analysis).    Note that the details of a recipe in
  general depend on the camera / telescope of interest.  As a result,
  the identify of the recipe files which define different recipes are
  included as part of the camera configuration information.
\end{itemize}
For all of the analysis programs, the source of the configuration
files can be overridden when the program is executed, and individual
configuration values may also be specified on the command line.  The
details of the configuration file formats and the configuration
variable names used by different programs and functions are discussed
in the Modules SDRS document.

\subsection{ppImage}

This program is not only one of the work-horse programs of the IPP, it
is also exemplary of the design of the top level analysis program.
ppImage is used to perform the complete Phase 2 analysis on a single
image data file.  This includes the complete detrending discussed
above (bias, dark, flat, fringe, etc), as well as the object detection
and classification, the astrometric calibration, and potentially the
photometric calibration as well.  The object analysis and astrometry
are in fact performed by ppImage using two of the other top-level
routines discussed above, psphot and psastro; these components will be
discussed independently and will mostly not be part of the ppImage
discussion.

The top-level design of ppImage is virtually identical to that of
several other stand-alone analysis programs (eg, psphot, psastro,
ppMerge).  The input to ppImage may be one of several options:
\begin{itemize}
\item a single FITS file by name (eg: /full/path/to/file.fits)
\item a collection of FITS files by glob (eg: /full/path/to/file.*.fits)
\item a single Nebulous storage ID (neb:nebID)
\item a collection of Nebulous storage IDs (neb:nebID.regex)
\end{itemize}
The Nebulous versions can be viewed as simply utility versions to save
the user from asking Nebulous for the corresponding file names.  In
addition to the input file and any optional configuration information,
ppImage also takes as part of the command-line arguments the root of
the output file names; ppImage defines naming conventions to
constructing the complete output files based on the recipe.  

After parsing the command-line options, ppImage performs the following
tasks:
\begin{itemize}
\item identify the camera associated with the file or files (all files
  must be of the same camera). This analysis is performed by examining
  the primary headers of the files and applying camera-identification
  rules specified as part of the camera configuration information.  It
  is also possible for the user to assert the camera which supplied
  the image(s).

\item construct a complete FPA structure representing the camera
  (without pixel data) and associate the input filenames with the
  different components of the FPA structure.  In this step, the camera
  configuration information is used to identify the cell, chip, etc
  which corresponds to each file, and the filenames are associated
  with elements at the appropriate levels.  Since each input file has
  already had its primary header read, this is also attached to the
  appropriate FPA structure level.

\item the complete FPA is then looped over, with nested loops for each
  of the levels: FPA, Chip, Cell, and Readout.  At the appropriate
  depth (depending on user options and the recipe), the pixels are
  loaded from the corresponding file.  The recipe may, for example,
  require the entire FPA image be read, or it may specify that each
  cell be loaded one at a time.  

\item the detrending analysis is peformed on each readout.  Note that
  the detrend images needed for the analysis are defined by the recipe
  and by querying the metadata database for the detrend image which
  matches the science image in hand.  The details of this match may be
  modified by the recipe as well, allowing for example, one recipe to
  apply only the best available flat-field images, while another for
  comparison only applies an earlier flat.  These types of options are
  needed for testing, and also to perform, e.g., the flat-field
  correction analysis.  Also note that the detrend images may be
  loaded at a different level from the science images; it is likely
  that a detrend image is only defined for each Cell, while the
  science images may be loaded by Readout.  

\item after each chip is processed, ppImage will optionally
  reconfigure the resulting pixels into a single contiguous array.
  This is the normal data source for the psphot function call used by
  ppImage.  The reconfigured chip image on this stage may also be used
  to generate thumbnail and rebinned sample images on the chip level.

\item ppImage may also be used to reconfigure the pixels from a
  complete FPA into a single pixel array.  For the GPC, this step
  would not normally be performed on the full data array, but would be
  used to generate rebinned sample images of the full mosaic (in
  either FITS or JPEG formats).  This step is used by Phase 3 to
  construct images for examining the state of the data processing or
  by the detrend creation analysis to test the quality of the
  residuals.  
 
\item the final stage is to output, at an appropriate depth, the
  chosen output data files and to send summary metadata to the
  metadata database.  these output files may include FITS images, FITS
  thumbnails (very small rebinned images), FITS samples, JPEG images,
  object tables, and/or astrometry calibration files.
\end{itemize}

Except for the details of the analysis (detrending, etc), the detail
of the processing steps are identical for psphot, psastro, and
ppMerge, as discussed below.

\subsection{ppMerge}

This program performs the job of stacking multiple images in which the
pixel grid is kept the same for all of the input images.  As part of
the combination process, the input images may be scaled and shifted.
This operation is the basis for the creation of all of the primary
master detrend frames: bias, dark, flat, fringe.  For all of these
cases, the output pixel values are determined by applying some global
statistic to the collection of input pixels after the input pixels
have been rescaled.  The combination statistics may be any of several
standard options, including mean, median, sigma-clipping, etc.  The
input files may be supplied to ppMerge using any of a command-line
glob, a text-based list, or an XML list.  The glob method forces the
zero and scaling to have values of 0 and 1, respectively.  The text
file listing allows for zero and scaling values, but only a single
value for each input file.  The XML-format list is needed if
subdivisions within a file (eg, cells) require independent zero and
scaling factors.  It is also necessary to specify more than one image
file if, eg, a full FPA composed of separate Chip files is to be
processed at once.  This analysis is performed using a very similar
structure to ppImage: the input list of images is parsed and
associated with the approrpiate elements of the FPA struture, though
in this case, an array of FPA structures is used to carry the
information.  The elements of the FPA are looped over, as with
ppImage, and the analysis is performed on the lowest level, after the
input data pixels are read and re-scaled.  The output is written out
after the image have been looped over. 

\subsection{psphot}

The details of psphot are discussed in the psphot design document.
Here we will just address some of the basic concepts.  PSPhot may be
used as a stand-alone program, or it may be called as a function
within another program (such as ppImage).  In the function form,
psphot is passed a pmReadout structure which is populated with the
image of interest; weight and mask elements may be supplied, or else
constructed based on standard rules.  As a stand-alone program, psphot
will use a structure similar to that used by ppImage to load in all of
the image components and loop over all of the readouts, performing the
analysis on readout-by-readout basis.  Note that for the images from
the Static Sky or Phase 4, the readout structure is also used, though
in that case, the readout is contained by a SkyCell, not a cell within
an FPA heirarchy.

PSPhot performs the complete object detection and analysis process,
using the following basic outline:
\begin{itemize}
\item {\bf assess image} examine the image statistics; if needed and
  desired, fit a model to the background and subtract it.
\item {\bf detect peaks} search the image for peaks above a defined
  threshold
\item {\bf basic source stats} examine the peaks to determine the local
  sky, centroid, first and second moments and other fast, simple
  statistics.
\item {\bf rough classification} based on the source moments,
  determine the likely stellar sources and classify sources based on
  their moments in relation to the stellar source moments, as well as
  the saturation level, etc.
\item {\bf basic deblend} identify peaks which are in the wings of
  brighter sources, mark them for careful attention later.
\item {\bf choose PSF} determine a PSF model from the bright possible
  stellar sources.  PSPhot allows a wide range of possible analytical
  PSF models, and may be set to automatically select the best one for
  an image.  The parameters of a PSF are allowed to vary spatially.
  Currently, pixel-based PSFs are not implemented, but can be added as
  an upgrade.
\item {\bf fit sources} the heart of PSPhot is the analysis of the
  brighter sources in the image.  In this stage, each source (above a
  certain threshold) is fitted with the PSF model, the source is
  identified as PSF-like or not.  If not, extend source models and
  double-PSF models are also attempted.  The best of the models is
  kept, unless the fits fail complete, and the model is subtracted.
\item {\bf photometry checks} curve-of-growth and aperture corrections
  are measured and applied to the PSF photometry.
\item {\bf output stage} as a stand-alone program, psphot will produce
  output tables in various formats.
\end{itemize}

\subsection{psastro}

This section discusses the design of psastro in general terms.  For
more specific usage information, see the document on psastro.  Like
psphot, psastro may be run as a stand-alone program or as a library.
This routine performs astrometric calibration of a collection of
sources from an image or a set of images.  As a stand-alone program,
the input to psastro is an image file or set of images files
representing data from single FPA.  Like ppImage, the input image or
images may be specified as a single file, a glob, or Nebulous storage
object IDs.  The heart of the psastro routine is wrapped with a loop
similar to the one used for ppImage.  The primary difference between
the two is that psastro expect a collection of files representing
sources detected in an image, rather than the pixels from an image.
Several standard source file formats, discussed in the psphot
documentation may be read.  The sources which are loaded are added to
the FPA heirarchy at the appropriate level, as part of the metadata
collection associated with a given level.  As a library function,
psastro is passed an assembled FPA structure with the sources already
placed in the appropriate locations in the metadata containers.

Psastro has three major modes, which are used in each of Phases 1, 2,
and 3.

In the conceptually most straightforeward mode, each readout contained
in the incoming FPA structure is treated independently.  The metadata
describing the approximate astrometry of the readout are used to guess
the coordinates of the sources and to select likely reference sources
from the selected reference catalog.  The observed sources are matched
to the reference sources, using either a two-point grid search or
optionally a triangle match.  Once an approximate match is found, a
linear fit between detector coordinates an projected celestial
coordinates is attempted.  The projected coordinate system may
optionally make use of the default telescope distortion model, if it
is known.  The radius of the match between observed and reference
sources is reduced to improve the statistics of the match.  This
anaysis mode is used in the Phase 2 processing.

The second type of psastro analysis is used by Phase 3 to improve the
solution determined in Phase 3.  In this analysis, psastro expects
detections from a complete mosaic FPA of chips.  A complete mosaic
solution is performed in which a single distortion model is used for
the telescope optics and additional linear or higher order terms.
This analysis breaks the degeneracy of the chip position / telescope
distortion by fitting the local gradient of the distortion on chips to
model the telescope distortion.  The result of this analysis is a
complete mosaic astrometry model, consisting of (at the moment) 3rd
order polynomials for the telescope distortion and up to 3rd order
polynomials for each of the individual chips.

In the third mode, a collection of detections from across a full
mosaic are used, in conjunction with a model for the chip positions
and the telescope / optical distortion, to determine a single
low-order model for the complete FPA.  In this model, only the
boresite coordinates, rotation, and X and Y plate scales are allowed
to vary.  This analysis is used for the Phase 1 analysis, in
conjunction with an implementation of psphot. 

For any complete mosaic, there are three complementary representations
of the astrometric solution which are used within the IPP.  The most
basic description of the astrometry is the collection of header
keywords which define the boresite center coordinates (RA, DEC), the
location of the chip data arrays within the full mosaic (the IRAF
DATASEC keywords).  More detailed astrometric information may be
defined using the WCS keywords.  These unfortunately do not have a
standard representation of higher-order terms.  Furthermore, the two
competing systems which have been proposed define only a single
transformation frame.  In the IPP, it is important to carry around
more information which can be used to improve our astrometric
solutions in the future.  Specifically, we would like to maintain at
least the transformation for the telescope optics independent from the
individual chip warps or tilts.  Even more, we would like to have a
flexible astrometry definition format which can be extended in a
flexible fashion.  We have defined a FITS table convention to carry
all of the elements of the astrometric transformation of a full FPA.
Within the table, transformations are generally defined to convert one
layer (eg, focal plane) to another.  The form and the parameters of
the conversion make up the columns of the table.  With this structure,
it is possible to add arbitrary layers as needed.  The IPP, and
portions of the project (particularly Otis), will share a common
default astrometry model for the telescope.  This will be defined on
the basis of measurements over the first weeks of observations.  The
IPP will perform the astrometric analysis of individual images with
psastro, and the results can be saved in this tabular format.  Over
time, these result tables can be used to improve the astrometric model
for the telescope, and to improve the astrometric reference catalog.

\subsection{poisub}

Poisub is the image difference analysis program.  \tbd{finish this
discussion}.

\subsection{stac}

STAC is the program which warps and optimally combines images from the
same region of the sky.  It consists of two major stages: the warping
stage and the image combination stage with robust outlier rejection.
\tbd{update / finish this discussion}

\subsection{Command Sequences}

It is useful in order to understand the analysis sequence to examine
the complete series of processing steps involved in the analysis
stages discussed above.  We first illustrate the Phase 1-3 sequence,
giving the commands which start with a raw image available on disk and
results in a collection of detrended chip images, a high-quality
astrometric calibration, and a collection of object detections.  

\subsubsection{Phase 1-3 Analysis Commands}

In the example below, we imagine a GPC image available on disk with
the exposure ID 654321.obj and chip IDs 00 through 88.  The IPP design
does not mandate specific naming convensions for the exposure IDs and
the chip IDs; these values are opaque strings supplied by the data
source (eg, GPC).  Below, in Section~\ref{ipptools}, we discuss how
the names and inputs are constructed, and how the relationships are
tracked between an exposure and the data containers which make up the
exposure.  Also, the details of directory naming and organization are
just examples, though some nightly folder scheme is a likely option.
For the moment, these are assumed to be known by the system

Also discussed in Section~\ref{ipptools} it the concept of multiple
analysis passes for a data element.  Within the IPP, any data may be
processed multiple times; the system tracks each attempt to process a
particular set of data, tracking the analysis versions numbers which
increment sequentially for each new attempt.  In the sequence below,
we are performing the first analysis attempt on the data, so the
version numbers are all 0. 

\begin{verbatim}
Phase 1:
  ppImage -recipe PHASE1 
	  -inglob    file:/data/2006.11.01/7654321o/7654321o.??.fits 
	  -out_astro file:/data/2006.11.01/7654321o/7654321o.XX.P1.00.ast.fits

Phase 2:
  ppImage -recipe PHASE2 
	  -infile    file:/data/2006.11.01/7654321o/7654321o.24.fits
	  -in_astro  file:/data/2006.11.01/7654321o/7654321o.XX.P1.00.ast.fits
          -outfile   file:/data/2006.11.01/7654321o/7654321o.24.P2.00.img.fits
          -outmask   file:/data/2006.11.01/7654321o/7654321o.24.P2.00.msk.fits
          -outvar    file:/data/2006.11.01/7654321o/7654321o.24.P2.00.var.fits
          -objects   file:/data/2006.11.01/7654321o/7654321o.24.P2.00.cmf.fits
          -thumb     file:/data/2006.11.01/7654321o/7654321o.24.P2.00.thm.fits
          -binned    file:/data/2006.11.01/7654321o/7654321o.24.P2.00.bin.fits

Phase 3:
  ppImage -recipe PHASE3
          -inglob    file:/data/2006.11.01/7654321o/7654321o.??.P2.00.cmf.fits 
	  -out_astro file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.ast.fits
	  -objects   file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.cmf.fits

  ppImage -recipe MOSAIC
          -inglob   file:/data/2006.11.01/7654321o/7654321o.??.P2.00.thm.fits 
          -outjpeg  file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.thm.jpeg 

  ppImage -recipe MOSAIC
          -inglob   file:/data/2006.11.01/7654321o/7654321o.??.P2.00.bin.fits 
          -outjpeg  file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.bin.jpeg 
\end{verbatim}

In this example, the data are supplied by their file names in the UNIX
file system.  We are invoking the capability of ppImage to accept a
glob to supply a list of files.  In the Phase 1 stage, ppImage is used
with the PHASE1 recipe to load data from a full mosaic (the files
specified by the glob) and produce a single astrometry calibration
file.  The \code{XX} is supplied for the full-mosaic output files just
to make the output data products appear in a more easily readable
fashion is a directory listing.  Also, note that we attach the
\code{.fits} extension to all of these output files to make the data
type more explicit to the reader.  Again, none of these convensions
are required by the analysis programs.

The Phase 2 analysis uses the Phase 1 astrometry to improve the
astrometric starting guess.  The analysis for Phase 2 is illustrated
for just a single chip (24), though presumably equivalent commands are
executed for the other 63 chips.  The six output files selected in
this example include the detrended image (\code{*.img.fits}), along
with the corresponding mask (\code{*.msk.fits}) and variance images
(\code{*.var.fits}) and the photometry results file
(\code{*.cmf.fits}).  Note that the output object file contains the
astrometric solution parameters from this analysis stage.  This
process also constructs two smaller version output images: the binned
image (\code{*.bin.fits}) and the thumbnail (\code{*.thm.fits}).  The
binning scale for these images is specified in the recipe for the
camera; the first of these for GPC would likely be binned 32x32, while
the second would probably be binned 320x320.

For the Phase 3 analysis stage, three actual analyses are illustrated.
In the first, the photometry results files are identified by file glob
and the result is an improved astrometric model for the camera and
optics in our astrometry parameter table format.  The object files are
also grouped into a single multi-extension file along with the
astrometric calibration.

In the second and third analysis examples, the collection of chip
binned and thumbnail images are loaded by file glob, mosaic-ed
together into a single image, and written to disk as a JPEG.  These
images are used by the pipeline tracking tool, ippMonitor.  The binned
image results in a full GPC image represented by 1200x1200 pixels; the
thumbnail yields a 120x120 representation of the full GPC.

The example above would be sufficient if we were processing a small
number of images by hand or for test purposes.  However, the IPP is
designed to be more flexible about the physical location of the data
files that this illustration permits.  The use of nebulous allows us
to use a similar naming scheme and yet place the actual data files on
different hardware depending on the chip ID (among other
possibilities).  To convert the filename version above to a version in
which the files are stored on Nebulous simply requires changing the
\code{file:} tag to \code{neb:}.  The analysis programs recognize this
tag to indicate a file available from Nebulous, and make a request to
Nebulous for the actual file names.  Nebulous can supply files based
on a name match much like the file glob.  Nebulous also allows the
storage object ID to include path-like elements, allowing a structured
organization of the files within Nebulous (which does not reflect a
{\em physical location} relationship).

\subsubsection{Basic Detrend Creation Commands}

In the following example, we examine the steps to produce master
detrend images.  First, a few important points to note about this
process.  The construction of a master detrend frame (bias, flat, etc)
involves combining a number of individual frames of an appropriate
type of exposure, possibly after some preparatory processing.  For
example, in building a twilight flat-field image, 5 or 10 (or however
many) raw flat-field images are first masked and bias corrected before
being combined.  To build a night-time fringe-frame image, a
collection of raw night-time images are bias, dark, and flat-corrected
before they are combined.  In the combination, it may be necessary to
apply some scaling and/or offset correction to the images to place
them on a common footing.  For example, in the construction of a
master flat-field image, the individual images must be normalized in a
consistent fashion; in building a master fringe frame, the fringe
amplitude must be used as part of the scaling applied to the input
images.  In a mosaic camera, if individual chips are analysed
independently, the resulting master chip images may require
re-normalized to place the results on a common, consistent footing.

Beyond the details of the analysis steps, there is the question of the
choice of input images.  This choice is extremely dependent on the
implementation for a particular camera, telescope, type of detrend
image, etc.  The analysis {\em process} should not be designed to make
strong assumptions about the selection of the input data.  In the IPP,
the definition of the selection rules is part of the input
configuration information and the scheduling rules, and can be
considered outside of the discussion of the analysis commands.  The
IPP provides a tool, part of the \code{dettools} suite, which examines
the Metadata Database tables for raw images of the appropriate type to
select input images based on selection options such as time range,
filter, camera, chip, exposure type, airmass, exposure time, etc.  In
the discussion below, we assume that some selection is made with
\code{dettools}, resulting in a collection of input exposures and
their corresponding chips.  These lists are placed in tables which are
then provided as part of the input to the analysis programs below; the
corresponding images used as part of these inputs are also saved in
Metadata Database tables as discussed in Section~\ref{ipptools}.  In
practice, the database tables provide the primary source; the list
files are constructed from these tables and are simply intermediate
data sources for the analysis programs.

Another important distinction to clarify in the detrend processing is
between the detrend {\em run}, the {\em version}, and the {\em
iteration}.  These issues are discussed further in
Section~\ref{ipptools}.  Briefly, though, there are the following
concepts to keep in mind: The detrend {\em run} is a particular
attempt to construct a master detrend image.  One run defines a
collection of selection criteria for the initial set of input images.
The resulting master detrend image is given an identifier, equivalent
to the exposure ID.  If the same selection criteria are used multiple
times (eg, for multiple experiments on the analysis recipe used to
construct the image), the same detrend ID may be used for multiple
detrend runs.  In this case, each new detrend run is given a different
{\em version} number, equivalent to the version numbers used to track
the science analysis passes.  For the detrend image construction, this
concept must go one level further, however.  In order to produce a
single validated master detrend image, it is necessary in general to
produce multiple intermediate attempts.  The intermediate master
frames are applied to the input images; the statistics of the residual
images are then used to select a subset of the input images, rejecting
poor quality or deviant images.  This processing is a form of
image-level outlier rejection, and is particularly necessary for input
images which result from observations of the sky (eg, twilight flats
or night-time fringe frames); images obtained using stable calibration
sources may not require this level of iterative processing.  This type
of analysis can also be used to determine if a new master frame is
needed (all input images internally consistent, but deviant from the
current best master) or if conditions are unacceptable to produce a
new master (all input images mutually inconsistent).  In order to
track these multiple analysis passes, the IPP infrastructure assigns
iteration numbers to the data products associated with a particular
detrend run and version.

One final point to address is the issue of the validity domain of a
detrend image.  The end result of the detrend run is a master detrend
image of a particular type, e.g., a master r' flat-field image for
GPC-1.  As a result of the input selection criteria, the resulting
master detrend frame will have a primary domain of validity, which
consists of a particular camera, telescope, set of chips, and which
may include a time range, filter, airmass range, etc.  The primary
domain of validity defines those images which would be best processed
with the particular master detrend image.  Beyond this primary domain
of validity is a wider, relaxed domain of partial validity.

Clearly, a SkyProbe flat-field image would be inapporiate in all
context to be applied to a GPC image.  Likewise, a GPC-1 $r'$ flat would
be inappropriate for a GPC-1 $g'$ science image.  However, in some
circumstances, it is appropriate or desireable to apply a detrend
frame to an image outside of its primary domain of validity.  For
example, if a master flat-field image was produced using input images
from a certain week, an image from a different week may be viewed as
outside the primary domain; however, for some experiments or because a
flat-field in the appropriate time range could not be produced, it may
be acceptable to apply the out-of-date flat-field image to the science
image.  In general, any image of the appropriate type, camera, filter
(if a valid construct) and detector is in the partial domain of
validity as a detrend image with those same values.  The detrend
creation system assigns a primary domain of validity to the masters
which it creates; it is the choice of the analysis routines to apply
images from a more relaxed domain, if necessary or desired, or to
require the primary domain and yield an error if it is not available.

Below we give the series of analysis commands used to construct a
master detrend frame.  In this example, we construct a master $r'$
flat-field image for GPC1, using the detrend images for week 050.  In
practice, we will likely build detrend frames on a nightly basis, but
the choice of timescale will depend to some extent on the observing
process and the stability of the system.  In this example, we
construct a detrend ID using the camera, type, filter, and week
number, though this choice is completely arbitrary.  We also
illustrate the example for one of the input images with exposure ID
7654321f and for chip 24.  In this example, this is the third time
this image has been used for the analysis, thus the processing results
for this frame/chip are given a version number of 02.

\begin{verbatim}
dettools -define [selection criteria] -detID GPC1.flat.r.w050.00

ppImage -recipe MKDET.PROCESS
        -infile   file:/data/2006.11.01/7654321f/7654321f.24.fits
        -output   file:/data/2006.11.01/7654321f/7654321f.24.PC.02.img.fits

ppMerge -recipe MKDET.STACK
        -inlist   file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c24.list
        -output   file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c24.fits

ppNorm  -recipe MKDET.NORM
        -inglob   file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c??.fits

ppImage -recipe MKDET.RESID
        -infile   file:/data/2006.11.01/7654321f/7654321f.24.PC.02.img.fits
        -output   file:/data/2006.11.01/7654321f/7654321f.24.RS.06.img.fits
        -thumb    file:/data/2006.11.01/7654321f/7654321f.24.RS.06.thm.fits
        -binned   file:/data/2006.11.01/7654321f/7654321f.24.RS.06.bin.fits

ppImage -recipe MOSAIC
        -inglob   file:/data/2006.11.01/7654321o/7654321o.??.RS.06.bin.fits
        -outjpeg  file:/data/2006.11.01/7654321o/7654321o.XX.RS.06.bin.jpeg

ppImage -recipe MOSAIC
        -inglob   file:/data/2006.11.01/7654321o/7654321o.??.RS.06.thm.fits
        -outjpeg  file:/data/2006.11.01/7654321o/7654321o.XX.RS.06.thm.jpeg

ppImage -recipe MKDET.MOSAIC.
        -inglob   file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c??.fits
        -outjpgt  file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.thm.jpeg
        -outjpgb  file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.bin.jpeg

\end{verbatim}

Again, this data illustrates the use of files in the UNIX file system;
the substitution of \code{neb:} for \code{file:} will inform the
programs to retrieve the file names via Nebulous.  In this example, we
use output names for the intermediate images which are equivalent to
those used for the science processing; the version numbers used for
these data products are sequential over all detrend runs which use
those input frames.  Also, for the detrend image products, we have
added the 'v', 'n', 'c' tags to clarify which of the two-digit numbers
represents a version (v), an iteration (n), and a chip ID (c).  Note
that this sequence of analysis steps makes heavy use of
\code{ppImage}, with different choices of the recipe and the output
options to change the behavior somewhat.  However, all of the uses of
\code{ppImage} represented here are consistent with the primary
responsibilities of \code{ppImage}: read in the file or mosaic image
into the correct level of the image hierarchy, perform a detrend
analysis (including rebinning in this category), re-structure the
collection of image arrays as described by the recipe, write out the
image in the desired format.

\section{IPPTools}
\label{sec:ipptools}

PanTasks is the IPP tool which manages the sequencing of data analysis
steps and, with the related tool `PControl', distributes the data
processing across a cluster of computers. However, by itself, PanTasks
does not determine the organization of data or the analysis sequencing
for a particular pipeline.  This level of information is contained
within specific PanTasks scripts.  To use the tasks defined by a set
of PanTasks scripts, additional helper programs are needed.  This
section discusses these programs and the PanTasks scripts used by the
IPP.

IPPTools is a collection of programs, Metadata Database table
definitions, and PanTasks scripts used to define the actual data
organization and the sequencing of operations by the IPP.  Within the
IPP, the Metadata Database is used to store the analysis state, as
well as result processing data points.  This section discusses the
tasks needed to define each of the IPP analysis stages (Phase 1-4,
detrend creation, etc) and examines the relevant MDDB tables.  

\subsection{Persistent vs Ephemeral State in PanTasks}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/ipptools.01.ps}
\caption{\label{queues} PanTasks queues and MDDB tables}
\end{center}
\end{figure}

The IPP, a fairly complex analysis system, uses PanTasks to select
jobs, distribute them to the cluster, and harvest the results.  It
uses the Metadata Database to record the results of a given analysis
step, and to determine which jobs must be performed when.

There are some subtleties in the interaction between PanTasks, the
Metadata Database tables which store the system state, and the jobs
which are currently being performed.  There is a choice to be made
between rigorously maintaining the system state in the Metadata DB at
all times or keeping an intermediate set of state tables.  Keeping the
exact system state in the Metadata DB tables would require many extra
queries to/from the database and may introduce additional latencies
which are undesirable.  This is because any attempt by PanTasks to
initiate a new job would require PanTasks to mark the corresponding
data item in the Metadata DB (the item which acts as the trigger) with
a `pending' state, and then mark it again as `done' when the job
actually completes.  This also has the drawback that, if the system
crashes (eg, hardware failure), some initial process would be required
on start up to find all Metadata DB items which are in the `pending'
state (examining all possible items which can be in such a state) and
reset them to the `new' state.

We implement an alternative in which PanTasks maintains an internal,
ephemeral stack of the pending jobs, and only updates the system state
entries in the Metadata DB when jobs are actually completed.  In this
scenario, as far as the Metadata DB tables are concerned, data items
transition only between a `new' and a `done' state.  Any jobs which
are pending when the system crashes or the power is lost are simply
dropped, and will be automatically re-constructed when the system
restarts.  In this paradigm, no intermediate operation state is saved,
and no partially completed job can be recovered.  Since the IPP is
defined in terms of a fine granularity, with jobs lasting no more than
30 - 120 seconds, crashes under this model will not have a large
impact on the data processing.

Figure~\ref{queues} illustrates this ephemeral vs persistent state
information and the interrelation between the metadata tables and
PanTasks.  The left-hand portion of the diagram illustrates the
recommended interaction between the metadata database tables and
PanTasks' internal queues.  Some table in the metadata database
defines a list of data items which are to be processed by some
analysis job.  PanTasks uses a two-step approach to define the
analysis jobs based on this list.  First, one task queries the MDDB
for a list of pending items, adds the returned items to an internal
PanTasks queue.  The process of adding the elements to the queue is
defined so that only unique items are added: already existing items
are skipped.  The entries in the queue consist of the data items of
interest and an internal temporary state.  At first, this would be
`pending'.  A second tasks pops `pending' entries one-by-one from this
internal queue, submits a job based on the entry, and sets the
temporary state in the internal queue to `running'.  The internal
state is needed to prevent PanTasks from re-submitting a job for the
same data item before the first job is done or assessed.  Since the
job make take an arbitrary amount of time, PanTasks requires a
mechanism to remember which data items it has already submitted.  When
the job eventually completes, the metadata database table is updated
noting the completion.  This may be done either by the job itself or
by PanTasks as part of the job exit rules.  In addition, the state of
the entry in the queue can be set to either `done' or the entry can be
simply removed from the queue.


The purpose of this interaction is to maintain the temporary state
information within non-persistent elements of PanTasks rather than
using the metadata database tables to store this information.  This
concept has two advantages.  First, PanTasks internal queues are in
memory and relatively small, thus interfacing with them is quite fast
for PanTasks -- this should reduce the system latency.  Second, by
keeping this information non-persistent, the system responds correctly
to stopping and restarting PanTasks.  Any jobs which have not been
completed will not be marked in the database, and will be restarted
naturally by PanTasks.  The alternative, of writing a temporary state
marker in the database would require PanTasks, on startup, to
initially clean all database tables of these temporary state markers.

The right-hand portion of the diagram illustrates this process using
the process of copying the images from the summit as an example.  The
metadata database table of interest in this case is the list of
pending images, with entries supplied by a job which queries the
summit data systems.  The job which is actually performed is a remote
copy of the image file from the location specified by the summit data
system to the appropriate location within the IPP Image Server
(Nebulous).  (As an alternative to the above, the `pending images'
table may be part of the summit database system, and the `get images'
command may query the summit directly.  In this scenario, the `copy
image' command reports to the summit data system that an individual
image file has been copied.)

In the rest of this document, the use of PanTasks internal queues to
manage the temporary data states is glossed over and assumed part of
the tasks defined in the process.


\subsection{IPP Pipelines Overview}

\tbd{add the use of Q/A measurements from the IPP CDR Response
document}

The IPP as a whole performs all of the image analysis functions
required by the Pan-STARRS telescopes, including images from the full
Gigapixel camera (or cameras), the test camera TC-3, and the SkyProbe
camera.  The IPP is designed to be very flexible, with instrument
specific details isolated in configuration files associated with the
different cameras known to the system.  As a result, the organization
of the top level analysis infrastructure must be sufficiently general
that a wide range of cameras can be accomodated.  We have a few
general principles regarding constraints on the data to be processed
which are used to guide the IPP design and developement:

\begin{itemize}
\item {\bf Camera Focal Plane Hierarchy} The IPP analysis programs
  assume that the images to be processed are obtained by a camera
  which can be represented by our Camera Focal-Plane Hierarchy of data
  structures.  This hierarchy is discussed in detail in the Modules
  SDRS, and defines a top-level {\em Focal-Plane Array (FPA)}, which
  may contain 1 or more {\em Chips}, each of which may contain one or
  more {\em Cells}.  An {\em FPA} is identified as having a single
  optical system feeding photons to the detectors.  A {\em Chip} is
  identified as a unit of data all deriving from a single detector
  (piece of silicon), while a {\em Cell} is identified as a collection
  of pixels read out as a continuous cartesian grid.  Finally, a
  single collection of data from an {\em FPA} may include multiple
  {\em Readouts} from any or all of the {\em Cells}.  

\item {\bf Exposures vs Groups} The processing presumes that the data
  is organized into {\em exposures} and exposure {\em groups}.  An
  exposure represents the data from a single FPA, with the possible
  subdivision of the exposure into multiple readouts for some or all
  of the cells.  Exposure {\em Groups} are any group of exposures
  which are related together in some way; the definition of the {\em
  Groups} may be provided by the observers, or they may be derived
  from the characteristics of the exposures.  The use of a particular
  {\em group} depends on the context of that group.  A few examples of
  exposure groups:

  \begin{itemize}
  \item a dithered sequence of exposures to be stacked for cosmetics
  and improved signal-to-noise.
  \item a twilight flat-field sequence.
  \item all images of the same filter within a 10 degree region to be
  used to construct an sample astrometric reference.  
  \end{itemize}

\item {\bf Image Files (imfiles) vs Exposures}  Any single exposure
  may consist of a number of different data files.  The number of {\em
  imfiles} for a given exposure will depend on the camera, as will the
  data organization within those image files.  Also, a particular
  camera will supply files corresponding to one of the particular
  Focal-Plane Hierarchy elements.  The IPP analysis must be able to
  interpret the incoming data correctly.
\end{itemize}

As discussed elsewhere, there are several major types of analysis
performed by the IPP.  For the purposes of data organization and
parallel processing efficiencies, we have identified the following
divisions of the analysis tasks.  These will be discuss in much more
detail below.

\begin{itemize}
\item {\bf Science Image Analysis} : This represents the analysis
  performed on the images obtained by the telescope, and generally
  performed in real-time, night-by-night.  The science image analysis
  tasks are further subdivided as follows:

  \begin{itemize}
  \item {\bf Phase 1} : The full focal-plane array is examined quickly
  to determine an initial astrometric calibration.  In this step, the
  OTA guide stars may be used as the astrometric reference; if none
  are available, predicted bright star positions are examined.  This
  step is only used for mosaic images, and may be skipped if no guide
  stars are available {\em and} the astrometric calibration for the
  telescope / camera is reliable (better than 10 arcseconds).

  \item {\bf Phase 2} : Each image file is analysed independently: the
  image is detrended (bias, dark, flat, fringe, etc), sources are then
  detected to a modest level, improved astrometric calibration is
  performed.

  \item {\bf Phase 3} : The collection of sources measured from all of
  the image files for the camera are used to determine a global
  astrometric, and possibly photometric, solution for the exposure.
  This step is only required for mosaic cameras.

  \item {\bf Phase 4.1} : An exposure group consisting of images
  obtained in a specific region of the sky are merged together.  In
  this step, the images are first warped to a common pixel grid, defined by
  the static sky images.  The collection of images are then used to
  construct a single, cleaned image by rejecting the outliers from the
  source images in the stack.  The corresponding static sky pixels are
  then used to construct a difference image from the resulting stack.

  \item {\bf Magic} : In this step, the difference images are examined
  to find the trailed images introduced by artificial satelites.
  These so-called {\em streaks} are excised from the difference
  images, as well as all of the source images which were used to
  generate the difference images; the public data sources are updated
  with the precise, correct time.  Note that this step requires that
  separate difference images be generated for each of the input
  images, a step which would be skipped if {\em magic} were avoided.
  Also note that, until {\em magic} is performed, the publically
  available time has a limited precision (probably $\sim 1$ minute
  errors).  This step is only necessary in the operational IPP system
  given the restrictions from the Air Force.

  \item {\bf Phase 4.2} : After {\em magic} the final difference and
  the final cleaned stacked image are produced and objects in both
  images are detected.  The difference sources are used to mask the
  extreme outliers in the cleaned stack, which is then used to update
  the Static Sky images. 
  \end{itemize}

 \item {\bf Static Sky Image Analysis} : While the science image
 analysis is performed as images are availablef, the static sky image
 analysi occurs on a very different timescale.  In steady state, the
 full static sky analysis will take place over the course of a full
 year.  At any given time, the portion of the sky corresponding to the
 location of the sun will be under-going the analysis.  In practice,
 for PS-1, the static sky is produced in a somewhat different fashion
 than in the steady-state model.  In PS-1, the different survey
 strategies introduce very different update rates for the static sky.
 At one extreme, the AP Survey will not have enough data for a
 complete static sky analysis until nearly 22 months after the survey
 begins.  At the other extreme, the deep survey, which observes a much
 smaller portion of the sky, may best be analysed quite frequently.
 These details are part of the science guidelines of the PS-1 surveys,
 and are beyond the scope of this document.  Rather, the IPP Static
 Sky Image Analysis must provide the capability of defining the static
 sky analysis in a flexible and dynamic fashion.

\item {\bf Basic Detrend Creation Analysis} : The analysis of most of
  the detrend data is grouped together in a common analysis stage.
  The differences between the analysis of the bias, dark, flat, and
  fringe images is primarily one of how the input images are
  pre-processed, what statistic is used to characterize a given input
  image, how the input images are scaled before being combined, and
  what normalization is applied to the resulting image.  All of these
  types of detrend images can thus be processed with a single analysis
  pipeline which is made aware of these minor differences.  This stage
  is never the less fairly complex, and as a result is subdivided into
  several compenents, as discussed below.

\item {\bf Other analyses} There are a number of other tasks which the
  IPP must perform that are not well-defined by the different analysis
  types discussed above.  Some analysis tasks are not automatically
  triggered, and are thus outside the scope of this document; these
  are the tasks which are more properly considered as research
  projects than analysis systems.  The other important automatic tasks
  are:
  \begin{itemize}
    \item {\bf Summit Copy} : In this stage, the data source or data
    sources are queried for new exposures and image files, which are
    then copied to the IPP data area.  This stage also includes the
    copying of other metadata which are not included in the image
    files.
    
    \item {\bf Image Classification} : new images which are introduced
    to the IPP are examined by this analysis stage and placed in the
    appropriate table for processing.  This step includes a small
    amount of accumulating statistics about the images.

    \item {\bf Data File management} : a few tasks are necessary to
    monitor and maintain the clustered storage system.  These tasks
    include the automatic duplication and deletion of different types
    of files from Nebulous, the file storage archive.  This also
    includes automatic redistribution of machine assignments as
    hardware is added or removed from the system.  This collection of
    tasks also includes monitoring of system parameters to alert
    people in case of dangerous hardware situations.

    \item {\bf Irregular Calibration Data} certain types of
    calibration information is extracted on different intervals from
    the more regular detrend images.  These types of calibration data
    include improved telescope pointing models, astrometric
    calibrations, photometric calibrations, flat-field correction
    frames.
  \end{itemize}
\end{itemize}

\subsection{Tables, Tasks and Tools}

The following sections discuss the database tables, the tasks within
PanTasks, and the collection of programs used by PanTasks to examine
and manipulate the state tables.  These later programs do not, in
general, perform any in depth analysis; instead they perform actions
such as selecting from one table images ready for analysis in a
following processing step.  This collection of tools is grouped under
the name of the {\tt ippTools}, and consists of a separate tool for
each of the different major analysis steps.

The {\tt ippTools} make use of {\em glueforge} to simplify the
management of the database table schema.  Glueforge provides a single
mechanism to generate a collection of C data structures, database
tables, database access APIs, and I/O routines from a simple table
description configuration file.  All APIs generated by {\em glueforge}
for the same type of interaction have common naming schemes.  This
technique has several important advantages.  It makes the writing of C
database interactions very quick and easy.  It also makes it easy to
modify the database schema without disrupting the software
development.  Finally, it provides a simple, self-documenting source
for data structure of multiple types which can be shared between
programs or platforms.

Within the following diagrams, we illustrate the database tables used
to track the state of the IPP.  We also show the commands provided by
{\tt ippTools} to connect the tables.  Finally, we show the IPP tasks
which initiate the different analysis steps.  The following set of
diagrams uses several consistent features.  The blue-and-grey
rectangles define the metadata database tables.  The blue section
contains the table name, while the grey section lists a minimal subset
of the table columns.  The ellipses represent programs (or program
portions in some cases) executed by PanTasks.  The blue filled
ellipses represent the {\tt ippTools} commands which are executed
locally on the computer hosting PanTasks.  The grey-blue ellipses
represent the commands executed on the parallel cluster, monitored by
{\tt pcontrol}.  The green ellipses represent commands executed by
hand for testing and manual intervention.

In most of the analysis tasks, we use a two-table approach to the data
in order to avoid excessive latencies.  One table is used to track
quantities which are still pending for a particular stage.  When the
analysis is completed, these items are moved from the 'pending' tables
to corresponding 'done' tables.  Although this introduces a somewhat
higher number of tables and complexity, it will avoid the system from
slowing down as the number of data items grows with time.  The pending
tables are searched repeatedly by the {\tt ippTools} programs as they
attempt to select new data of interest.  In contrast, the done tables
are searched much less frequently.  

\subsection{Summit Copy Tasks}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/ipptools.02.ps}
\caption{\label{pcopy} Summit Copy Tasks}
\end{center}
\end{figure}

Figure~\ref{pcopy} illustrates the MDDB tables used to copy data
(images and metadata tables) from the summit.  The left-hand portion
of the diagram shows the tables involved in copying images from the
summit system.  The table of pending image files lists the URLs of the
individual image files available for transfer, along with their
associated exposure ID and the camera which generated the image.  Two
other entries assist in interpreting the file: the class and the class
ID.  The final entry in this table is the current copy state of the
file, can have the value of `ready' or `copied'.

The class defines the data grouping represented by this image file and
may have values of: FPA, Chip, Cell.  This value indicates that the
provided image file represents the specified portion of the camera
FPA.  If the value is FPA, the file represents data from a complete
FPA, though the file may contain pixel data in multiple extensions or
other groupings to be identified later.  If the value is chip, the
file contains only data for a single chip, presumably of multiple
chips available, and equivalently for Cell.  Further discussion of the
FPA image hierarchy is given in the IPP documents (eg, Modules SDRS).
The class ID gives the identifier used to name the class level
corresponding to this file.  This value is necessary to make decisions
on how to copy the data based on the chip / cell before the data is
available to IPP components.  Table~\ref{classes} lists likely values
for the class and class ID for some common cameras.  The system
described is sufficiently flexible to allow us to transfer the GPC
images by cell if we eventually decide that is more efficient.

The copy process copies the file from the given URL to the appropriate
IPP node and adds an entry to the table of new image files, consisting
of the same information as the pending image file table, though with a
new value for the URL.  This URL may be an explicit filename, a
reference to an entry in the image server, or a web address, or
located on the image server (marked with file:, neb:, and http:,
respectively).  (TBD: other possible file storage types?  perhaps the
path could be abstracted without going to the level of the image
server?  eg: ref:DIR0001/file0001.fits might be in a directory which
is defined in a table of directories.) After an image file is
successfully copied, the corresponding state in the `pending chip'
table is updated from `ready' to `copied'.

\begin{table}
\begin{center}
\caption{Camera and Data Classes\label{classes}}
\begin{tabular}{llll}
\hline
\hline
camera   & class  & classID \\
\hline
GPC	 & chip   & chip02 \\
skyprobe & fpa 	  & sp01 \\
Megacam  & fpa	  & MegacamSpliced \\
Suprime	 & chip	  & chip0 \\
\hline
\end{tabular}
\end{center}
\end{table}

The right hand portion of this diagram illustrates the process of
copying a metadata table.  The table of pending tables lists the URLs
for the tables which are ready, a unique table ID for each table, and
the table type.  The copy function copies the listed table and uploads
the data to the IPP version of the same metadata database.  Two
examples of metadata tables needed by the IPP for the basic image
processing system are illustrated: the table of new exposures and the
table of pending matches.  The first lists the exposures which are
avilable from the summit system, and all represent entries which are
available from the Image server.  the second represents the matches
between exposure IDs and chips

\subsection{Phase 0}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/ipptools.03.ps}
\caption{\label{phase0} Phase 0 Tasks}
\end{center}
\end{figure}

Figure~\ref{phase0} illustrates phase 0, in which the image files are
categorised, examined for summary information and basic statistics,
and moved to the later phase 'pending' tables to trigger further
analysis.  The command {\tt p0search -pending} examines the `new
imfiles' and 'new exposure' tables.  It selects images from this table
which have not yet been examined (state is `new').  These are returned
to PanTasks, which sends each image file to a separate analysis node
running the {\tt p0search -update} command.  With this command, the
file header is examined and relevant metadata is extracted (eg, RA,
DEC, times, and so forth to be defined later).  The process may also
select a portion of the image pixel data to determine a rough bias and
background level.  These statistics, whether derived from the header
or the pixel values, are placed along with image summary information
in the `raw image files' table, and the state field of the `new image
files' table is set to `ready'.

The {\tt p0search -update} command is also responsible for moving the
exposures to the tables used for triggering the analysis process.  If
the image class is FPA, the image can be advanced without waiting for
any other image files.  If the class is Chip or Cell, the process must
also examine the `new exposure' table for this exposure ID.  The
number of class files available for this exposure is listed in this
table.  The process must the select all image files matching the
exposure ID with state of `ready' and compare the number avalable to
the number expected.  If the two match, then a new exposure is ready.
Based on the image type (from the most recently examined image file
header or new exp table?), the exposure is added to the `raw exposure'
table for images of that type.  The allowed types are `detrend', (all
bias, dark, flat images), `object', `focus'(??), etc.  The different
tables represent different analysis modes.  This process also adds an
entry to the exp ID / image file match.  This process also adds all
science (OBJECT) exposures to the P1 exposure table (for mosaic data)
or the P2 chip table (for single detector data).  These tables are
used to trigger the Phase 1 and Phase 2 analysis stages.

\subsection{Phase 1}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/ipptools.04.ps}
\caption{\label{phase1} Phase 1 Tasks}
\end{center}
\end{figure}

Figure~\ref{phase1} shows the tables involved in running the Phase 1
analysis stage.  There are paths for exposures to enter the analysis
automatically from the Phase 0 analysis (arrow on left) or to be added
manually based on a selection from the raw exposure table.  Exposures
to be analysed by Phase 1 are added to the P1 exposure table with the
state `new'.  Exposures may be added multiple times for processing and
reprocessing. The P1 done exposure table keeps a record of the old
attempts for debugging and analysis.  Each time an exposure is added
to the P1 exp table, it is given a new, unique version number,
allowing the system as a whole to track different analysis attempts.
This method is used in all of the image analysis stages (and
extrapolated to iterations in the detrend analysis steps below).  The
top portion of the diagram shows the use of the command {\tt p1search
-define} to select and submit an exposure or a group of exposures,
potentially selected on the basis of a query from the raw science
exposure table.

The P1 pending exposure table is examined by {\tt p1search -pending}
to select the new exposures, which are sent to PanTasks.  PanTasks
initiates a separate analysis job (p1astro) for each exposure, which
are sent to the parallel processing nodes.  Within the analysis job,
the chips (image files) associated with the exposure are select from
the raw image file table.  The analysis examines the contents of these
files, either extract the guide star information from the image files
(GS table extension) or searches for and centroids the pixels on
appropriate bright stars.  The analysis results in astrometric
calibration terms which are written to the astrometric calibration
file for this exposure.  The location of the astrometric calibration
file and the statistics of the measurement are written back to the P1
exposure table.  The images associated with exposures which are
successfully processed by P1 are then added to the P2 image table,
which is used to trigger the Phase 2 analysis.  This last step is
performed by the command {\tt p1search -done}, which is executed
regularly to search for completed Phase 1 jobs.

\subsection{Phase 2}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/ipptools.05.ps}
\caption{\label{phase2} Phase 2 Tasks}
\end{center}
\end{figure}

Figure~\ref{phase2} shows the tables involved in running the P2
analysis stage.  There are paths for images to enter the analysis
automatically from the P1 analysis (arrow on left) or to be added
manually based on a selection from the raw exposure and raw image file
tables.  Image files to be analysed by Phase 2 are added to the P2
pending imfiles table with the state `new'.  When images are added to
this table, a single entry is also added to the P2 exposure table
listing the P1 and P2 versions for this exposure.  These version
numbers must be integers starting with 1.  If this image did not have
a P1 analysis, the P1 version is set to 0.  Exposures may be added
multiple times for processing and reprocessing. The P2 image table
keeps a record of the old attempts for debugging and analysis.  As
with P1, each time a collection of associated images from an exposure
is added to the P2 image table, it is given a new, unique version
number, allowing the system as a whole to track different analysis
attempts.  Note that these version numbers are unique for each {\em
exposure} processed by Phase 2, not just for any image file.  The top
portion of the diagram illustrates the behavior of the commands {\tt
p2search -define} and {\tt p2search -quick}.  The first may be used to
re-submit the images for an exposure or a group of exposures,
potentially selected on the basis of a query from the raw science
exposure and raw image file tables.  The second version sends images
files directly to PanTasks for processing; these entries will not be
included in the processing tables, and is used only for testing
purposes.

The P2 pending image table is examined with the command {\tt p2search
  -pending} to select the `new' images.  These images are used by
PanTasks to generate P2 analysis jobs, running the analysis command
{\tt ppImage}.  The P2 analysis uses the input url to find and load
the image file.  The url may be a file on disk, an entry in the image
server, Nebulous, etc.  The master detrend images matching the
specific science image and the conditions are selected by examining
the table of master detrend frames.  The specific detrend image files
are selected by using the master detrend ID to select the matching the
entries in the table of master detrend files.  After the analysis, the
output image, mask, and FITS table of objects, including the
astrometry calibration, are written back to the P2 image table, along
with summary statistics from the P2 analysis.  The state is also
updated (to `done').

The completed images are examined by the command {\tt p2search -done},
and when all image files for a single exposure are completed, this
command migrates them to the P2 done table.  This process is also
responsible for populating the P3 pending tables so exposures may be
processing by Phase 3.

\subsection{Phase 3}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/ipptools.06.ps}
\caption{\label{phase3} Phase 3 Tasks}
\end{center}
\end{figure}

Figure~\ref{phase3} illustrates the tables and commands involved in
the Phase 3 analysis.  The P3 pending exposure table lists the
exposure ID, the P3 analysis version, the P2 analysis version to be
used as input to this P3 analysis, and the recipe to be used.  The
command {\tt p3search -pending} extracts exposures from this table and
provides them to PanTasks for processing.  PanTasks launches a Phase 3
analysis (the command {\tt psastro}?) for each exposure.  In this
analysis, the P2 exposure and image tables are used, in conjunction
with the P2 version information, to select the P2 output measured
objects and the astrometric calibrations from P2 and P1.  These
measured objects are matched with the reference catalog objects, and
calibrated astrometry {\em and eventually photometry} is produced for
the full exposure.  The location of the resulting astometry
calibration table is stored back in the P3 exposure table.  If the
recipe file specifies, the 2-D photometric and background / fringe
corrections may also be performed at this stage.  Since these analyses
require reference data, the recipe may also be used to skip these
analysis if such reference data is unavailable or unreliable.  At the
end of Phase 3, the objects from the exposure are inserted into the
photometry database (this is not shown).

The astrometric calibration portion of Phase 3 is principally needed
for a mosaic camera.  For single-chip cameras, Phase 3 may be used to
perform the photometric calibration and simply pass the astrometric
results along to the output file to be listed in the P3 exposure
table.  In this way, later stages of the analysis (ie, Phase 4) can
use the P3 exposure table as input for all cameras, even if all the
funcionality of Phase 3 is not necessary for that camera.  This would
be the case for the skyprobe camera, for example.

\subsection{Phase 4}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/ipptools.07.ps}
\caption{\label{phase4} Phase 4 Tasks}
\end{center}
\end{figure}

At the end of Phase 3, the images are ready for Phase 4.  The Phase 3
diagram shows the output line adding the exposures to be processed by
Phase 4 to a Phase 4 table.  However, this line is just for
illustration purposes.  The rules for initiating a Phase 4 run are
somewhat more complicated than those for running Phases 1-3.  Groups
of exposures which have an appropriate overlap should be chosen for
the Phase 4 analysis.  In the steady-state period of PS-4, it may be
straightforward to choose the exposure groups: they would simply be
the exposures obtained nearly simultaneously by the four separate
cameras.  The circumstance for PS-1 will be much more complicated (and
even PS-4 will probably be more complex than it seems at first
glance).  For example, in PS-1, we will not have a static sky for most
of the AP Survery.  In this circumstance, we cannot run P4, at least
until after the complete AP Reference catalog is built, and
potentially all exposures re-run through Phase 3.  It may be useful
for the AP Survey data to split the Phase 4 analysis into two stages:
image combination and image differencing.  It may even be the case
that only the combination portion of Phase 4 is performed on the AP
Survey data.

More generally, the image groups selected for Phase 4 analysis may be
chosen on the basis of a query of the AP Database (DVO) with some
rules.  

\tbd{Phase 4 run can be defined by selecting an observation group, a
  set of exposures, or a set of rules related to a spatial region (eg,
  region, time range, and filter}.

\tbd{Phase 4 discussion (and diagram) needs more work}

\subsection{Analysis Version and Recipes}

Note that each of the stages P1-P4 refer to the processing version
from the previous stage.  This allows the processing stage to request
the correct version of the results from the previous stage, and makes
it possible to run and re-run the analysis at any stage without
deleting the earlier results.  As different analysis attempts are
performed for a given image, the versions branch out.

Also note that at every stage, the entries include a recipe
identifier.  This is used to select the analysis recipe which should
be used for this version.  By default, the recipe should be set to the
current best recipe (use a default name for this?).  This feature
allows the user to run test analyses with variations on the recipe
without altering the analysis system.  For example, it is possible to
use a different flat-field set by specifying alternate rules for the
flat-field selection in a recipe file.  If it is necessary to run the
P1-P3 analysis with the raw master flats, for example, the user simply
defines that selection in the recipe file and submits the images of
interest to P1 (or P2, etc), with the corresponding entry for the
recipe.

The recipe file may also be used to specify alternative analysis paths
and desitinations.  For example, it is not necessary that all analysis
stops with P4: the recipe file may be used to halt the analysis at P2
or P3.  In addition, the recipe file may be used to specify an
alternative destination for the output results.  For example, to
generate the photometric flat-field correction frame from a collection
of dithered images, the user may not want the photometry results in
the main DVO database.  By using the recipe to set an alternative DVO
database target, and by specifying the use of the raw master flat
rather than the corrected one, the analysis of the dithered images is
kept isolated from the other photometry database results.  The
resulting photometry may be used to construct the new, corrected
flat-field images, and the processing of the same images using the new
flat-field images may be sent to the master DVO database.  

\subsection{Basic Detrend Creation}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/ipptools.08.ps}
\caption{\label{detrend} Detrend Creation Tasks}
\end{center}
\end{figure}

Figure~\ref{detrend} illustrates the tables needed for the generic
detrend construction process, using the flat-field construction as an
example.  This diagram is somewhat more complex than the preceeding
versions.  In this diagram, both single jobs and multiple jobs are
represented by the process elements (the blue ellipses).  In some
cases, more that one task will be needed to perform the function
illustrated by a single process task.  The complexity of this diagram
is enhanced by the need for multiple iterations and both single chip
and full mosaic processing.  At the moment, the distinction between
mosaic and single chip cameras is not specifically discussed.
Finally, the triggers which initiate a specific detrend analysis are
glossed over.

The detrend analysis is initiated by choosing a type of detrend image
to be constructed and by specifying the criteria which will be used to
select the input raw detrend frames for the construction.  For
example, these criteria could specify that all twilight flat images
over a certain period of days, perhaps with restrictions on the flux
levels or the time-from-sunset of the images.  The detrend analysis
run is given an ID (det ID) which will also be used to identify the
resulting master detrend frame.  

Given the definition of a master detrend run, the input exposures are
selected from the raw detrend exposure table, and written to the input
detrend exposure table.  In the next step, the corresponding image
files are selected from the table of raw image files.  Since there
will be a different set of input raw images for each attempt at
creating a master detrend image, and since any given attempt may use
some of the same input images as any other attempt, a separate table
of input raw images is constructed.  

Each of the input raw images may be pre-processed before it may be
used to construct the detrend frame.  For example, the input
flat-field images should (probably) be dark- and bias-corrected before
they are stacked.  The information about these input processed images
is written to the input images table.  If no processing is needed,
this step simply copies the appropriate information to the table, and
points back to the raw image, rather than a processed version.  

The input processed images are combined (stacked) to create a master
detrend image for the particular data element defined by the image
class (chip/cell/fpa).  At this stage, not all input images should
necessarily be included in the stack.  If residual statistics have
been measured for the input images (say, using a prior stack), then
some of the input image may be excluded.  The table of residual images
is used to guide this process.  The information describing the
resulting master image is written to the master images table.  

The statistics of the master detrend images must examined so that any
necessary renormalizations may be performed.  For example, after
stacking the individual flat images, the resulting stacks must be
renomalized to account for the different ranges of input image fluxes.
This analysis is least-squares solution in which an appropriate scale
is determined for each input exposure and a separate gain is
determined for each of the chips or cells in the camera.  This
analysis can only performed after all image stacks (ie, for all chips)
have been constructed.  The resulting information is written to the
table of master detrend frames.  

Once the master detrend is constructed, the master detrend images may
be used to construct residual images for each of the input images.
These residual statistics, as well as the locations of the residual
images and other related data products (jpeg thumbnails?) are written
to the residual image table.  Note the red arrow which by-passes the
stack construction and merge steps and skips directly to the residual
analysis.  In some cases, it may be useful to have the input images
confronted with an existing detrend image, and the resulting residual
values used to guide the rest of the process.  For example, in the
flat-field analysis, applying an earlier flat can result in a very
good first-pass rejection of poor input images.  The logic to make
this leap must be part of PanTasks, since each of the individual
blocks represent complete processing jobs.

Finally, the residual statistics from the complete mosaic (all input
images, all chips) are used to assess the quality of the newly
constructed master detrend image, and to potentially modify the
selection of input images.  This latter process is performed by
marking the state of the residual images from this iteration.  The
stacking process always examines the state information for the
residual images from the previous iteration, if it exists, when
constructing the master stack.  Once a master detrend frame has been
judged of high enough quality, the state of the entry for the frame in
the master detrend frames table is set to an appropriate value to tell
the other routines that this image should be used as a master detrend.
The exact choice of which master detrend frame is used for a given
science image depends on the recipe along with information such as the
time period used or the conditions used.

Note that, although this discussion focuses on the construction of
flat-field images, the same structure should be capable of
constructing the biases, dark, fringes, etc.  In some cases, as noted
above, the `process' stage is a null operation.

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/ipptools.09.ps}
\caption{\label{detprocess} Detrend Creation : Process Tasks}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/ipptools.10.ps}
\caption{\label{detresid} Detrend Creation : Residual Tasks}
\end{center}
\end{figure}

\begin{figure}
\begin{center}
\includegraphics[scale=0.85]{pics/ipptools.11.ps}
\caption{\label{detstack} Detrend Creation : Stack and Norm}
\end{center}
\end{figure}

\section{Interfaces}
\label{sec:interfaces}

\subsection{Internal Interfaces}

Internal interfaces consist of interactions between the analysis
scripts and the IPP Metadata Database, Image Server or AP Database.
There are also interfaces between the IPP Scheduler, Controller, and
the Metadata Database.  

The science and calibration image processing pipelines make requests
for images from the Image Server, metadata from the Metadata Database,
and push their results back onto the Image Server and Metadata
Database.  The Scheduler specifies analysis tasks and sends them to
the Controller, and determines the next action based on the contents
of the Metadata Database.  The various subsystems specify the API for
client / server interactions, and are discussed in their individual
section.  Commands will be sent using either text-based commands, SOAP
or an equivalent protocol.  The format of the exchanged data may be in
one of several forms discussed below.

FITS Images will be used to transport images between the components of
the IPP.  The Static Sky FITS Images will use standard rectangular
FITS images and mask values to identify pixels which are included in a
specific image.  Astrometric transformations will be stored in the
image headers.

FITS Tables will be used to store and transport tabular data,
especially large queries from database subsystems.  The Auto-coding
technique discussed in Appendix~\ref{sec:AutocodeIO} is used to define many
different table interactions.

SQL queries and C wrappers of SQL queries will be used as the direct
interface to the databases.

\begin{tabular}{lll}
{\bf Interface}           & {\bf Data}          & {\bf Format} \\
ippTools    : PanTasks    & image state info    & psMetadataConfig \\
PanTasks    : ippTools    & image state updates & ippTools CLI, psMetadataConfig \\
PanTasks    : ppImage     & image operations    & ppImage CLI, recipe files \\
ppImage     : psphot      & detrend images      & pmReadout in pmFPA structure \\
ppImage     : psastro     & detected objects    & pmSources in pmFPA structure \\
DVO/getstar : psastro     & ref object coords   & FITS Tables \\
psphot      : DVO/addstar & detection tables    & FITS Tables \\
psastro     : DVO/addstar & astrometry          & FITS Header \\
ippTools    : ppImage     & detrend images      & FITS Images, psMetadataConfig \\
PanTasks    : ppMerge     & image operations    & ppMerge CLI, recipe files \\
PanTasks    : ppStats     & image info       	& ppStats CLI, recipe files \\
PanTasks    : ppNorm      & image results    	& ppNorm CLI, recipe files \\
DVO         : relphot     & DVO database        & FITS Tables \\
DVO         : relastro    & DVO database        & FITS Tables \\
psastro     : stac        & astrometry          & FITS Header \\
psphot      : poisub      & psf model           & psMetadataConfig \\
psphot      : poisub      & detected stars      & FITS Table \\
Static Sky  : stac        & astrometry          & FITS Header \\

\end{tabular}

\subsection{External Interfaces}

This subsection describes the interfaces between the IPP and other
Pan-STARRS systems and the external clients.  The interfaces are
illustrated in Figure~\ref{fig:overview}.  The details of the IPP
interfaces with other external subsystems are specified in the ICDs
between those subsystems.

\subsubsection{Camera}

Images are pulled from the Summit Pixel Server, part of the Camera
team's purview, via the Data Store mechanism.  The locations of the
files are specified by the Data Store (see the Data Store SDD,
PSDC-940-010).  IPP grabs these via {\tt http}.  This interface is
described in detail by the Camera-IPP ICD (PSDC-940-003).

\subsubsection{OTIS}

The IPP Scheduler receives metadata from OTIS as a collection of FITS
Tables.  The IPP sends quality assessment information for each FPA and
major frame as FITS tables.  These metadata tables are exchanged in
both directions using the Data Store mechanism (PSDC-940-010).  The
interface is described in detail by the OTIS-IPP ICD (PSDC-940-004).

\subsubsection{MOPS}

Data will be sent to MOPS from the IPP as part of the Phase 4
analysis.  The data to be transfered include:
\begin{itemize}
\item Image Metadata tables - to be transferred as FITS tables
\item Transient Detections - to be transferred as FITS tables
\end{itemize}
This interface uses the Data Store mechanism (PSDC-940-010), and is
described in detail by the IPP-MOPS ICD (PSDC-940-005).

\subsubsection{PSPS}

Data is sent to PSPS from the IPP on a regular basis.  The data to be
transfered include:
\begin{itemize}
\item Static Sky images - to be transferred as FITS images or
  FITS triangular image regions.
\item Postage Stamps - to be transferred as FITS images.
\item Metadata tables - to be transferred as FITS tables
\item Detections \& Object associations - to be transferred as FITS tables.
\end{itemize}
This interface uses the Data Store mechanism (PSDC-940-010), and is
described in detail by the IPP-PSPS ICD (PSDC-940-006).

\subsubsection{Other Preferred Client Science Pipelines}

These cannot be completely defined until the Clients are defined and
their requirements are specified.  The expectation is that the data
products will be the same as for the MOPS.  The data will be pushed
from the IPP to the Client Science Pipeline when they are available.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Computer Hardware}
\label{sec:Hardware}

\subsection{PS-1 Cluster Design}
  
The PS-1 IPP computer system is designed as a cluster of 'fat bricks':
computers with both processing power and large amounts of local disk
storage.  These computers are large rack-mount boxes with space for
10s of disks (24 and 36 disk cases are available) and a motherboard
with two CPUs and two Gig-E ethernet ports.  One set of machines is
specified for storage and processing of the individual OTAs up through
Phase 2 (the `OTA nodes'), another set of machines are specified for
storage of the Static Sky and processing of data from Phase 3 and
Phase 4 (the `Sky nodes').  Other machines will be necessary to
support the Metadata DB and the AP DB.  

The IPP PS-1 SRS (PSDC-430-005) specifies the processing throughput
requirements for the IPP.  Benchmark tests of the IPP processing
algorithms have been used to drive the design needed to achieve the
throughput requirements.  The details of this study are presented in
the IPP Computational Challenge (PSDC-400-006), summarized here.  The
analysis measures the processing time (excluding I/O) for both Phase 2
and Phase 4 on an Intel Pentium 4 processor, and expresses the
processing time in GHz-seconds, under the assumption that a machine
with the same architecture and twice the processor speed will perform
the same analysis in half the time.  This is probably a valid
assumption within a limited range on hardware using the same
architecture.  Independent tests show that 32-bit Pentium processors
perform somewhat slower (up to a factor of 2) than equivalently rated
64 bit Opteron processors.  This discrepancy makes the measured
numbers somewhat conservative, and compensates for the simplified
analysis performed.  The benchmarks show that the Phase 2 analysis
takes 12000 GHz-seconds for a complete major frame (4 FPAs) while the
Phase 4 analysis takes 7800 GHz-seconds for the same major frame.

The total data I/O required for each processing node, both locally to
disk and across the network to other machines, has also been measured.
These numbers in turn depend on whether the data is optimally stored
on the OTA nodes (raw images matched to their calibration images) or
if the data are randomized across the storage nodes.  There are also
differences in the analysis for the number of bits per pixel and the
number of calibration images used in the processing.  For PS-1, the
`minimal' data set is appropriate, resulting in a total Phase 2 I/O of
21 GBs per major frame and a total Phase 4 I/O of 36 GBs.  The
randomized numbers are used as a conservative estimate, under the
assumption the network, not local disk access, is the dominant I/O
bottleneck.

The analysis assumes each CPU (rated at 2.2 GHz) is associated with
one RAID array (maximum throughput 110 MB/sec) and one network
controller (maximum throughput 70 MB/s). In this case, given the CPU
load and I/O throughput above, Phase 2 will require a total of 190
seconds of I/O and 5500 seconds of processing distributed across the
cluster.  Likewise, the Phase 4 analysis will require a total of 330
sec of I/O and 3500 seconds of processing.  Given the 160 seconds
available per major frame, these numbers imply a total of 63
processors are needed to keep up with the processing and I/O load.

The other major driver on the IPP PS-1 cluster is the data storage
requirements.  It is necessary to store the raw images from the entire
AP Survey, the MOPS Verification Program (MVP) and the IPP
Verification Program (IVP), and to have storage enough to represent
the Static Sky by the end of the two year mission.  These storage
requirements as a function of time are shown in
Figure~\ref{fig:StorageProfile}.  Based on the PS-1 Design Reference
Mission (PSDC-230-001), by the end of the second year, the total
storage requirements for raw images and the Static Sky will be 850 TB,
along with and an additional 55 TB needed for the AP DB storage

To meet these requirements, the IPP cluster is designed to use fat
bricks which will be capable of holding 24 disks each.  The 5U / 24
disk rack mount computer cases are one of the highest density
solutions currently available.  A 4U / 36 disk box is also available
and will be considered.  The disk purchases will be staggered in three
waves.  Before PS-1 goes on the sky, the first 1/3 of the disks (600
disks total) will be purchased.  Since the lead time for disks is
fairly short, the purchase will be made only when other portions of
Pan-STARRS are clearly on a timeline to success.  After 9 months
(tentatively 2006 September), the next 1/3 of the disks will
purchased, and the remaining disks 9 months after that (tentatively
2007 June).  Using conservative estimates of the available disk sizes
at these purchase dates (400 GB, 600 GB, and 900 GB), and allocating 1
of 12 disks to the RAID and 10\% of the volume to file system and
binary Gigabyte overheads, the disk purchases outlined above result in
a total volume after the last purchase of 950 TB.  This meets the
requirements with 10\% spare excess.  The disk volume profile is also
shown in Figure~\ref{fig:StorageProfile} and shows that the disk space
will be available in the time it is required.

The total number of computers to be purchased is 80.  This provides
the 1800 disk slots and more than enough processors to meet the
processing requirements.  This also leaves 5 live spare machines.

There are two details which are not included in the analysis above:
compression and replication.  Compression of the older raw data will
reduce the volume requirements by a factor of roughly two.  However,
replication of the data across the network is necessary to ensure the
data against catastrophic failures on a single machine.  Replication
doubles the total data space needed.  These two factors will tend to
cancel each other, and are ignored in the calculations above.

The IPP PS-1 clusters will have the following allocations of computers
from this cluster:
\begin{itemize}
\item Phase 2 Nodes: 32
\item Phase 4 Nodes: 30
\item AP Database: 10
\item Metadata Database: 1
\item Image Server Database: 1
\item Controller /  Scheduler: 1
\end{itemize}
This distribution meets the projections for computational power for
each of these data systems, and leaves 5 computers as live spares for
redundancy.

\subsection{PS-1 Cluster Expected Reliability}

With 80 computers and 1920 disks, component failures are inevitable.
The cluster design and management must be chosen to minimize their
impact on operations and data integrity.

There are several factors which reduce the cluster's exposure to
hardware failures.  First, the use of RAID controllers and RAID-5
striping of the data will protect the data on a single RAID set
against the failure of a single disk in the array.  Second,
duplication of data across the cluster will protect against
catastrophic failures of the array (loss of two disks, loss of the
array controller card).  Finally, the flexibility of the distributed
computing plan minimizes the impact the loss of individual machines
has on operations by making changes in the data and processing
assignments on the cluster a trivial matter.

The components which are most likely to fail in the experience of our
team are, in order: hard drives, RAID controllers, ram, power
supplies, and other components.  The hard drive and RAID controller
failure rates are by far the dominant concerns as they potentially
affects the data integrity.

Most sources (REFS: UCSD article, Samsung White Paper) currently imply
hard disk failure rates (MTBF) in the range 400,000 hours and 500,000
hours.  These are used as an upper limit, with the more historically
conservative value of 100,000 hours used instead.  With 1920 disk,
this MTBF implies a failure of one disk every 2.2 days.  Since the
disks are in a RAID which reports the disk failures immediately and
drops the array into degraded mode, these failures will not have a
huge impact on the operations, and recovery time is only 10s of
minutes.  This failure rate implies that the maintenance plan must
include checks for hard disk failures on a daily basis, and should
make use of email notification and early warning information (ie,
SMART messages).  

A catastrophic failure for the array would require two of the 12 disks
to fail before the first failed disk is replaced.  Assuming that
maintenance is poor and it is possible for a disk to take 1 week to
be replaced, the probability of a catastrophe is 1.8\% each time the
first disk fails.  Combined with the disk failure rate, RAID
catastrophes are expected 6 times over the 2 year operation of PS-1.
These numbers can be used as a guideline for the level of support
needed to avoid these RAID failures.  Note that these 6 failures
should not cause loss of data since the data is duplicated across the
cluster, but they require over 1 day for recovery (as the entire array
must be replicated across the network).

A detailed IPP computer cluster commissioning and maintenance plan is
specified in the document `Pan-STARRS PS-1 IPP Cluster Support'
(PSDC-430-014).

\begin{figure}
\begin{center}
\resizebox{6in}{!}{\includegraphics[angle=-90]{pics/ps1_ipp_storage.ps}}
\caption{ \label{fig:StorageProfile} Storage Profile}
\end{center}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\clearpage
\appendix
\section{Image Server Database Table Contents}
\label{sec:ImageServerTableContents}

Tables~\ref{tab:ImageServerTables:SO} - \ref{tab:ImageServerTables:VOL} list
the basic contents of the Image Server database tables.  

\begin{table}[bh]
\begin{center}
\caption{Storage Object Table Contents\label{tab:ImageServerTables:SO}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype} & {\bf Description} \\
\hline
\code{so_id}      & integer        & internal storage object identifier \\
\code{ext_id}     & string         & external storage object identifier (file ID) \\
\code{comment}    & string         & user description of object \\
\code{epoch}      & date/time      & last date of access \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Instance Table Contents\label{tab:ImageServerTables:INT}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype} & {\bf Description} \\
\hline
\code{ins_id}     & integer        & internal instance identifier \\
\code{so_id}      & integer        & key to storage object table \\
\code{uri}        & string         & location in hardware collection \\
\code{sha1sum}    & string         & checksum information \\
\code{assigned_location} & boolean & is location user-specified? \\
\code{epoch}      & date/time      & last date of access \\
\code{atime}      & date/time      & last date of access \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Volume Table Contents\label{tab:ImageServerTables:VOL}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype} & {\bf Description} \\
\hline
\code{vol_id}     & integer        & internal volume identifier \\
\code{uri}        & string         & node name \\
\hline
\end{tabular}
\end{center}
\end{table}
\clearpage

\section{Metadata Database Table Contents}
\label{sec:MetadataTableContents}

Tables~\ref{tab:WeatherTable} -- \ref{tab:overlaps} list the basic contents of
each of the Metadata Database tables listed in Section~\ref{sec:Metadata}.

\begin{table}[bh]
\begin{center}
\caption{Weather Table: some sample weather points\label{tab:WeatherTable}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Time             & date/time       & The time the weather information was measured. \\
Temperature 01   & float           & The external temperature \\
Temperature 02   & float           & The temperature at top of the dome \\
Temperature 03   & float           & The temperature on the primary mirror \\
Humidity         & float           & The relative humidity. \\
Pressure         & float           & The (external) atmospheric pressure. \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{SkyProbe Transparency Table (sample entries)\label{tab:SkyprobeBVTable}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Time             & date/time       & The time the SkyProbe image was taken. \\
Filter           & string	   & Filter used for SkyProbe image. \\
Transparency     & float	   & The derived transparency. \\
Number of stars  & int		   & The number of stars used to measure the transparency. \\
Astrometry       & coords	   & The astrometry used on the SkyProbe image. \\
Exposure time    & float	   & The exposure time of the SkyProbe image. \\
Sky brightness   & float	   & The measured sky (surface) brightness, counts / second \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Skyprobe Line Absorption Table (sample entries)\label{tab:SkyprobeATable}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Time             & date/time       & The time the LRProbe observation was taken. \\
Disperser ID     & string          & ID of the dispersing element \\
Atm Component 1  & float	   & The strength of the 1st atmospheric component. \\
Atm Component 2  & float	   & The strength of the 2nd atmospheric component. \\
Atm Component 3  & float	   & The strength of the 3rd atmospheric component. \\
Disperser ID     & string          & ID of the dispersing element \\
Number of stars  & int 	           & Number of stars used to measure the absorptions. \\
Astrometry       & coords	   & The astrometry used on the LRProbe image. \\
Exposure time    & float	   & The exposure time of the LRProbe image. \\
Sky brightness   & float	   & The measured sky (surface) brightness, in physical units. \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Skyprobe Line Emission Table (sample entries)\label{tab:SkyprobeETable}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Time             & date/time       & The time the LRProbe observation was taken. \\
Disperser ID     & string          & ID of the dispersing element \\
Atm Component 1  & float	   & The strength of the 1st atmospheric component. \\
Atm Component 2  & float	   & The strength of the 2nd atmospheric component. \\
Atm Component 3  & float	   & The strength of the 3rd atmospheric component. \\
Continuum        & float	   & The strength of the continuum emission. \\
Disperser ID     & string          & ID of the dispersing element \\
Exposure time    & float	   & The exposure time of the LRProbe image. \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{DIMM Measurements Table\label{tab:DimmTable}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Time             & date/time       & The time the DIMM observation was taken. \\
$\sigma_x$       & float           & Raw dispersion in $x$. \\
$\sigma_y$       & float	   & Raw dispersion in $y$. \\
FWHM             & float	   & Dervied seeing full width at half maximum. \\
RA               & float	   & The coordinates of the measured star. \\
DEC              & float	   & The coordinates of the measured star. \\
Exposure time    & float           & The exposure time of the DIMM observation. \\
Telescope ID     & string          & source of the DIMM data \\
\hline		 
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Near IR Wide-field Camera Results Table\label{tab:NIR-Table}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Time           	 & date/time       & The time the NIR observation was taken. \\
Sky brightness 	 & float           & The sky (surface) brightness in the NIR observation. \\
Sky variance   	 & float	   & The variance in the sky (surface) brightness. \\
Astrometry     	 & coords          & The astrometry used on the NIR image. \\
FOV X            & float           & field width \\
FOV Y            & float           & field height \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Dome Status Table\label{tab:DomeStatusTable}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Time          	 & date/time       & The time for which the dome status is valid. \\
Azimuth       	 & float           & The azimuth of the dome. \\
Open status   	 & boolean	   & Whether the dome is open or not. \\
Lights status 	 & boolean	   & Whether lights are on in the dome or not. \\
Track status 	 & boolean	   & Whether dome is tracking telescope or not. \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Telescope Status\label{tab:TelescopeStatusTable}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Time         	 & date/time       & The time for which the telescope status is valid. \\
Guide status 	 & enum            & The status of the guiding. \\
Altitude     	 & float	   & The telescope altitude. \\
Azimuth      	 & float	   & The telescope azimuth. \\
RA  	     	 & float	   & The telescope Right Ascension (ICRS $\approx$ J2000). \\
Dec 	     	 & float	   & The telescope Declination (ICRS $\approx$ J2000).\\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Raw FPA Images\label{tab:RawFPAs}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
ID               & string          & FPA image ID \\
RA               & float	   & Coordinates of the boresight (i.e. telescope pointing). \\
DEC              & float	   & Coordinates of the boresight (i.e. telescope pointing). \\
Filter           & string	   & Filter used for the exposure. \\
Image Type       & enum            & image exposure type \\
Exposure time    & float	   & Exposure time for the image. \\
Airmass          & float	   & Airmass at which the image was taken. \\
ObsFrame ID      & int   	   & Observation frame identification number, ties FPAs into major frame \\
ObsGroup ID      & int   	   & Observation group identification number, ties FPAs into observing group \\
Observer         & string	   & The name of the observer, or the version of the telescope scheduler software. \\
Program          & string	   & The observing program being executed. \\
Nchips readout   & int   	   & Number of detector chips read out \\
Camera           & string   	   & Identification of camera source \\
Telescope        & string   	   & Telescope used for observation \\
Astrometry       & coords	   & The astrometry used for the FPA. \\
Chip Metadata    & string          & metadata resource file \\
Cell Metadata    & string          & metadata resource file \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Pending Science Chips\label{tab:PendingChips}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
FPA ID           & string          & FPA image ID \\
Chip ID          & string          & Chip identification number. \\
Proc Status      & enum            & Current Processing Status. \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Processed Science Chips\label{tab:ProcessedChips}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
FPA ID           & string          & FPA Image ID \\
Chip ID          & string          & Chip identification number. \\
Status           & enum            & Current Processing Status. \\
Residual Stats   & float           & quality statistics. \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Observation Group Information\label{tab:OBSGroup}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
ObsGroup ID      & string          & Identification number for the observation group. \\
Number of images & string          & Number of images in the observation group. \\
Type             & string          & Type of observation. \\
Status           & string          & Status of the observation group. \\
etc & \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Observation Frame Information\label{tab:OBSFrame}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
ObsFrame ID      & string          & Identification number for the observation frame. \\
Number of images & string          & Number of images in the observation group. \\
Type             & string          & Type of observation. \\
Status           & string          & Status of the observation group. \\
etc & \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Science Processing Stats\label{tab:PSStats}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Chip ID          & string	   & The chip identification number. \\
State            & string	   & The state of the processing. \\
ObsFrame ID      & string	   & The major frame the chip belongs to. \\
ObsGroup ID      & string	   & The observation group the chip belongs to. \\
P1 astrom        & string	   & The Phase 1 astrometry results file. \\
P2 astrom        & string	   & The Phase 2 astrometry results file. \\
P3 astrom        & string	   & The Phase 3 astrometry results file. \\
N guide stars    & string	   & Number of guide stars used for the exposure. \\
Astrometry stats & string	   & Summary statistics for astrometry (number of stars, $sigma_x$, $sigma_y$) \\
Astrom catalog   & string	   & The reference catalog that was used for the astrometry. \\
Bias method      & string	   & Method used to correct the bias. \\
Bias stats       & string	   & Summary statistics for bias \\
Flat-field image & string	   & The flat-field image that was applied. \\
Kernel data      &       	   & A description of the OT kernel. \\
Flat-field stats &       	   & Summary statistics for flat-field (sigma of sky). \\
Mask image       & string	   & The mask image that was applied. \\
Mask method      & string	   & The algorithm used to mask the bad pixels. \\
Fringe images    & string	   & The fringe model images that were used. \\
Fringe stats     &       	   & Summary statistics for fringes (fringe amplitude, sky sigma) \\
Object stats     &       	   & Summary statistics for object detection (number of objects, depth, other input parameters). \\
Photometry data  &       	   & photometry information: magnitude zero point and other corrections. \\
Photometry stats &       	   & Summary statistics for the photometry (number of stars, $sigma_m$) \\
Photom catalog   & string	   & The reference catalog that was used for the photometry. \\
PSF stats        &       	   & Summary statistics of the PSF. \\
Software ver     & string	   & Versions of each of the modules used in the processing. \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Chip / Sky overlaps\label{tab:overlaps}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Chip ID     	 & string	   & The identification number of the chip. \\
Sky Cell ID 	 & string	   & The identification number of the sky cell. \\
State       	 & string	   & Processing state of overlap \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Processed Sky-Cell stats\label{tab:ProcessedSky}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Input Chips        & string 	   & Identification numbers of the chips used to produce the sky cell. \\
PSF adjustments    & string 	   & Adjustments to the PSF. \\
CR rejection stats & string 	   & Statistics from the CR rejection (number of CRs, distribution, limiting flux). \\
Image comb params  & string 	   & Parameters used for the image combination. \\
Diff image params  & string 	   & Parameters used for the image differencing. \\
Average weight     & string 	   & The weight of the reference image \\
P4D object stats   & string 	   & Summary statistics of the object detection  \\
P4S object stats   & string 	   & Summary statistics of the object detection  \\
Software versions  & string 	   & Software versions of modules used in the sky cell processing. \\
Processing stats   & string 	   & Summary statistics of the processing (CPU time, etc). \\
\hline
\end{tabular}
\end{center}
\end{table}
\clearpage 

\section{AP Database Table Contents}
\label{sec:DVOTableContents}

\begin{table}[bh]
\begin{center}
\caption{Images\label{tab:images}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Image ID          & & \\ 
time/date	  & & \\
Exposure Time	  & & \\
Nstars		  & & \\
NX		  & & \\
NY		  & & \\
photcode	  & & \\
Mcal		  & & \\
Mcal error	  & & \\
Mcal chisq	  & & \\
Airmass           & & \\
Astrometry	  & & \\
PSF		  & & \\
flags		  & & \\
Camera		  & & \\
\hline		  
\end{tabular}	  
\end{center}	  
\end{table}	  
		  
\begin{table}[bh]
\begin{center}
\caption{Image Overlaps\label{tab:ImageOverlaps}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Image ID          & & \\
Region Table	  & & \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Objects\label{tab:Objects}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline 
ID                & & \\
$\alpha$	  & & \\
$\delta$	  & & \\
$\mu_{\alpha}$	  & & \\
$\mu_{\delta}$	  & & \\
$\sigma_{\alpha}$ & & \\
$\sigma_{\delta}$ & & \\
$\chi^2$ position & & \\
$N_{\rm det}$	  & & \\
$N_{\rm miss}$	  & & \\
flags		  & & \\
\hline		  
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Average Magnitudes\label{tab:AveMags}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
object ID         & & \\
$M_{\rm int}$	  & & \\
$M_{\rm ext}$	  & & \\
$\chi^2_{\rm mag}$& & \\
$\sigma_{\rm mag}$& & \\
photcode	  & & \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Solar System Objects\label{tab:SSObjs}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
SSO ID     	  & & \\
$N_{\rm det}$	  & & \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Matched Detections\label{tab:Detections}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
$\alpha$          & & \\
$\delta$	  & & \\
$\sigma_{\alpha}$ & & \\
$\sigma_{\delta}$ & & \\
$M_{\rm inst}$	  & & \\
$M_{\rm cal}$	  & & \\
$\sigma_{\rm mag}$& & \\
photcode	  & & \\
type		  & & \\
flags		  & & \\
time/date	  & & \\
airmass		  & & \\
$\sigma_{x}$	  & & \\
$\sigma_{y}$	  & & \\
$\theta$	  & & \\
object ID         & & \\
exptime		  & & \\
sky		  & & \\
$\sigma_{\rm sky}$& & \\
etc		  & & \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Orphaned Detections\label{tab:Orphans}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
$\alpha$          & & \\
$\delta$	  & & \\
$\sigma_{\alpha}$ & & \\
$\sigma_{\delta}$ & & \\
$M_{\rm inst}$	  & & \\
$M_{\rm cal}$	  & & \\
$\sigma_{\rm mag}$& & \\
photcode	  & & \\
type		  & & \\
flags		  & & \\
time/date	  & & \\
airmass		  & & \\
$\sigma_{x}$	  & & \\
$\sigma_{y}$	  & & \\
$\theta$	  & & \\
exptime		  & & \\
sky		  & & \\
$\sigma_{\rm sky}$& & \\
etc		  & & \\
\hline		  
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Non-detections\label{tab:NonDetects}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline  
object ID          & & \\
$N_{\rm non-det}$	   & & \\
last time/date 	   & & \\
last mag	   & & \\
faintest time/date & & \\
faintest mag	   & & \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Regions\label{tab:Regions}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
$\alpha_0$        & & \\
$\alpha_1$	  & & \\
$\delta_0$	  & & \\
$\delta_1$	  & & \\
Region ID	  & & \\
Parent ID	  & & \\
Nchildren	  & & \\
Images		  & & \\
Objects		  & & \\
Detections 	  & & \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Filters\label{tab:Filters}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Filter ID         & & \\
Filter name	  & & \\
Photcode	  & & \\
$\lambda_0$	  & & \\
$\delta_\lambda$  & & \\
$\epsilon$	  & & \\
transmission curve& & \\
time/date	  & & \\
\hline		  
\end{tabular}	  
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Photcodes\label{tab:Photcodes}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Photcode          & & \\
Telescope	  & & \\
Camera		  & & \\
Detector	  & & \\
Filter		  & & \\
Nominal ZP	  & & \\
airmass terms	  & & \\
color terms	  & & \\
Target		  & & \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Zero Point History\label{tab:Zpts}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline 
Photcode          & & \\
start Time/date	  & & \\
end Time/date	  & & \\
Zero Points	  & & \\
airmass		  & & \\
color		  & & \\
error		  & & \\
N measurements	  & & \\
N stars		  & & \\
photom ref set    & & \\
\hline
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Distortion History\label{tab:Distortions}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
Camera            & & \\
Telescope	  & & \\
distortion terms  & & \\
time/date	  & & \\
residuals / error & & \\
N stars		  & & \\
N images	  & & \\
astrom ref set	  & & \\
\hline		  
\end{tabular}
\end{center}
\end{table}

\begin{table}[bh]
\begin{center}
\caption{Database Hosts\label{tab:DVOHosts}}
\begin{tabular}{lll}
\hline
\hline
{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
\hline
machine name	  & & \\
machine ID	  & & \\
\hline
\end{tabular}
\end{center}
\end{table}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{PanTasks Example : Pcopy.pro}

Below is an example script for psched which demonstrates the
scheduling system. This parallel-copying script implements the
Pan-STARRS image copying system, which requests images from the summit
and copies them to the appropriate computer. The first task in the
script queries an external system for new image names with the
function new.images. In the case of Pan-STARRS, this would be a
request from OTIS, the observatory controlling system. The second task
initiates the individual image copies, with separate CCDs being copied
to separate computers. This script uses the concept of having specific
machines assigned to specific CCDs (as Pan-STARRS intends to
operate). The association is determined by calling the external
function chip.host, providing the identifier of the chip in
question. This returns an appropriate host. The copy.image function
copies the file and also sends a message to the summit system to
inform it that the image has been successfully copied.

\begin{verbatim}
verbose on
 queueinit newImages
 exec echo 0 > new.last
 exec cp -f raw.list new.list
 
 controller host add po01
 controller host add po02
 controller host add po03
 controller host add po04
 
 # identify the images ready for copy 
 # new entries are added to queue newImages
 # need to compare the new list with the ones already being processed
 task	       new.images
   command      new.images
   host         local
 
   periods      -poll 1
   periods      -exec 5
   periods      -timeout 5
 
   # success
   task.exit    0
     local i j Nstdout Nimages
     # compare output with new.image queue
     # keep only new entries
     queuesize stdout -var Nstdout
     for i 0 $Nstdout
       queuepop stdout -var line
       queuepush newImages -uniq -key 0 "$line"
     end
   end
 
   # locked list
   task.exit    1
     echo       "new.images: exec failure"
     $new.image.failure ++
   end
 
   # default exit status
   task.exit    default
     echo       "new.images: unknown exit status: $EXIT"
     $new.image.failure ++
   end
 
   # operation times out?
   task.exit    timeout
     echo       "new.images: timeout"
     $new.image.failure ++
   end
 end
 
 # copy new images, sending job to desired host
 task	       copy.images
   periods      -poll 0.2
   periods      -exec 1
   periods      -timeout 5
 
   task.exec
     queuesize  newImages -var N
     if ($N == 0) break
     # if ($network == 0) break
     # if ($filesystem == 1) break
     
     queuepop newImages -var line
     list tmp -split $line
     $filename   = $tmp:0
     $chip       = $tmp:1
     $state      = $tmp:2
     if ($state == new) 
       # copy this image
       queuepush newImages -replace -key 0 "$filename $chip run"
     else
       # ignore this image
       queuepush newImages -replace -key 0 "$filename $chip $state"
       break
     end
     # echo $chip
     $host = `chip.host $chip`
     # echo $host
     host $host
     # echo "starting copy for $filename on $host..."
     command copy.image $filename $chip
   end
 
   # can I have access to argc,argv?
 
   # success
   task.exit    0
     echo "done copy..."
     queuepop stdout -var line
     list tmp -split $line
     $filename   = $tmp:0
     $chip       = $tmp:1
     exec mark.image $filename
     queuepush newImages -replace -key 0 "$filename $chip copy"
   end
 
   # default exit status
   task.exit    default
     echo       "new.images: unknown exit status: $EXIT"
     $new.image.failure ++
   end
 
   # operation times out?
   task.exit    timeout
     echo       "new.images: timeout"
     $new.image.failure ++
   end
 end
 
\end{verbatim}

\input{glossary.tex}

\end{document}

------

* output data products
* DVO
* ipptools / ippMonitor?
* analysis stages, versions and iterations
