Index: /trunk/doc/design/specs.tex
===================================================================
--- /trunk/doc/design/specs.tex	(revision 423)
+++ /trunk/doc/design/specs.tex	(revision 424)
@@ -1,3 +1,3 @@
-%%% $Id: specs.tex,v 1.4 2004-04-13 02:18:48 eugene Exp $
+%%% $Id: specs.tex,v 1.5 2004-04-15 01:38:47 eugene Exp $
 \documentclass[panstarrs]{panstarrs}
 
@@ -5,13 +5,14 @@
 \title{Pan-STARRS Image Processing Pipeline}
 \subtitle{Software Requirements Specification}
-\author{Eugene Magnier, Paul A. Price}
 \shorttitle{IPP SRS}
+\author{Eugene Magnier, Paul A. Price, Josh Hoblitt}
 \group{Pan-STARRS Algorithm Group}
 \project{Pan-STARRS Image Processing Pipeline}
 \organization{Institute for Astronomy}
-\version{01.DR}
+\version{DR}
 \docnumber{PSDC-430-005}
 
-\setcounter{tocdepth}{4} % lowest level to be included in toc
+% allow paragraphs to be listed in TOC for now 
+\setcounter{tocdepth}{4}
 
 \begin{document}
@@ -19,11 +20,9 @@
 
 % -- Revision History --
-% provide explicit values for the old versions
-% use '\theversion' for the current version (set above)
 \RevisionsStart
 % version     Date         Description
-01          & 2003.01.01 & First draft \\
-\hline
-\theversion & 2003.03.10 & Second draft \\
+DR.01 & 2003.01.01 & First draft  \\ \hline
+DR.02 & 2003.03.10 & Second draft \\ \hline
+DR.03 & 2003.04.13 & Most paragraphs fleshed out \\ \hline
 \RevisionsEnd
 
@@ -42,7 +41,11 @@
 
 This document establishes the system requirements for the Pan-STARRS
-Image Processing Pipeline (IPP).
+Image Processing Pipeline (IPP) as applied to Pan-STARRS 1 (PS-1), the
+initial demonstration telescope to be constructed on Haleakala by Jan
+2006.
 
 \subsection{System Overview}
+
+\tbd{description of the Pan-STARRS System and PS-1.}
 
 \subsection{Document Overview}
@@ -64,8 +67,8 @@
 
 \DocumentsInternalSection
-PSCD-430-xxx  &   PS-1 Design Reference Mission \\ \hline
-PSCD-430-004  &   Pan-STARRS IPP C Code Conventions \\ \hline
-PSCD-430-006  &   Pan-STARRS IPP ADD \\ \hline
-PSCD-430-007  &   Pan-STARRS IPP PSLib SDR \\ \hline
+PSDC-430-xxx  &   PS-1 Design Reference Mission \\ \hline
+PSDC-430-004  &   Pan-STARRS IPP C Code Conventions \\ \hline
+PSDC-430-006  &   Pan-STARRS IPP ADD \\ \hline
+PSDC-430-007  &   Pan-STARRS IPP PSLib SDR \\ \hline
 \DocumentsExternalSection
 Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\
@@ -74,13 +77,13 @@
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\section{Requirements} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\section{Requirements} 
 
 \subsection{Required States and Modes}
 
-The IPP has NN states:  active mode, paused mode, interactive mode.
+The IPP has 3 states: active, paused, and interactive.
 
 \begin{itemize}
 
-\item {\bf active mode} In active mode, the IPP shall accept images
+\item {\bf active state} In active state, the IPP shall accept images
   and metadata from OATS and automatically perform the complete set of
   image processing tasks, including both calibration and science image
@@ -88,13 +91,15 @@
   client science pipelines \tbd{and IPP monitoring team}.
 
-\item {\bf paused mode}  In paused mode, the IPP shall refuse data and
+\item {\bf paused state}  In paused state, the IPP shall refuse data and
   metadata from OATS and data requests from the client science
   pipelines.
 
-\item {\bf interactive mode}  In interactive mode, the IPP shall
+\item {\bf interactive state}  In interactive state, the IPP shall
   accept data and metadata from OATS, but will not automatically
   process the data.  The IPP shall respond to user commands to
   initiate portions of the data analysis.
 \end{itemize}
+
+\tbd{what is a mode?}
 
 \subsection{System Capability Requirements}
@@ -237,41 +242,57 @@
 x86/Linux combination.
 
-
 \paragraph{Software Configuration}
 
-\tbd{Makefiles, directory structures, etc}
+\tbd{Makefiles, directory structures, UPS, etc}
 
 \subsubsection{Architectural Components}
 
-The IPP is organised into several different software elements, listed
-as follows:
+In order to achieve the required functionality, it is necessary to
+divide the IPP into a number of clearly-defined software elements,
+listed as follows:
 
 \begin{enumerate}
-\item Pixel Server
-\item Object Database
-\item Metadata Database
-\item Analysis Stages
-\item Controller
-\item Scheduler
+
+\item {\bf Pixel Server:} This component is a large data store for all
+ images used by the IPP, including the raw images from the telescope,
+ the master calibration images, the reference static-sky images, and
+ any temporary image data products produced by the IPP.  The Pixel
+ Server is required to meet all of the image storage needs identified
+ in the top-level requirements above.  The Pixel Server must accept
+ the incoming data and store it until it is no longer needed by other
+ portions of the IPP.
+
+\item {\bf Photometry \& Astrometry Database (PnA):} This component is
+  required to store and manipulate astronomical objects detected in
+  various images, as identified above, including individual
+  measurements of objects on the images, the summary information about
+  those objects, and reference object data.
+
+\item {\bf Metadata Database:} This component is required to store the
+  data which is not directly related to images or astronomical objects
+  as needed to perform the analysis specified above.
+
+\item {\bf Analysis Stages:}  Specific programs are required to
+  perform the processing steps listed above.  These can be divided
+  into well-defined analysis stages, each of which operates on a
+  particular unit of data, such as a single OTA image or a colletion
+  of astronomical objets. 
+
+\item {\bf Controller:} In order to perform the analysis stages
+  required by the IPP, it is necessary to use distributed computing
+  processes on a large number of computers.  The Controller is
+  required to manage the collection of analysis stages performed on
+  these machines.
+
+\item {\bf Scheduler:}  This component is a decision-making mechanism
+  required to guide the operation of the IPP: to evaluate the
+  currently available collection of data, to identify the necessary
+  analysis, and to assign the analysis tasks to the Controller.
+
 \end{enumerate}
 
 The relationship between these software elements is shown in
 Figure~\ref{overview}.  This figure also shows the interactions
-between the IPP and other Pan-STARRS systems.  The Pixel Server is a
-respository for all image pixel data, including the raw images from
-the telescope, the master calibration images, the reference static-sky
-images, and any temporary image data products produced by the IPP.
-The Object Database is a facility to store all of the information
-about astronomical objects, including individual measurements of
-objects on the images, the summary information about those objects,
-and reference object data.  The Metadata Database is a storage element
-for all data which is neither image pixel data or astronomical object
-data.  The analysis pipelines are all of the top-level analysis
-processes which are performed on images or collections of object data.
-The Controller is a system which manages the process of executing in
-parallel analysis pipelines on specific datasets on the cluster of
-computers.  The Scheduler is a system which evaluates the current
-state of data in the various repositories and makes decisions about
-which analysis processes should be executed at any given time.  
+between the IPP and other Pan-STARRS systems.  
 
 \begin{figure}
@@ -461,5 +482,5 @@
 \tbd{queries}
 
-\paragraph{Configuration Database -- a subset of the metadata database?}
+\subparagraph{Configuration Database -- a subset of the metadata database?}
 
 The IPP requires a Configuration Database to store and provide access to
@@ -668,34 +689,66 @@
 \paragraph{Overview}
 
-We now consider the collection of analysis tasks which are performed
-by the IPP.  Depending on the task, they may be performed on
-individual images, collections of images, or on derived data products.
-Because of the nature of the image data, many of the analysis tasks
-can be performed in parallel because, for example, the analysis of an
-OTA in one image does not depend on the results from another OTA.  We
-define the term 'analysis stage' to refer to the largest complete
-analysis task which may be performed on a single data item.  The
-analysis stages are divided into three categories, and further
-subdivided as follows:
+We now consider the collection of analysis tasks which must be
+performed by the IPP.  These tasks represent the core of the required
+IPP functionality; the architectural components discussed above can be
+viewed as primarily supporting infrastructure to enable the analysis
+tasks to be executed on the appropriate data and to store the results.
+
+Depending on the task, the basic data unit may be individual images,
+collections of images, or derived data products such as collection of
+detections of astronomical objects.  Because of the granularity of
+these data units, many of the analysis tasks can be performed in
+parallel because, for example, the intial analysis of an OTA in one
+image does not depend on the results from another OTA.  We define the
+term 'analysis stage' to refer to the largest complete analysis task
+which may be performed on a single data item.  The analysis stages are
+divided into three categories, and further subdivided as follows:
 
 \begin{enumerate}
- \item Science Image Analysis Stages
+ \item {\bf Science Image Analysis} is performed on the night-sky
+ science images to extract the science data from these images.  The
+ science image analysis is divided into 4 phases:
+
+ \begin{itemize}
+  \item {\bf Phase 1:} The image processing preparation phase, in
+  which a basic analysis of the complete FPA image is performed.
+
+  \item {\bf Phase 2:} The image reduction phase, in which the
+  individual detector images (OTAs) are processed as much as possible
+  without reference to other chips in the same FPA image or other
+  exposures.
+
+  \item {\bf Phase 3:} The exposure analysis phase, in which the
+  results of the multiple detectors are combined to improve the
+  calibrations for the complete FPA images. 
+
+  \item {\bf Phase 4:} The image combination phase, in which several
+  difference exposures of the same part of the sky are combined to
+  produce high-quality difference and summed images.
+ \end{itemize}
+
+ \item {\bf Calibration Image Analysis} is required to generate the
+ calibration images used in the science image analysis.  There are
+ three types of calibration images which are produced.
+
  \begin{enumerate}
-  \item Phase 1 : image processing preparation
-  \item Phase 2 : image reduction
-  \item Phase 3 : exposure analysis
-  \item Phase 4 : image combination
+  \item {\bf Calibration 1:} The basic master-detrend creation images,
+  which are constructed from a simple stack of multiple input
+  calibration images.  
+
+  \item {\bf Calibration 2:} Sky-model \& fringe-model images, which
+  are constructed by combining a collection of images which require
+  substantial processing before the combination.
+
+  \item {\bf Calibration 3:} Flat-field correction image, which is
+  constructed on the basis of photometry observations of objects from
+  certain science images.
+
  \end{enumerate}
- \item Calibration Image Analysis Stages
- \begin{enumerate}
-  \item Calibration 1 : basic master-detrend creation
-  \item Calibration 2 : Sky-model/fringe-mode generation
-  \item Calibration 3 : Flat-field correction image Creation
- \end{enumerate}
- \item Reference Catalog Analysis Stages
- \begin{enumerate}
-  \item Astrometry reference catalog generation
-  \item Photometry reference catalog generation
- \end{enumerate}
+
+ \item {\bf Reference Catalog Creation} is required by the IPP to
+ generate improved astrometric and photometric reference catalogs on
+ the basis of Pan-STARRS observations.
+
 \end{enumerate}
 
@@ -704,10 +757,8 @@
 the Controller.  The thick lines represent the flow of pixel data, the
 thin lines represent the flow of metadata and object data, and the
-grey lines represent the flow of commands.  \tbd{All subsystem
-interactions, except that between the scheduler and controller, are in
-the form of updates to and queries from the databases}.  The hatched
-systems represent external PanSTARRS systems (OATS, the Sky Server,
-the SAIC Object Database, the Moving/Transient Object Pipeline, and
-other Client Science Pipelines.  
+grey lines represent the flow of commands.  The hatched systems
+represent external PanSTARRS systems (OATS, the Sky Server, the SAIC
+Object Database, the Moving/Transient Object Pipeline, and other
+Client Science Pipelines.
 
 The individual analysis stages can be accessed as a UNIX command-line
@@ -717,34 +768,9 @@
 \tbd{Python}.  
 
-\subparagraph{Science Image Pipelines}
-
-The IPP science image pipelines perform analyses on the night-sky
-science images to extract the science data from these images.  These
-consist of: Phase 0, the night preparation stage; Phase 1, the image
-processing preparation stage; Phase 2, the image reduction stage;
-Phase 3, the exposure analysis stage; and Phase 4, the image
-combination stage.  These pipelines must process the images in a
-timely manner so that the incoming data stream will not overload the
-IPS.  The decision to execute a specific pipeline for a specific
+The decision to execute a specific analysis stage for a specific
 dataset is made by the Scheduler, which sends the infomation to the
-Controller.  The Controller executes the pipeline for the data on an
-appropriate machine and monitors the success or failure of the job.
-
-\subparagraph{Calibration Image Pipelines}
-
-The IPP Calibration Image Pipelines perform the tasks needed to
-generate high-quality calibration images from the input image
-dataset.  These operations may be performed on whatever timescales are
-appropriate and necessary to maintain the quality and relevance of the
-calibration images.  There are four distinct types of calibration
-image pipelines:  the basic detrend creation pipeline, the photometric
-correction image creation pipeline, the fringe pattern generation
-pipeline, and the sky foreground pattern generation pipeline.
-
-\subparagraph{Reference Catalog Pipelines}
-
-The IPP reference catalog pipelines use the data in the IPP Internal
-Database and the IPP Object Database to determined improved
-astrometric and photometric calibration references.
+Controller.  The Controller executes the analysis stage for the data
+on an appropriate machine and monitors the success or failure of the
+job.
 
 \begin{figure}
@@ -755,4 +781,47 @@
 \end{figure}
 
+\paragraph{Science Image Analysis}
+
+The Science Image analysis stages together represent the basic data
+analysis required by the IPP.  These analysis stages must process the
+images in a timely manner so that the incoming data stream will not
+overload the Pixel Server.  The required processing time is derived
+from the rate at which science images are obtained by PS-1.  At a
+minimum, the Science Image Analysis must keep up with the average
+image rate over the course of 1 day.  \tbd{The Science image analysis
+is required to process images at the maximum science image rate from
+PS-1 of 1 image every 30 seconds -- does this fall out of the science
+requirements?}  \tbd{In order to give time for uncertainties in the
+Pan-STARRS system as a whole, the Science Image Analysis must be able
+to process all images from a night within 12 hours.}
+
+\tbd{number of images per night, data volume per image, output
+products}
+
+The science image analysis which must be performed by the IPP consists
+of:
+
+\begin{itemize} 
+\item detrending the images to remove the instrumental signature
+
+\item astrometric and photometric calibration of the individual images
+
+\item merging a collection of several images of the same portion of
+the sky obtained over a short period of time (to remove image defects
+and gaps)
+
+\item subtracting the appropriate reference static-sky image
+
+\item cleaning the image of any transients
+
+\item adding the cleaned image to the static sky
+
+\item object detection of images at specific stages
+\end{itemize}
+
+These analysis steps can be grouped into four phases, each of which
+deals with a single data unit.  We identify and discuss the
+requirements of the four phases below.
+
 \paragraph{Phase 1 : image processing preparation}
 
@@ -760,9 +829,10 @@
 calculate basic astrometric \tbd{and photometric} data needed by the
 later stages.  Phase 1 must use the static (pre-determined) telescope
-distortion model, combined with the guide star pixel and celestial
-coordinates, to determine the correct telescope bore-site, field
-rotation and magnification.  The astrometric accurate required from
-this analysis stage is 2 arcsec across the field, sufficient to match
-the vast majority of reference stars with their detections.  
+distortion model and table of nominal OTA positions and rotations,
+combined with the guide star pixel and celestial coordinates, to
+determine the correct telescope bore-site, field rotation and
+magnification.  The astrometric accurate required from this analysis
+stage is \tbd{2 arcsec} across the field, sufficient to match the vast
+majority of reference stars with their detections.
 
 In some circumstances, science images may have no guide stars.  This
@@ -773,12 +843,12 @@
 are significantly above the background level.  The threshold levels
 for this object detection stage must be configurable.  The object
-extraction must be performed in less than 3 seconds.  
+extraction must be performed in less than \tbd{3 seconds}.
 
 In order for astrometry of an image to succeed, it is necessary that
 approximate image coordinates be known.  The Phase 1 analysis must be
-able to succeed despite initial coordinate errors as large as 5 times
-the field width.  However, the search process must attempt the near
-matches first in the assumption that the given coordinates are
-accurate. 
+able to succeed despite initial coordinate errors as large as \tbd{5
+times} the field width.  However, the search process must attempt the
+near matches first in the assumption that the given coordinates are
+accurate.
 
 A table of the overlaps between the science image to be processed and
@@ -786,7 +856,8 @@
 guide the processing of the static sky in Phase 4.  The overlaps must
 be generously calculated so that small errors in astrometry at Phase 1
-will not cause any valid static sky / science image pairs to be
-missed.  It is acceptable for a small number of invalid overlaps to be
-identified as these will be excluded in Phase 4.
+will not cause any valid static sky / science image pairs to be missed
+because of the astrometric error at this phase.  It is acceptable for
+a small number of invalid overlaps to be identified as these will be
+excluded in Phase 4.
 
 It is not unusual that an image be obtained with invalid coordinates
@@ -801,10 +872,13 @@
 \paragraph{Phase 2 : image reduction}
 
-Phase~2 processing within the Pan-STARRS image processing pipeline is
-the detrend stage, where the images from the detector are processed to
-remove instrumental signatures.  In addition, basic object detection
-is performed along with improved astrometric and photometric
-calibration.  The following operations need to occur within Phase~2
-processing:
+The Phase~2 analysis is the detrend stage, in which the images from
+the detector are processed to remove instrumental signatures.  In
+addition, basic object detection is performed along with improved
+astrometric and photometric calibration.  \tbd{what component selects
+the appropriate calibration data?  is it the phase~2 program, the
+individual modules, or the scheduler above it?}  In each step of the
+analysis process, an image mask and noise map must be carried and
+updated when appropriate.  The following operations need to occur
+within Phase~2 processing:
 
 \begin{enumerate}
@@ -823,9 +897,13 @@
 \subparagraph{Convolve detrend images with the OT kernel}
 
-Detrend images must be convolved by the OT kernel, so that
-they accurately represent the detrend images appropriate for
-the object images, which have been shifted using OT.  The detrend
-images which must be convolved include: the flat-field and the
-high-spatial-frequency fringe images. 
+Detrend images must be convolved by the OT kernel, so that they
+accurately represent the detrend images appropriate for the object
+images, which have been shifted using OT.  The detrend images which
+must be convolved include: the flat-field and the
+high-spatial-frequency fringe images. \tbd{Must this be a formal
+convolution with the analytical OT kernel, or can it be a convolution
+with a decomposed kernel?} The appropriate kernel for each cell of an
+OTA must be determined from the guide star history.  \tbd{what is the
+source of the OT kernel?  pixel server?}
 
 \subparagraph{Flag bad and saturated pixels}
@@ -867,7 +945,10 @@
 \subparagraph{Trim object image}
 
-The overscan must be trimmed from the object image, along with
-those pixels near the edges that have been compromised due to OT
-operation.
+The image must be trimmed to remove the non-imaging pixels, such as
+the overscan and any pre-scan pixels, along with those pixels near the
+edges that have been compromised due to OT operation.  The definition
+of the imaging area of the detector must optionally be determined from
+the camera configuration data or from the metadata associated with the
+image.
 
 \subparagraph{Correct for non-linearity}
@@ -883,38 +964,74 @@
 must be corrected for sensitivity variations as a function of
 position, dividing by a flat-field image.  The flat-field images must
-be appropriately normalized (see section \ref{mkcal}.  \tbd{what
-component selects the appropriate flat-field image?  scheduler or
-flat-field module?}  The flat-fielded image must have a consistent
-photometric zero-point across the chip, and across the full FPA, to
-within 0.2\%.
-
-\subparagraph{Sky subtraction}
+be appropriately normalized (see section \ref{mkcal}).  The
+flat-fielded image must have a consistent photometric zero-point
+across the chip, and across the full FPA, to within 0.2\%.  
+
+\subparagraph{Sky \& Fringe subtraction}
 
 The flux contribution of the sky (from both continuum emission and the
 line emission that causes fringing) must be subtracted from the
-flat-fielded object image. 
-
-\subparagraph{Identify CRs}
-
-CRs should be identified, if possible on the basis of their morphology
-in the flat-fielded object image (from a single focal plane), and
-masked.  The mask must be grown by an additional pixel.
+flat-fielded object image.  The subtraction must remove background
+(technically, foreground) variations which are not celestial but
+generated in the atmosphere or by more localized scattering.  This
+background subtraction does not address the artefacts generated by
+bright stars: bleeding columns, ghosts, or other localized reflection
+effects.  The background subtraction must remove the variations with
+an accuracy such that the residual variations do not introduce on
+average more than \tbd{0.2\%} photometric scatter or more than
+\tbd{1\%} extremely deviant outlier stars (stars for which the
+photometry is in error by more than 3\%.  \tbd{what is the requirement
+on galaxy photometry? morphology determinations?}  \tbd{What is
+allowed power-spectrum of background variations?}
+
+\subparagraph{Identify 'cosmic rays'}
+
+Charged particles in the detector frequently cause features which do
+not have the morphology of astronomical objects.  In a well-sampled
+image, these may be easily identified by the sharpness of the image.
+In a near critically-sampled image, these 'cosmic rays' may be
+indistinguishable from stellar objects.  The IPP must have the
+capability of making the morphological identification of cosmic rays
+if the imaging data is suitable.  The identified cosmic rays must be
+masked with a configurable growth factor (additional pixels beyond the
+detected pixels in the feature).  \tbd{The determination if the image
+can be treated with morphological cosmic ray rejection must be made by
+Phase~2.}
 
 \subparagraph{Find objects in the image}
 
 Objects on the flat-fielded object image must be found, and general
-parameters, including the centre, magnitude and shape measured.
+parameters, including the object centroid, instrumental magnitude,
+local background level, and basic shape parameters ($\sigma_{\rm min},
+\sigma_{maj}$) measured.  The detection threshold must be
+configurable, and be a function of the average background flux or the
+image noise map.  Minimal object classification must be performed to
+distinguish objects which are consistent with a single PSF, objects
+which are inconsistent, and objects which are saturated.  The
+resulting collection of detected objects must be saved along with the
+relevant image metadata (\ie, filter, exposure time, etc).
 
 \subparagraph{astrometry}
 
-\tbd{per-OTA astrometry to improve per-OTA parameters}
+Objects detected in Phase~2 must be matched with known astrometric
+reference objects, using reference object coordinates which have been
+adjusted for proper motion.  The matched objects must be used to
+improve the astrometric solutions for the individual OTAs.  At this
+stage, a user-defined collection of OTA astrometry parameters must be
+fitted on the basis of the matched stars.  The Cell astrometric
+parameters must not be allowed to flow at this stage.  The fit must be
+robust, rejecting outlier matches, either stars with poorly determined
+proper motion or spurious matches.  The resulting astrometric solution
+must be consistent across the OTA field to within \tbd{0.2 arcsec}.  
 
 \subparagraph{Postage Stamps}
 
-Objects on the flat-fielded object image falling within a specified
-magnitude range should have subimages saved for the purpose of more
-accurate photometry and astrometry.
-
-\paragraph{Phase 3}
+The IPP must have the capability of extracting regions surrounding a
+specified subset of objects from the flattened images.  These postage
+stamp images must be saved for additional use by client science
+pipelines.  The identification of these objects must be on the basis
+of a set of rules applied to the object magnitude and position.
+
+\paragraph{Phase 3 : exposure analysis}
 
 The Phase 3 analysis stage works with the results from a complete FPA
@@ -927,8 +1044,8 @@
 significant \tbd{level TBD}, the zero-point variations must be modeled
 with an up-to 3rd order chebychev polynomial correction.  The complete
-FPA image must be categoriezed as photometric on the basis of the
-zero-point consistency, the transparency compared with recent
-long-term measurements in the filter, and with the external indicators
-of photometricity.
+FPA image must be categorized as photometric or not \tbd{numerical
+scale?} on the basis of the zero-point consistency, the transparency
+compared with recent long-term measurements in the filter, and the
+external indicators of photometricity.
 
 Phase 3 must use the objects detected in Phase 2, matched with an
@@ -936,32 +1053,82 @@
 astrometric solutions.  The distortion model appropriate to this image
 must be determined.  The resulting astrometric accuracy must be
-\tbd{50 mas? 10 mas?}
-
-\paragraph{Phase 4 Concept}
-
-Phase 4 processing within the Pan-STARRS image processing pipeline is
-the final stage of processing.  It operates on each sky cell that has
-overlapping imaging data from the exposure(s) being processed, and
-produces the main output image data products of the stage --- the
-difference images and a deep static sky image --- along with the
-associated catalogues of static and variable sources.
-
-Here we give the specifications for the implementation of Phase 4
-processing.
-
-
-\subparagraph{Functionality}
-
-Phase 4 must consist of the following elements:
-\begin{enumerate}
-\item Combine images --- the images from each telescope are to be
-combined in order to obtain a deep image free from artifacts (e.g.\
-cosmic rays, low-altitude streaks);
-\item Identify variable sources --- the combined image is to be
-compared with the static sky image and variable sources identified; and
-\item Add to static sky --- the combined image is to be added to the
-static sky so that an incrementally-deeper static sky image may be
-made.
-\end{enumerate}
+limited by the astrometric reference catalog \tbd{30 mas for USNO?}
+
+\paragraph{Phase 4 : image combination}
+
+Phase 4 is the image combination stage, in which multiple images of
+the same portion of the sky are merged and confronted with the static
+sky image.  Phase 4 operates on the smallest data unit of the static
+sky, the sky cell, along with the associated pixels from a collection
+of image which have been processed through phases 1 - 3.  For each sky
+cell, the corresponding pixels are extracted from the exposures being
+processed and mapped to the projection of the sky cell. The pixels
+from the multiple input processed images are combined into a single,
+cleaned image.  This image is then confronted with the static sky cell
+data to produce a difference image.  Residual objects in the
+difference image, above a threshold are detected and excised from the
+original cleaned image.  The remaining pixels are added to the
+existing static sky image.  Object detection must be performed of the
+difference and cleaned images.  \tbd{when is static sky object
+detection \& classification performed?}  Phase 4 naturally divides
+into several stages, each of which are discussed in detail below.
+
+\subparagraph{Extract image pixels}
+
+For the given sky cell, the corresponding set of image pixels must be
+determined and extracted from the input images.  This process must use
+the astrometric information for each OTA and Cell to determine the
+overlaps.  It must not miss any pixels, and it must read no more than
+20\% more pixels than necessary from the input images.
+
+\subparagraph{Transform pixel coordinates}
+
+Pixels which have been extracted from the input images must be mapped
+to the corresponding pixels in the sky image.  The tranformation must
+be based on the measured astrometric solution for the input images
+relative to the reference catalog used to generate the static sky
+image.  This warping must use a locally linear astrometric solution to
+minimize computational effort. The output image must maintain be
+photometric consistent with the input image to within 0.2\%.
+\tbd{interpolation method?}
+
+\subparagraph{PSF matching}
+
+The multiple input images must have their PSF mutually matched to
+allow for proper image subtraction.
+
+\subparagraph{Flux matching}
+
+The multiple input images must have their object fluxes mutually
+matched by intercomparison of the stars measured in Phase 2 in order
+to properly combine them photometrically. 
+
+\subparagraph{Image outlier pixel rejection}
+
+Pixels from the group of images which are inconsistent with the
+ensemble of pixel values must be identified and flagged.  The
+resulting collection of pixels must be used to construct a single
+output image, cleaned of the outliers.  This outlier rejection must be
+performed optionally since moving objects will be rejected in images
+obtained over a wide range of times.
+
+\subparagraph{Image Subtraction}
+
+The static sky image must be subtracted from the merged, cleaned
+image.  All objects in the difference image must be detected and the
+pixels flagged in the input image.  Object detection at this stage is
+the same as that used for Phase 2.  
+
+\subparagraph{Cleaned Input Image}
+
+The flagged pixels must be excluded from the input images and a new,
+cleaned image constructed.  This image must have object detection
+applied to it.  \tbd{parameters}
+
+\subparagraph{Update static sky}
+
+The final, cleaned input image must be added to the static sky so that
+an incrementally-deeper static sky image may be made.
+\tbd{parameters, weight map}
 
 \subparagraph{Products}
@@ -977,5 +1144,4 @@
 \end{enumerate}
 
-
 \subparagraph{Timing}
 
@@ -1004,5 +1170,4 @@
 \end{itemize}
 
-
 \subparagraph{Robustness}
 
@@ -1012,5 +1177,16 @@
 to an error upstream in the processing).
 
-\subsubsection{Calibration Stage 1}
+\paragraph{Calibration Stages}
+
+The Calibration analysis stages may be performed on whatever
+timescales are appropriate and necessary to maintain the quality and
+relevance of the calibration images.  We distinguish two major classes
+of calibration images which require significantly different techniques
+for their construction.  We list the specific calibration images which
+must be constructed in the calibration analysis stages. The
+requirements for each of these stages are discussed in more detail
+below.
+
+\paragraph{Basic Calibration Stages}
 
 The IPP must generate basic calibration images using the raw
@@ -1026,25 +1202,231 @@
 are consistent and valid. 
 
-\paragraph{bias images}
-
-\paragraph{dark images}
-
-\paragraph{flat-field images}
-
-\subsubsection{Calibration Stage 2}
-
-\paragraph{mask images}
-
-\paragraph{fringe frames}
-
-\paragraph{low-k sky models}
-
-\subsubsection{Calibration Stage 3}
-
-Flat-field correction frame
-
-\subsubsection{Astrometry Reference Creation}
-
-\subsubsection{Photometry Reference Creation}
+\subparagraph{bias images}
+
+Bias images may be needed to correct for structure in the bias.  The
+IPP must have the capability of constructing a master bias image from
+a stack of raw bias frames.  The input bias images, representing
+offsets from the overscan level, must have the overscan removed,
+including 1D structure if needed.  The bias construction must
+incorporate outlier image and outlier pixel rejection.  The statistic
+used to determine pixel values must optionally be derived from the
+sample mean, median, and mode, robust mean, median, and mode, and the
+clipped mean and median.  Residual images, in which the master bias is
+applied to the input images must be constructed and their statistics
+used to exclude any significant outlier input images.
+
+\subparagraph{dark images}
+
+Dark images may be needed to correct for structure in the dark
+current.  The IPP must have the capability of constructing a master
+dark image from a stack of raw dark frames.  The input dark images
+must first be corrected for the bias using whatever method is
+appropriate for the science images.  The master dark frame must be
+specified for a particular exposure time.  As such, the input dark
+frames must have a limited range of exposure times.  The dark frame
+construction must incorporate outlier image and outlier pixel
+rejection.  The statistic used to determine pixel values must
+optionally be derived from the sample mean, median, and mode, robust
+mean, median, and mode, and the clipped mean and median.  Residual
+images, in which the master dark image is applied to the input images
+must be constructed and their statistics used to exclude any
+significant outlier input images.  \tbd{The dark frames must be
+examined to determine the non-linearity of the measured dark current
+-- by what component?}.
+
+\subparagraph{flat-field images}
+
+Master flat-field images must be constructed from a collection of
+input flat-field images.  An appropriate set of input images must be
+selected on the basis of their flux levels, time of observations, and
+the observing conditions.  The input flat-field images must be
+processed (bias and dark corrected if needed) and the resulting images
+stacked.  The master flat-field construction must incorporate image
+and pixel outlier rejection.  The statistic used to determine pixel
+values must optionally be derived from the sample mean, median, and
+mode, robust mean, median, and mode, and the clipped mean and median.
+Residual images, in which the master flat-field image is applied to
+the input images must be constructed and their statistics used to
+exclude any significant outlier input images.  
+
+\paragraph{Other Calibration Stages}
+
+\subparagraph{mask images}
+
+Initial bad-pixel mask images must be generated on the basis of
+comparison between raw flat-field images and a cleaned, stacked
+master.  The mask creation analysis stage must accept a collection of
+flat-field images and identify pixels which are repeatedly
+inconsistent from image to image.  If too many pixels are
+inconsistent, an error should be raised. 
+
+\subparagraph{fringe frames}
+
+Fringe-correction frames must be generated to remove the fringe
+pattern caused by thin-film interference in the top layers of CCDs,
+particularly in the redder passbands.  Fringe correction frames must
+be constructed on the basis of observations of the night-sky in the
+appropriate filters.  The images must first be flattened to remove the
+pixel-to-pixel sensitivity variations of the detector.  The
+combination of multiple input fringe frames may not be simply stacked
+since the amplitude of the fringe pattern varies independently of
+other variations in the image.  The amplitude of the fringe frames
+must be measured and the images scaled to normalize the fringe
+amplitude to the range -1 to +1 before combining with one of the
+standard combination statistics (mean, median, mode, etc).
+
+\subparagraph{low-k sky models}
+
+Large-scale background structure in images which is not caused by
+thin-film interference must also be detected and corrected.  Models of
+this background structure may be the necessary input to the correction
+proceedure.  The IPP must have the capability of generating image
+models of the large-scale structure patterns observed with the
+telescope.  \tbd{discuss principal components, SVD?}
+
+\subparagraph{Flat-field correction frame}
+
+Flat-field images, whether constructed from the dome, twilight, or
+night-sky images, rarely will perfectly correct the detector response
+in a consistent fashion across the full field of the camera.  The IPP
+must have the capability of generating flat-field photometric
+correction frames on the basis of the measured photometry of objects
+which are placed at a variety of locations on the detector in a
+sequence of images. 
+
+\paragraph{Reference Catalog Creation}
+
+For PS-1, one of the primary goals is the creation of photometric and astrometric
+reference catalogs for the general community and for the future
+Pan-STARRS requirements.  The generation of these catalogs is
+inherently a research project, and will require human control and
+intervention.  The IPP will be required to provide the data access,
+manipulation and visualization tools needed to construct these
+reference catalogs and to assess their quality.  In this section, we
+list the requirements of the tools needed for this effort.
+
+\paragraph{Astrometry Reference Creation}
+
+The existing astrometric reference catalogs are known to have
+limitations at the level of \tbd{NN} milli-arcsec.  The internal
+accuracy of the Pan-STARRS dataset can potentially be much higher than
+the external reference catalogs.  The IPP must have the capability of
+generating an astrometric reference on the basis of the observations
+obtained by the PnA survey.  The IPP must provide the analysis tools
+needed to generate the master astometric reference catalog.  Much of
+the required functionality is covered by the PnA Database.
+
+The necessary ingredients for the construction of the PS-1 Astrometric
+Reference Catalog are: the observed coordinates of stars and the
+existing astrometric reference catalogs.  A variety of reference
+catalogs will be required, including:
+\begin{itemize}
+\item Hipparcos
+\item Tycho2
+\item UCAC
+\item YBx
+\item USNO-Bx
+\item 2MASS
+\end{itemize}
+These catalog must be available and accessible to the PnA Database.
+It is necessary to have the tools to convert the reference catalog
+object coordinates to all of the possible coordinate frame of
+relevance in the telescope and camera system, including:
+\begin{itemize}
+\item Catalog to mean positions
+\item Mean to apparent positions
+\item Apparent positions + pointing to focal plane coordinates
+\item focal plane to specific detector (OTA)
+\item specific detector to detector cell
+\end{itemize}
+
+In addition to the reference catalogs, it will be necessary to
+determine and have available for the analysis system a variety of
+approximate calibration data, including the telescope and camera
+optical distortion models and the variation of the image PSF across
+the camera field, as a function of color.
+
+The final ingredient in the astrometry solution is the observation of
+stars with the PS-1 telescope.  The object detections are produced by
+several of the analysis stages discussed in the Science Analysis
+section.  The likely measurement of relevance to the astrometric
+reference catalog is the object extraction for the individual,
+detrended images (section~\ref{foo}).  \tbd{is it necessary to have
+  multiple centroiding methods available?}.  The detected objects must
+be matched against the reference catalogs, and it must be possible to
+determine fit coefficients as a function of simply position, or with
+combinations of magnitude or color.  The fitting method must include
+robust outlier rejection.  It is also necessary to have information
+about the objects which are detected in the catalog, but not the
+science image or vice-versa, as well as an assessment of the
+centroiding errors for each object.  It must be possible to plot the
+fit residuals against a wide variety of parameters, including the
+object positions, magnitudes, colors, etc, and to make subset
+selections of the objects on the basis of these parameters.  .  
+
+An alternative measurement of the stellar positions is derived from
+the guide stars, which are much brighter than the typical saturated
+stars.  It must be possible to compare the coordinates of the guide
+stars with the coordinates of the other stars in the image.  It must
+also be possible to perform the various fitting steps for the guide
+stars rather than for the normal image data.
+
+\paragraph{Photometry Reference Creation}
+
+The IPP must provide the analysis tools needed to generate a master
+photometric reference catalog.  The tools needed for generation of the
+photometric reference catalogs are similar in essence to those used
+for the astrometric reference catalog.  It is necessary to confront
+the observed objects against the existing reference catalogs to
+determine the necessary calibrations.  Again, much of the required
+functionality is covered by the PnA Database.  
+
+The necessary ingredients for the construction of the PS-1 Photometric
+Reference Catalog are: the observed magnitudes of stars and the
+existing photometric reference catalogs.  A variety of reference
+catalogs will be required, including:
+\begin{itemize}
+\item SDSS
+\item CFHT-LS standards
+\item Landolt
+\item etc
+\end{itemize}
+These catalog must be available and accessible to the PnA Database.
+
+The final ingredient in the photometry solution is the observation of
+stars with the PS-1 telescope.  The object detections are produced by
+several of the analysis stages discussed in the Science Analysis
+section.  The likely measurement of relevance to the photometric
+reference catalog is the object extraction for the individual,
+detrended images (section~\ref{foo}).  It is necessary to have the
+tools to convert between different photometric systems, including:
+\begin{itemize}
+\item instrumental to nominal detector magnitude
+\item nominal detector magnitude to average filter system
+\item average filter system to reference photometry system
+\end{itemize}
+These transformations are based on a set of measured coefficients for
+the color and airmass dependency of the measurement.  In addition to
+these types of transformations, it is necessary to have the ability to
+measure and apply relative photometry corrections.  
+
+The detected objects must be matched against the reference catalogs,
+and it must be possible to determine fit coefficients as a function of
+airmass, magnitude, color and detector coordinates, or with
+combinations of the above.  The fitting method must include robust
+outlier rejection.  It is also necessary to perform exclusions on the
+basis of object locations, instrumental magnitudes, observed and
+reference errors, and in particular time of the observations. It must
+be possible to plot the fit residuals against a wide variety of
+parameters, including the object positions, magnitudes, colors, etc,
+and to make subset selections of the objects on the basis of these
+parameters.  .
+
+An alternative measurement of the stellar positions is derived from
+the guide stars, which are much brighter than the typical saturated
+stars.  It must be possible to relate the magnitudes of the guide
+stars with the magnitudes of the other stars in the image.  It must
+also be possible to perform the above fitting steps for the guide
+stars rather than for the normal image data.
 
 \subsubsection{Modules}
@@ -1099,9 +1481,12 @@
 
 \begin{itemize}
+\item Hipparcos
+\item Tycho2
+\item HST-GSC
 \item USNO-A
-\item USNO-B
-\item HST-GSC
-\item Tycho
+\item UCAC
 \item 2Mass
+\item USNO-Bx
+\item YBx
 \end{itemize}