Context Navigation

← Previous Changeset
Next Changeset →

Changeset 2176

Timestamp:

Oct 19, 2004, 10:17:36 PM (22 years ago)

Author:

eugene

Message:

continuing major revisions

File:

: 1 edited

trunk/doc/design/ippSDRS.tex (modified) (23 diffs)

Legend:

: Unmodified
: Added
: Removed

trunk/doc/design/ippSDRS.tex

-              r2172
+              r2176
 %%% $Id: ippSDRS.tex,v 1.8 2004-10-19 03:28:04 eugene Exp $
+%%% $Id: ippSDRS.tex,v 1.9 2004-10-20 08:17:36 eugene Exp $
 \documentclass[panstarrs]{panstarrs}
 …
 Database) are also shown.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \section{System Design : Architectural Components}
 …
 \begin{itemize}
+\item AP Database database tables
+\item AP Database database engine
 \item AP Database servers
 \item AP Database client APIs
-\item AP Database storage hardware
-\item AP Database database engine
-\item AP Database database tables
 \end{itemize}
 …
 contents of these tables are outlined in
 Appendix~\ref{APDBTableContents}.  Below, we discuss how these tables
+are used by the AP Database software.  Three of the tables are not
+simple tables but instead are divided into many subtables, each of
+which represents a portion of the sky.  These subtables may also be
+distributed across different computers to distribute the processing
+load.
+The table of Images lists all of the images which provided the data in
+the AP Database.  In general, these images correspond to the Chips.
+This table includes sufficient astrometric parameters to represent the
+coordinates of the detections to a sufficient accuracy.
+The Images in the image table group are stored in the Image table
+which contains the (center? 0,0 pixel?) of the chip.  A specific
+coordinate can be specified to a single Image region table.  However,
+it is frequently useful to determine all regions which a specific
+image overlaps.  The Image Overlaps tables contains a list of the
+image regions which are overlapped by each image.
+The Objects table group (divided by region) stores the average
+parameters for each astronomical object.  Certain details of this
+table have not yet been specified.  In particular, objects with
+are used by the AP Database software.  Several of the tables are not
+just simple tables in the database but are instead divided into many
+subtables, each of which represents a portion of the sky.  These
+subtables may also be distributed across different computers to
+distribute the processing load.
+The {\tt Images} table group lists all of the images which provided
+the data in the AP Database.  These tables are subdivided by region.
+In general, the images listed in this table correspond to the Chips.
+This group of tables includes sufficient astrometric parameters to
+represent the coordinates of the detections to a sufficient accuracy.
+The specific subtable of {\tt Images} which contains a given image is
+that one which contains the center pixel \tbr{or 0,0 pixel} of that
+image.  An additional table group, {\tt Image Overlaps} (with the same
+subtable organization as the {\tt Images} subtables), lists images
+which overlap that specific subtable.  Thus, given a particular
+coordinate, in order to find that images which overlap that
+coordinate, it is necessary to load the images in the {\tt Images}
+subtable which includes that coordinate, and all images in the {\tt
+ImageOverlaps} table for that coordinate.
+The {\tt Objects} table group (also divided by region) stores the
+average parameters for each astronomical object.  Certain details of
+this table have not yet been specified.  In particular, objects with
 significant parallax and/or proper motion may potentially be stored in
+a distinct table.  Solar system objects, to the extent average
+properties are maintained, are certainly stored in a separate table.
+A related table, also divided in the same regions, is the Average
+Magnitudes table.  In this table, there are multiple rows per average
+a distinct table.  Solar system object identifications, to the extent
+average properties are maintained in the AP Database, will certainly
+be stored in a separate table.
+A related table, also divided in the same regions, is the {\tt Average
+Magnitudes} table.  In this table, there are multiple rows per average
 object, one for each of the primary filters of interest for which
+photometric averaging is performed.
+The Detections table stores all of the measurements of astronomical
+objects on specific images.  \tbd{is this table divided into P2, P4S,
+P4D tables?  3$\sigma$ objects vs 5$\sigma$ objects?  We don't want to
+store all detections in a single table, I think}.
+The Non-detections table stores information about detection failures
+for each object.  If an image is added to the database which overlaps
+an object but the object is not detected, an entry is made in this
+table.  In fact, this table may store only the most recent
+photometric averaging is performed.  This organization makes the
+number of primary (averaged) filters a configurable value.
+The {\tt Matched Detections} table stores all of the measurements of
+astronomical objects on specific images.  This table includes all
+detections associated with the average {\tt Objects}.  As discussed
+below, bright objects (above a configuration-specified signal-to-noise
+level) are assigned an object even if only one detection has been
+found at that position.  Faint orphaned objects are not added to this
+list or the list of objects.  The different types of detections (P2,
+P4$\Delta$, P4$\Sigma$) are distinguished by their photometry codes.
+\tbr{This is only valid if the AP Database does not store different
+quantities for these types of detections}
+The {\tt Orphaned Detections} table stores the detections which have
+not been correlated with an existing object.  This table is only
+populated for objects below a configuration-specified signal-to-noise
+limit.  Otherwise, even orphaned detections are assigned an object and
+added to the {\tt Matched Detections} table.
+The {\tt Non-detections} table stores information about detection
+failures for each object.  If an image is added to the database which
+overlaps an object but the object is not detected, an entry is made in
+this table.  In practice, this table may store only the most recent
 non-detection and the total number, or a similar reduced set of
 non-detection statistics.
+The table of regions is used to subdivide the tables of images,
+objects, and detections.  The AP Database divides the sky into a
+hierarchy of regions (portions of the sky) each of which is in turn
+sub-divided into smaller portions.  These three tables are the three
+largest in terms of both data volume and number of rows.  Since nearly
+The {\tt Regions} table is used to subdivide the tables of images,
+objects, and detections, etc, as discussed above.  The AP Database
+divides the sky into a hierarchy of regions (portions of the sky) each
+of which is in turn sub-divided into smaller portions.  Since nearly
 all interactions with the AP Database performed by the IPP are limited
 in spatial coverage, subdividing the tables allows a specific
 interaction to search only a small subset of the data.  The table of
 images is the smallest of the three; the table of detections is likely
+to be the largest.  As a result, the images tables will be subdivided
+at a shallow hierarchical level, while the objects and detections are
+subdivided on deeper (more finely sampled) levels.  The region table
+defines the sky regions and specifies if the region corresponds to an
+image table, and object table, and/or a detection table.  It also
+specified which regions in the next level of the hierarchy are
+to be the largest.  As a result, the {\tt Images} table group will be
+subdivided at a shallow hierarchical level, while the {\tt Objects}
+and {\tt Detections} are subdivided on deeper (more finely sampled)
+levels.  The {\tt Regions} table defines the boundaries of the sky
+regions and specifies if the region corresponds to an {\tt Images}
+table, an {\tt Objects} table, and/or a {\tt Detections} table.  It
+also specifies which regions in the next level of the hierarchy are
 contained by the region, and which parent region it belongs to.  In
 addition to improving the spatial access to the image, object, and
 detection data, the region table allows for the multiple computers to
 serve the database tables.  The region file specifies the machine
+detection data, the {\tt Regions} table allows for multiple computers
+to serve the database tables.  The region file specifies the machine
 which stores the specific table.  Figure~\ref{ABDBRegions} illustrates
+this subdivision of the sky and the association between different
+levels of the hierarchy with different subtables.
+The Filters table identifies all of the physical filters (specific,
+named pieces of glass) known to the system.  A related table,
+photcodes, defines relationships between specific photometry systems.
+A system may consist of a detector, telescope, and specific filter, or
+it may be a derived photometry system. \tbd{distinguish between
+reference, average, and detection photcodes}.
+schamatically the subdivision of the sky and the association between
+different levels of the hierarchy with different subtables.
+The {\tt Filters} table identifies all of the physical filters
+(specific, named pieces of glass) known to the system.  A related
+table, {\tt Photcodes}, defines relationships between specific
+photometry systems.  A system may consist of a detector, telescope,
+and specific filter, or it may be a derived photometry system.  The
+{\tt Database Machines} table identifies all of the computers
+available to the AP Database.
 \begin{table}
 …
 {\bf Table Name} & {\bf Description} \\
 \hline
+Region Table       & spatial distribution of tables \\
+Images             & The images that have objects in the DB. \\
+Image Overlaps     & Image regions which are touched by specific images. \\
+Objects            & The objects --- average properties of multiple detections of the same object. \\
+Average Magnitudes & Average photometry in multiple filters \\
+Detections         & Detections of sources in an image. \\
+Non-Detections     & Non-detections of objects in an image. \\
+Filters            & Filters understood by the system. \\
+Photcodes          & Transformations between different photometric systems \\
+Database Machines  & computers used to store the tables \\
+Images              & The images that have objects in the DB. \\
+Image Overlaps      & Image regions which are touched by specific images. \\
+Objects             & The objects --- average properties of multiple detections of the same object. \\
+Average Magnitudes  & Average photometry in multiple filters \\
+Matched Detections  & Detections of sources in an image identified with an Object. \\
+Orphaned Detections & Detections of sources in an image not identified with an Object. \\
+Non-detections      & Non-detections of objects in an image. \\
+Region Table        & spatial distribution of tables \\
+Filters             & Filters understood by the system. \\
+Photcodes           & Transformations between different photometric systems \\
+Database Machines   & computers used to store the tables \\
+% Zero Points       & Transformations between different photometric systems \\
+% Distortion Models & Transformations between different photometric systems \\
 \hline
 \end{tabular}
 …
 the sky is defined by the Region table.  Each machine has a
 corresponding AP Database server which runs on that machine to
+interact with the tables available on that machine.  A client chooses
+one of the machines and sends its query or data to be inserted to that
+machine.  The server then uses the region table to determine which
+machines contain the relevant portion of the sky.  The data to be
+inserted is divided into corresponding region chunks and sent to the
+appropriate servers.  In the case of queries, the queries are
+redirected to the appropriate server(s).  The original server may
+collect the results and return them to the original client.
+\subsubsection{AP DB Operations}
+\begin{itemize}
+\item addstar
+\item delstar
+\item relphot
+\item uniphot
+\item mosastro
+\item distortion
+\end{itemize}
+interact with the tables available on that machine.  Two possible
+interaction models are considered.
+{\bf Option A:} A client chooses one of the machines and sends its
+query or data to be inserted to that machine.  The server then uses
+the region table to determine which machines contain the relevant
+portion of the sky.  The data to be inserted is divided into
+corresponding region chunks and sent to the appropriate servers.  In
+the case of queries, the queries are redirected to the appropriate
+server(s).  The original server may collect the results and return
+them to the original client.
+{\bf Option B:} The client downloads the region table and performs the
+division of the data into appropriate subsets.  The subsets are then
+sent to the corresponding servers by the client.
+The differences between these models is small.  The first option may
+make the code more testable, placing all of the logic in the servers
+and making each server symmetric.  The smaller tables (ie, Region,
+Filters, etc) could either be downloaded from a single server or
+replicated to all AP DB servers.
+\subsubsection{AP Database engine}
+The backend database engine for the AP Database stores the tables and
+provides them to the servers on demand.  The AP Database will use a
+\code{mysql} database engine for the function.
+\subsubsection{AP DB Client operations}
+The AP Database client interactions consist of a collection of basic
+queries of the database, along with more complex operations to perform
+particular tasks.  \tbd{queries are not yet listed; provide list from
+  DVO}.  The complex operations are listed below.
+\paragraph{Insert Image \& Detection Set (addstar)}
+One of the most basic operations needed by the AP Database is to
+insert a collection of detections derived from a specific image, and
+add the definition of that image to the database.  This operation is
+critical in terms of the processing throughput.  After the detections
+have been assigned to the appropriate regions, they are matched
+against all objects in the {\tt Objects} table.  Matches are performed
+only on the basis of positional coincidence, using a matching radius
+which may depend on the image astrometry errors, or may be a fixed
+distance.  Any matched detections are added to the {\tt Matched
+Detections} table.  Any unmatched detections brighter than the Faint
+Detection cut-off are specified as a new {\tt Object} and also added
+to the {\tt Matched Detections} table.  Any faint unmatched detections
+are added to the {\tt Orphaned Detections} table.  This division is
+important because it lets us automatically associate new detections
+with existing bright objects and limits the I/O volume required to
+make the detections.  In general, there will be many few {\tt Objects}
+than {\tt Detections}, and there will be fewer bright orphans than
+faint orphans.
+\paragraph{Insert Reference Objects (addrefs)}
+This operation is very similar to the previous one.  A collection of
+reference objects are added to the database as a collection of
+detections.  The reference photometry should in general be given its
+own photometry code.  The reference data is different from the image
+detection set because the associated image information is not
+included.  Thus, no corresponding images are added to the database.
+\paragraph{Determine Relative Photometry in region (relphot)}
+This operation uses the overlaps of images and multiple observations
+of the same objects to determine the relative photometry zero-points
+for a collection of images.  This is a task which would be run much
+more infrequently than the object insertion tasks.
+\paragraph{Determine Consistent Photometry Zero Points (uniphot)}
+This operation uses the time history of relative photometry zero
+points for images and the spatial overlap information to determine a
+best set of image zero points which have a specific time scale for the
+atmospheric stability.
+\paragraph{Determine Distortion Model (mosastro)}
+This operation uses the reference and image detections to determine an
+optical distortion model for the camera.  ñ
 \begin{table}
 …
 \subsection{Scheduler}
+\begin{figure}
+\begin{center}
+\resizebox{6in}{!}{\includegraphics{pics/Scheduler}}
+\caption{ \label{Scheduler} IPP Scheduler}
+\end{center}
+\end{figure}
 The IPP is responsible for a variety of analysis tasks: processing of
 the science images through several stages; routine assessment of the
 …
 The IPP Scheduler acts as an intermediary between several components
 of the IPP and also between the IPP and external agents such as OTIS
+and the users who must monitor the behavior of the IPP.
+and the users who must monitor the behavior of the IPP.  The IPP
+Scheduler may be viewed as the central brain of the IPP.
+Figure~\ref{Scheduler} illustrates the design of the IPP Scheduler.
+The IPP Scheduler performs two types of actions.  'Tasks' are
+long-running programs which are executed by the Controller.  These are
+not only background tasks, but are distributed computing tasks.
+Examples of these include the science analysis tasks (eg, Phase 1, 2,
+, 4), the Calibration construction tasks, and data copy tasks (such
+as copying images and metadata from the summit system).  'Tests' are
+short-running programs which are used to decide which tasks should be
+run.  Tests should be designed to return immediately ($< 100 ms$) and
+are not run in the background; the Scheduler will block until the test
+is complete.  The IPP Scheduler daemon, which runs continuously,
+performs tests (eg, queries of the IPP Metadata Database, queries of
+OTIS, checks of the IPP hardware status, etc).  Based on these tests,
+the daemon defines appropriate tasks and sends them to the Controller.
+When tasks are completed, their results may be used by the Scheduler
+to update the external systems (update the Metadata Database), or the
+tasks themselves may send their results directly to the Metadata
+Database or other subsystems.  Based on the successful completion (or
+not!) of the tasks, and the new state of entries in the Metadata
+Database, the Scheduler can define new tasks.
 The IPP Scheduler sends commands to the IPP Controller for execution.
 …
 Controller does not need to have any information about the details of
 the tasks which it executes, while the IPP Scheduler does not need to
 have detailed information about the available computer hardware.
+monitor the computer hardware.
 Communication between the IPP Scheduler and the IPP Controller is
 …
 The analysis stages are written as UNIX commands, which may be
 executed by the IPP Controller, or may be executed individually by
 hand.  This aspect makes testing of the complete analysis system much
+hand.  This option makes testing of the complete analysis system much
 easier because the individual analysis stages may be tested
+independently of each other and the IPP infrastructure.
+In keeping with this design model, the analysis stages have several
+methods for accepting and returning the input and output data.  All of
+the analysis stages load an analysis recipe file, which defines the
+details of the analysis.  This includes the location of the data
+sources (from the metadata, from the image headers, from other
+external files, or supplied directly), and which steps to employ.  For
+example, in the discussion of the Phase 2 analysis below, the recipe
+file may specify {\em if} a bias subtraction should be applied, {\em
+where} to find the overscan region and {\em which} bias image, if any,
+to apply.
+\tbd{further discussion of the recipe / configuration files?}
+independently of each other and the IPP infrastructure.
+As part of this design model, the analysis stages have several methods
+for accepting and returning the input and output data and for defining
+optional choices in the analysis.  All of the analysis stages load an
+analysis recipe, which defines the details of that analysis.  The
+recipe includes the location of the data sources (from the metadata,
+from the image headers, from other external files, or supplied
+directly), which steps to employ, and how to assign optional
+parameters.  For example, in the discussion of the Phase 2 analysis
+below, the recipe file may specify {\em if} a bias subtraction should
+be applied, {\em where} to find the overscan region and {\em which}
+bias image, {\em if any}, to apply.
+The recipe is loaded as part of the runtime configuration information
+loaded when the analysis script starts.  We define four levels of
+runtime configuration information.  The {\tt site} configuration
+defines values specific to the particular installation of the
+software.  For example, the name of the machine which hosts the
+Metadata Database or a default path for data files could be part of
+the {\tt site} configuration.  Multiple installations or versions of
+the IPP software would need to have separate {\tt site} configuration
+entries.  For example, a version of the IPP installed at the IfA would
+use a different computer for the Image Server from the live IPP
+installation running on the Pan-STARRS cluster.  The {\tt base}
+configuration defines general data sources which may be needed by any
+portion of the IPP.  The list of known telescope or filters might be
+an example.  The {\tt camera} configuration consists of information
+which defines the parameters relevant to the camera known by the IPP.
+For example, the default layout of the detectors or the names of
+specific header keyword values would be defined for each camera in a
+camera-specific configuration collection.  Finally, each analysis
+script loads its own recipe.  The location of this configuration
+information may be a collection of configuration files available on
+disk or some subset of the information may be stored in the Metadata
+Database.  The source of these configuration entries can be overridden
+when the script is executed, and individual configuration values may
+also be specified on the command line.  Examples of the recipe and
+other runtime configuration options are given in
+Appendix~\ref{RuntimeConfig}.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 …
 \item Form OT kernel
 \item Convolve detrend images with the OT kernel
+\item Bias/dark/overscan subtraction
 \item Mask bad pixels
-\item Mask diffraction spikes and optical ghosts
-\item Bias/dark/overscan subtraction
 \item Trim overscan
 \item Non-linearity correction
 \item Flat-field
+\item Mask diffraction spikes and optical ghosts
 \item Subtract sky
+\item Find and photometer objects in the image
 \item Identify CRs by morphology
 \item Determine PSF model
+\item Find and photometer objects in the image
+\item Improved astrometry
+\item Improve astrometry
 \item Extract Bright object postage stamps
 \end{itemize}
+Several of the steps are explained in detail below.
+\subsubsection{Form OT Kernel}
+Certain detrend images are convolved by the OT kernel, so that they
+accurately represent the detrend images appropriate for the object
+images, which have been shifted using OT.  The detrend images which
+must be convolved include: the flat-field and the
+high-spatial-frequency fringe images. The appropriate kernel for each
+cell of an OTA must be determined from the guide star history,
+extracted from the IPP Metadata Database\footnote{or image header}.
+If the OT kernel is not available, but the image metadata notes that
+it should be, the convolution is skipped, with a warning.
+The first module for Phase 2 forms the OT kernel from the list of
+pixel shifts made during the exposure.  This involves decoding the
+metadata and converting it to a data type that can be used to convolve
+by.  The output is the OT convolution kernel.
+\subsubsection{Convolve detrend images}
+This module convolves the detrend images with the OT convolution kernel
+so that they can be used to detrend the object image.  The inputs
+are:
+\begin{itemize}
+\item The OT convolution kernel --- from the previous module;
+\item The appropriate dark frame --- from the IPP Pixel Server;
+\item The appropriate flat-field --- from the IPP Pixel Server;
+\item The appropriate fringe frame(s) --- from the IPP Pixel Server; and
+\item The appropriate static bad pixel mask --- from the IPP Pixel Server.
+\end{itemize}
+The module convolves each of the dark frame, flat-field, and the fringe
+frame(s) by the OT convolution kernel.  Specific flags in the static
+bad pixel mask are grown by the outline of the OT convolution kernel
+(see Section \ref{ap:masks}).  The output results are:
+\begin{itemize}
+\item The convolved flat-field;
+\item The convolved fringe frame(s); and
+\item The updated pixel mask.
+\end{itemize}
+Each of these will be used for a later module.  The convolution method
+depends on the size and structure of the OT kernel.  If the kernel is
+small ($< 5x5$ pixels), direct convolution may be employed.  If the
+kernel is large, but may be decomposed using Gaussians, then it may be
+convolved using a decomposition method.
+The steps are explained in detail below.
+\subsubsection{Load Images}
+The Phase 2 analysis must load the science image to be analysed into
+memory, as well as the corresponding metadata (either from the image
+header and/or from the IPP Metadata Database).  It must use the
+metadata for the image, along with information from the processing
+recipe, to determine the appropriate detrend images to be used for
+this analysis.  The Metadata Database stores the information necessary
+to associate a specific science image with one of the registered
+master detrend images for each type.  These images are also loaded by
+the Phase 2 analysis (note that the design of Phase 2 may perform the
+actual loading of pixels in blocks or groups to minimize the memory
+impact).
+\subsubsection{Form OT Kernel \& Convolve with Detrend Images}
+Science images which have been obtained with Orthogonal-Transfer
+Guiding have had thier pixel response smoothed by the image correction
+motion.  For these images, some of the detrend images need to be
+convolved by the same OT kernel, so that they accurately represent the
+effective pixel response.  The detrend images which must be convolved
+include: the flat-field image, the high-spatial-frequency fringe
+images, and dark images, if they are used. The appropriate kernel for
+each cell of an OTA must be determined from the guide star history,
+extracted from the IPP Metadata Database or from the image header.  If
+the OT kernel is not available, but the image metadata notes that it
+should be, the convolution is skipped, with a warning.
+The convolution method depends on the size and structure of the OT
+kernel.  If the kernel is small ($< 5x5$ pixels), direct convolution
+may be employed.  If the kernel is large, but may be decomposed using
+Gaussians, then it may be convolved using a decomposition method.
+The module convolves each of the dark frame, flat-field, and the
+fringe frame(s) by the OT convolution kernel.  Specific flags in the
+static bad pixel mask are also grown by the outline of the OT
+convolution kernel (see Section \ref{ap:masks}).
 \subsubsection{Bias Correction / Overscan Subtraction}
+This module corrects the object exposures for the electronic pedestal
+introduced by the readout electronics.  The inputs are:
+\begin{itemize}
+\item The object image --- from the IPP Pixel Server;
+\item The pixel mask --- from the previous module;
+\item The overscan and physical detector regions --- from the
+Metadata; and
+\item Detector characteristics (gain, read noise) --- from the
+Metadata.
+\end{itemize}
+The overscan is averaged (either in bulk, or individually by rows) or
+fit with a polynomial, and the result is subtracted from the image.
+Overscan rows having a standard deviation which exceeds a threshold of
+twice (configurable) the detector read noise should be masked.  Pixels
+saturated in the A/D converter should also be masked, and these
+regions grown by an additional pixel to counter CCD ``blooming''.  The
+output is:
+\begin{itemize}
+\item The overscan-subtracted object image; and
+\item The updated pixel mask.
+\end{itemize}
+These will be used for a subsequent module.
+\subsubsection{Trim}
+This module trims the object image and each of the calibration frames to
+remove the outer edge which was affected by the OT during the
+exposure.  The inputs, each from previous modules, are:
+\begin{itemize}
+\item The overscan-subtracted object image;
+\item The corresponding pixel mask;
+\item The convolved flat-field;
+\item The convolved fringe frame(s); and
+\item The dimension of the OT convolution kernel in each direction.
+\end{itemize}
+Each of the input frames (object image, flat-field, fringe frame(s)
+and pixel mask) are trimmed by the extent of the OT convolution kernel
+in each direction ($+x$, $-x$, $+y$, $-y$).  The outputs are trimmed
+images for each of the input images, which will be used in later
+modules.
+\subsubsection{Non-Linearity Correction}
+This module corrects images for non-linearity in the detector.  The
+inputs are:
+\begin{itemize}
+\item The trimmed object image --- from a previous module; and
+\item The detector non-linearity correction coefficient(s) --- from
+the Metadata.
+\end{itemize}
+The module corrects the flux in each pixel for non-linearity by applying
+a polynomial correction, with the specified coefficients.  The output
+is the corrected object image, which is used for a later module.
+\subsubsection{Flat field}
+This module corrects the object image for variations in sensitivity over
+the image.  The inputs are:
+\begin{itemize}
+\item The object image corrected for non-linearity;
+\item The corresponding pixel mask; and
+\item The convolved, trimmed flat-field.
+\end{itemize}
+Each of these comes from a previous module.
+The module divides the object image by the flat-field, masking pixels
+that are non-positive in the flat-field.  The outputs are:
+\begin{itemize}
+\item The flattened object image; and
+\item The updated pixel mask.
+\end{itemize}
+Both of these will be used in later modules.
+\subsubsection{Subtract sky}
+This module subtracts the sky background from the object image.  The
+inputs are:
+\begin{itemize}
+\item The object image --- from the previous module;
+\item The list of objects on the image --- from the object database; and
+\item The convolved, trimmed fringe frame(s) --- from a previous module.
+\end{itemize}
+The module masks (though {\em not} in the ``official'' pixel mask) all
+objects on the image using the astrometric solution from the
+boresight, and fits for the sky background, consisting of a polynomial
+to model the continuum, and the fringe frame(s) to model the fringes
+from sky emission lines.  If the concentration of objects in the image
+is too high to reliably fit the sky background, the background
+solution from an exposure close in time and airmass to the current
+object image is used.  The output is the sky-subtracted object image,
+which is used for the next module.
+\subsubsection{Identify CRs by morphology}
+This module identifies cosmic rays (or other hot pixels missed in the
+static bad pixel mask) on the basis of their morphology.  The inputs
+are:
+\begin{itemize}
+\item The object image; and
+\item The corresponding pixel mask.
+\end{itemize}
+Both of these come from a previous module.
+The module identifies CRs, the pixels of which are masked in the pixel
+mask.  The pixels flagged as CRs are then grown by an additional pixel
+in each direction.  Masked pixels are interpolated over.  The outputs
+are the updated pixel mask, which is sent to the IPP pixel server for
+use in Phase 3, and is also used for the next module; and the object image,
+which is sent to the IPP Pixel Server.
+\subsubsection{Detect and Measure objects}
+This module finds objects on the object image.  The inputs are:
+\begin{itemize}
+\item The sky-subtracted object image; and
+\item The corresponding pixel mask.
+\end{itemize}
+Both of these come from a previous module.
+The module identifies objects on the image, which will be later used to
+register images from different focal planes.  The output is the
+catalog of objects (see Appendix~\ref{ap:catalogs}) identified on
+the image, which is sent to the metadata database, associated with the
+object image.
+Object catalogs from Phase 2 shall consist of at least the
+following elements for each object:
+\begin{itemize}
+\item Object centre, with corresponding errors;
+\item Object magnitude, with corresponding error;
+\item Object isophotal magnitude, with corresponding error;
+\item Object FWHM;
+\item Object elliptical axis lengths; and
+\item Object position angle for ellipse.
+\end{itemize}
+Though further details may be required for catalogs in Phase 4,
+the above details are minimum requirements for Phase 2 catalogs.
+\subsubsection{Bright object postage stamps}
+This module saves postage stamps of bright objects, so that extra care
+with regard to astrometry and photometry can be taken with them at a
+later stage.  The inputs, each from a previous module, are:
+\begin{itemize}
+\item The sky-subtracted object image;
+\item The corresponding pixel mask; and
+\item The catalog of objects.
+\end{itemize}
+The module makes postage stamps of all objects brighter than a given
+instrumental magnitude, along with corresponding pixel masks.  The
+outputs are these postage stamps and pixel masks, which are sent to
+the IPP Pixel Server.
+\subsubsection{Pixel Masks}
+\label{ap:masks}
+This section describes the requirements on Bad Pixel Masks (BPMs).
+These will consist of bit masks for each pixel.  For Phase 2, flags
+The image bias must be subtracted. Since different detectors behave in
+different ways, several options for modelling the bias are available.
+The bias is measured from the image overscan region.  The bias
+subtraction method must be capable of subtracting a single constant
+from the complete image, or to subtract a 1-D bias which varies as a
+function along the overscan.  The function used to represent the
+overscan region may be a spline or a chebychev polynomial derived from
+the data values along the overscan.  The values used to determine both
+the single constant or the inputs to the spline and polynomial fits
+are derived from groups of pixels on the basis of one of several
+statistics, including the sample and robust mean, median, and modes.
+In the case of a single constant, all of the overscan pixel values are
+used in the calculation of this statistic.  In the case of the 1-D
+functional representation, the input values to the fit must represent
+the coordinate along the overscan, with the statistic derived from the
+pixels in the perpendicular direction at each location.
+Sigma-clipping on the input data values must be an option.
+\subparagraph{Flag bad and saturated pixels}
+A static bad pixel mask is used to identify pixels which are known to
+be bad in the camera.  This mask is then processed with the science
+image. Bad pixels which are charge traps are grown by the extent of
+the OT convolution kernel.  Bad pixels above a charge trap (i.e.\ bad
+colums) must not be grown, since they were not affected by pixel
+shifting, but only became bad at read-out.
+Pixels which are saturated in the A/D converter, or with a signal
+level at which the response is excessively non-linear, must also be
+masked, and this area must be grown by an additional pixel to mask
+excess charge spillover.
+The bad pixel mask is carried with the science images.  Different bits
+are set to identify different reasons for masking the pixel.  Flags
 are required for at least each of the following pixel attributes:
 \begin{itemize}
 \item The pixel is a charge trap;
 …
 \end{itemize}
+Of these, only masks for the charge traps need to be grown by the
+extent of the OT convolution kernel.  For other pixel types,
+orthogonal transfer of the flux in this pixel will not (necessarily)
+affect the flux in neighbouring pixels
+\subsubsection{Phase 2 Metadata}
+The following metadata associated with the images are required for
+Phase 2 operation:
+\begin{itemize}
+\item The orthogonal transfer (OT) image shifts made during the
+exposure --- in order to create a convolution kernel;
+\item Time of observation --- for selecting the appropriate detrend
+images;
+\item Filter --- for selecting the appropriate detrend images;
+\item Telescope identification --- for selecting the appropriate
+detrend images;
+\item Exposure time --- for the photometric calibration;
+\item Detector gain --- for calculating photometric errors and
+determining the quality of the overscan;
+\item Detector read noise --- for calculating photometric errors and
+determining the quality of the overscan;
+\end{itemize}
+\subsubsection{Notes}
+\tbd{how long are processed images kept?}
+\tbd{what subsystem deletes processed images?}
+\subsubsection{Trim}
+The image is trimmed to remove the non-imaging pixels, such as the
+overscan and any pre-scan pixels, along with those pixels near the
+edges that have been compromised due to OT operation.  The definition
+of the imaging area of the detector is determined from the camera
+configuration data or from the metadata associated with the image,
+with the choice a user-configurable option.
+The input science and mask frames are trimmed by the extent of the OT
+convolution kernel in each direction ($+x$, $-x$, $+y$, $-y$).  Within
+the PSLib image handling functions, the trim function is a virtual
+operation which simply marks the boundaries of the trimmed image but
+does not remove the corresponding memory.  This allows the later
+corrections to work with untrimmed correction images and still apply
+the correct pixels.  At the end of Phase 2, the only the trimmed
+portions of the output images are written out to disk.
+\subsubsection{Non-Linearity Correction}
+If required, the science image (after bias correction) must be
+corrected for the effects of non-linearity through a provided
+polynomial fit to the pixel data values or a numeric lookup table as a
+function of pixel flux.  The choice to apply the correction must be
+set by the analysis recipe.
+\subsubsection{Flat field Correction}
+The object image (after bias correction and non-linearity correction)
+must be corrected for sensitivity variations as a function of
+position, dividing by a flat-field image.  The mask is also modified
+for zero-valued pixels in the flat-field image.
+\subsubsection{Sky \& Fringe subtraction}
+After the science image has been flat-fielded, the flux contribution
+of the sky (from both continuum emission and the line emission that
+causes fringing) must be subtracted from the image.  The subtraction
+needs to remove background (technically, foreground) variations which
+are not celestial but generated in the atmosphere or by more localized
+scattering.  This background should include the contribution from the
+zodiacal light.  This background subtraction does not address the
+artifacts generated by bright stars: bleeding columns, ghosts, or
+other localized reflection effects.  This process also produces a
+superbinned image of the background map which may be used as a
+debugging diagnostic.
+This analysis step temporarily masks objects on the image using the
+astrometric solution from the boresight and fits for the sky
+background, consisting of a polynomial to model the continuum, and the
+fringe frame(s) to model the fringes from sky emission lines.  If the
+concentration of objects in the image is too high to reliably fit the
+sky background, the background solution from an exposure close in time
+and airmass to the current object image is used.  The output is the
+sky-subtracted object image.
+\subsubsection{Detect and Measure objects}
+After the image have been processed by the preceeding steps, the Phase
+analysis performs a basic object detection analysis.  Objects on the
+flat-fielded object image are found, and general parameters are
+measured.  Object detection is performed at several stages by the IPP,
+with different object parameters measured in each case.
+Table~\ref{APdetections} gives a list of the different detection
+stages and the object parameters measured for those stages.  For the
+Phase 2 analysis, the object parameters are: the object centroid and
+the position covarience matrix, the instrumental PSF magnitude and
+error, local background level and error, a measurement of the
+star-galaxy separation, and a measurement of the object shape
+($\sigma_x, \sigma_y, \theta$).  The detection threshold must be
+configurable, and be a function of the average background flux or the
+image noise map.  Minimal object classification must be performed to
+distinguish objects which are consistent with a single PSF, objects
+which are inconsistent, and objects which are saturated.  The
+resulting collection of detected objects are saved in the AP Database
+along with the relevant image metadata (\ie filter, exposure time,
+etc).  In addition, this process constructs a model of the
+point-spread-function (PSF) as a function of position in the image.
+This PSF model is saved as part of the image metadata.
+\subsubsection{Identify CRs by morphology}
+Charged particles in the detector frequently cause features which do
+not have the morphology of astronomical objects.  In a well-sampled
+image, these may be easily identified by the sharpness of the image.
+In a near critically-sampled image, these `cosmic rays' may be
+indistinguishable from stellar objects.  This analysis step makes
+morphological identification of cosmic rays if the imaging data is
+sufficiently well sampled.  The identified cosmic rays are masked with
+a configurable growth factor so that additional pixels beyond the
+detected pixels in the feature are also masked.
+\subsubsection{Perform Astrometry}
+The detected objects are matched with known astrometric reference
+objects, using reference object coordinates which have been adjusted
+for proper motion.  The matches are then used to improve the
+astrometric parameters for the individual OTAs.  The OTA astrometric
+parameters which are determined may include terms up to 3rd order in
+position, though the terms which are actually fitted are
+user-configurable.  The Cell astrometric parameters are not allowed to
+vary at this stage.  The fit must be robust, rejecting outlier matches
+(either stars with poorly determined proper motion or spurious
+matches).  The resulting astrometric solution is consistent across the
+OTA field to within \tbr{0.2 arcsec}.
+\subsubsection{Perform Photometry}
+If possible (if a local photometry reference exists), the Phase 2
+analysis determines a photometry zero point for each image.  To do
+this, it extracts the appropriate reference objects (from the AP
+Database) and matches the stars between the two samples.  The
+zero-point is derived on the basis of a static atmospheric absorption
+model (eg, linear airmass slope).
+\subsubsection{Bright object postage stamps}
+The IPP must have the capability of extracting regions surrounding a
+specified subset of objects from the flattened images.  These postage
+stamp images must be saved for additional use by client science
+pipelines.  The identification of these objects must be on the basis
+of a set of rules applied to the object's magnitude and position.  The
+postage stamps are not restricted in shape to simple rectangles, but
+may represent more complex regions.  They are written to the Image
+Server.  The outputs are these postage stamps and pixel masks, which
+are sent to the IPP Pixel Server.
 \begin{figure}
 …
 \item photometric solution based on comparison to photometric
   standards
+\item FPA-wide PSF analysis
 \end{itemize}
 …
 assumption of a static distortion and by the accuracy of the
 astrometric reference.  In the phase 3 analysis, the astrometric
+solutions of the $N$ FPA images are improved by...
+For image combination in phase 4, should we use relative astrometry to
+map N-1 images to 1, or are we sufficiently accurate to use absolute
+astrometry to map N images to the sky-cells?
+solutions of the complete FPA images are improved by combining the
+astrometry for all chips.  The astrometry model consists of a
+projection of the celestial coordinates to the telescope boresite
+center, followed by a rotation to the average rotation of the FPA and
+adjustment for the central plate scale.  The free parameters in this
+stage are the boresite coordiates ($R_o, D_o$), the field rotation
+($\theta_o$) and the plate scale ($\rho_o$), and are fitted in Phase
+.  These tangent plane coordinates are then distorted by the optical
+distortion model, consisting of $N^{\rm th}$ order polynomials in two
+dimensions.  Finally, the focal plane coordinates are mapped to the
+individual chip coordinates, including the chip position and rotation,
+as well as possible higher order terms representing warping of the
+individual detectors.  A first pass at the chip positions is
+calculated in Phase 2, while the complete set of parameters is fitted
+as a whole during Phase 3.  The fitting process is determined in a
+robust and stable way by fitting the gradient of the distortion as a
+function of field position, removing the degeneracy of the distortion
+coefficients on the chip position parameters.
 In the Phase 2 analysis, the background is determined based only on
 …
 structures are normally correlated on the scale of the FPA, so an
 improved background solution can be determined by combining the
+information from many chips.  \tbd{is the background correlated
+between FPAs?}
+Phase 3 photometric improvement
+information from many chips.  A high-order polynomial model of the
+background may be used and subtracted from all chips.
+The Phase 3 photometric improvement is made by comparing the
+photometered objects from Phase 2 with the corresponding objects in a
+local reference catalog.  This analysis may only be performed if a
+local reference is available.  Note that improved relative photometry
+calculations may be performed in the absense of a reference catalog on
+the basis of image overlaps in the AP Database {\em after} the
+detections have been added to the Database.  Such a relative
+photometry analysis is outside the scope of Phase 3 and will likely be
+performed as an independent analysis process.  Given the presence of a
+local photometry reference, the zero point variations across the field
+may be measured, and possibly modelled.  If the zero-point variations
+are excessive, then the image is marked as non-photometric by the
+analysis.
 In the Phase 4 analysis, the $N$ FPA images are optimally combined to
 …
 \subsubsection{Overview}
+Phase 4 processing within the Pan-STARRS image processing pipeline is
+the image combination stage of processing for a science image.  It
+operates on each sky cell that has overlapping imaging data from the
+exposure(s) being processed, and produces a set of clean, combined
+images of the sky.  It also subtracts the current static sky image to
+generate a difference image, which it uses to identify transient
+objects.  These are then excised from the summed image, which is in
+turn then added to the static sky image.
+Prior to Phase 4, the Phase 3 process produces the following products:
+\begin{itemize}
+\item bias-subtracted, flattened, sky-subtracted images;
+\item photometric calibration;
+\item astrometric calibration with mapping to sky cells; and
+\end{itemize}
+These will each be used by the Phase 4 modules:
+\begin{itemize}
+\item Combine Images;
+\item Identify Sources;
+\item Transient Identification; and
+\item Add to Static Sky.
+\end{itemize}
+These modules are each explained below.
+\subsubsection{Combine Images}
+\tbd{for moving objects and images which are not simultaneous, do we
+  identify the moving objects?}
+\tbd{use the spatial information?  fit a 2-D Nth order polynomial to
+  the collection of pixels and then look for outliers}
+The first module for Phase 4 is to combine the images from each
+telescope, rejecting artifacts such as cosmic rays and low altitude
+streaks.  The inputs to this module are:
+\begin{itemize}
+\item the sky-subtracted images that overlap the sky cell (or portions
+thereof) --- from the IPP Pixel Server (or directly from Phase 3);
+\item a \tbd{linear} map for the image pixels of each detector to the
+sky cell pixels --- from Phase 3;
+\item photometric calibration (zeropoint) for each image --- from
+Phase 3; and
+\item a (relative) weighting for each image proportional to the
+signal-to-noise (i.e.\ sky noise divided by the square of the seeing)
+--- from metadata associated with the images.
+\end{itemize}
+The module maps the detector images to the sky cell using the specified
+linear transformations, combines the images with strong rejection
+criteria and uses the combined sky cell image to identify artifacts in
+the original detector images.  It is desirable that the artifacts are
+masked in the detector plane (i.e.\ before mapping to the sky cell) so
+that they are not smeared out by the mapping; alternatively, the CR
+mask needs to be grown by an additional pixel (which is likely
+faster).  The mapped and masked detector images are then optimally
+combined using the specified weighting.  Both sets of combinations use
+the photometric calibration for the images to set the relative scales
+of the input images.  The final combination should have the adopted
+Universal zeropoint (25 mag, configurable).  The limiting magnitude
+for the combined sky cell image should also be estimated.
+The outputs from this module are:
+\begin{itemize}
+\item The combined sky cell image --- sent to the IPP Pixel Server
+and/or the next module;
+\item Limiting magnitude of the combined sky cell image --- metadata
+associated with the combined sky cell image, and used for a later module
+in Phase 4; and
+\item Catalog of sources on the combined sky cell image --- sent to
+the IPP Object Database.
+\end{itemize}
+\subsubsection{Identify Sources}
+This module identifies sources in the combined sky cell image.  The
+input is the combined sky cell image, which is obtained from the IPP Pixel Server
+or the previous module.
+Sources are identified on the combined sky cell image by convolving
+with the PSF and searching for peaks above the noise.  The output
+is the catalog of sources on the combined sky cell image, which is to
+the IPP Object Database.
+\subsubsection{Transient Identification}
+\tbd{what about different stellar colors?}
+This module identifies variable/moving sources.  The inputs are:
+\begin{itemize}
+\item The combined sky cell image --- from the previous module or the
+IPP Pixel Server; and
+\item The current static sky image --- from the Sky Image Server.
+\end{itemize}
+The module subtracts the current static sky image from the combined sky
+cell image.  In order to do so, the PSFs need to be matched.  This is
+done by convolving the image that has the narrower PSF with the
+kernel, which is the ratio of the two PSFs (this should be done with a
+fit to the kernel instead of just using the data).  It should be
+sufficient to assume that the kernel is constant over the sky cell
+(otherwise, the sky cell can be broken into smaller sections).
+The subtracted image is scoured for point sources above the noise
+threshold, as well as short and long streaks caused by asteroids and
+satellites, respectively.  It may be neccessary to determine whether
+the detection is false by virtue of its PSF (a cosmic ray missed by
+the combination script should have a very narrow PSF, at least in one
+dimension), or negative pixels surrounding a positive core (caused by
+a bad subtraction, in turn caused by a bad kernel).
+If the subtraction is very bad (many false detections), then Phase 4
+for this sky cell should fail neatly, with a flag for the human
+supervisor.  Otherwise, all variable sources identified in the
+subtracted image should be masked in the combined sky cell image.  The
+pixels from the combined sky cell image for point sources and short
+trails (asteroids) should be saved (say, 3 $\times$ FWHM in radius
+surrounding the source, configurable).  The long trails (satellites)
+should be removed in the combined sky cell image and the subtracted
+image, from edge to edge.  The dividing limit between short and long
+trails shall be a configurable parameter, initially set to 15 degrees
+per day.
+The module outputs:
+\begin{itemize}
+\item Combined sky cell image, with all variable sources masked ---
+used for the next module;
+\item Subtracted image, with long trails masked --- sent to the IPP
+Pixel Server; and
+\item Catalog of variable sources --- sent to the IPP Object
+Database.
+\end{itemize}
+\subsubsection{Add to Static Sky}
+\tbd{how to handle variable stars?}
+This module adds the combined sky cell image into the static sky, so
+that a deep image of the sky may be formed.  This step should only be
+performed if the new data is of sufficient quality that it will not
+degrade the static sky image.  The inputs are:
+\begin{itemize}
+\item The combined sky cell image with variable sources masked ---
+from a previous module;
+\item The current version of the static sky --- from a previous module,
+or the IPP Pixel Server; and
+\item Relative weightings, based on the relative signal-to-noise in
+each of the images --- estimate made from metadata associated with
+each image.
+\end{itemize}
+The sky cell image is added to the static sky.  The sky cell image
+should already be photometrically accurate (when combined), and
+variable sources have been masked, so it is safe to simply add the
+images, employing the weightings.  Sources should be identified on the
+new static sky, and the limiting magnitude of the new static sky image
+estimated.
+The output is:
+\begin{itemize}
+\item The new static sky image --- sent to the Sky Image Server;
+\item The Catalog of sources on the new static sky image --- sent to the IPP Object Database; and
+\item The estimated limiting magnitude for the new static sky ---
+metadata associated with the the new static sky image.
+\end{itemize}
+\subsubsection{Notes}
+\begin{itemize}
+\item Catalogs should include positional information ($x,y$, with
+associated errors), photometry (with associated error), and shape
+parameters (FWHM, major and minor axes, position angle).
+\item Limiting magnitudes can be obtained by photometering many
+regions of blank sky (if possible), and (robustly) estimating the mean
+and standard deviation (in counts).  The limiting magnitude is the
+magnitude corresponding to 3 (configurable) standard deviations above
+the mean.
+\end{itemize}
+Phase 4 is the image combination stage, in which multiple images of
+the same portion of the sky are merged and confronted with the static
+sky image.  Phase 4 operates on the smallest data unit of the static
+sky, the sky cell, along with the associated pixels from a collection
+of images which have been processed through phases 1--3.  The size and
+exact representation of a static sky cell are yet to be determined.
+The working concept is that the static sky cells contain roughly the
+same number of pixels as an OTA (4k x 4k) and represent a portion of a
+local tangent plane projection.  As mentioned above
+(Section~\ref{IPP:ImageServer}), the pixel scale of the static sky is
+planned to be 0.2\arcsec, somewhat smaller than the 0.3\arcsec\ raw
+image pixel scale.
+For each sky cell, the corresponding pixels are extracted from the
+exposures being processed and mapped to the projection of the sky
+cell. The pixels from the multiple input processed images are combined
+into a single, cleaned image.  Outlier pixels may be optionally
+rejected at this stage (optionally, since moving objects will be
+rejected in images obtained over a wide range of times).  This image
+is then confronted with the static sky cell data to produce a
+difference image.  Residual objects in the difference image above a
+threshold are detected and excised from the original cleaned image.
+The remaining pixels are added to the existing static sky image.
+Object detection must be performed on the difference ($P4\Delta$) and
+cleaned ($P4\Sigma$) images.
+\subsubsection{Image Warping and Matching}
+The analysis first maps the detector images to the sky cell using the
+specified linear transformations, then combines the images with strong
+rejection criteria and uses the combined sky cell image to identify
+artifacts in the original detector images.  It is desirable that the
+artifacts are masked in the detector plane (i.e.\ before mapping to
+the sky cell) so that they are not smeared out by the mapping;
+alternatively, the CR mask needs to be grown by an additional pixel.
+The mapped and masked detector images are then optimally combined
+using the specified weighting.  Both sets of combinations use the
+photometric calibration for the images to set the relative scales of
+the input images.  The limiting magnitude for the combined sky cell
+image should also be estimated.
+%
+\subsubsection{Static Sky Subtraction}
+\tbd{add some details about the static-sky subtraction issues.
+  Allard-Lupton-Price method}.
+\subsubsection{Object Detection and Measurement}
+Objects in the difference image are detected and a specific set of
+object parameters are measured from these detections.
+Table~\ref{APdetections} gives a list of the different detection
+stages and the object parameters measured for those stages.  For the
+Phase 4 difference image (P4$\Delta$), the measured object parameters
+consist of: the object centroid and the position covarience matrix,
+the instrumental PSF magnitude and error, local background level and
+error, a measurement of the star-galaxy separation, and a measurement
+of the object shape ($\sigma_x, \sigma_y, \theta$).  The detection
+threshold must be configurable, and be a function of the average
+background flux or the image noise map.  Detections must be performed
+for both positive and negative (static sky only) sources.  Minimal
+object classification must be performed to distinguish objects which
+are consistent with a single PSF, objects which are inconsistent, and
+objects which are saturated.  The resulting collection of detected
+objects are saved along with the relevant image metadata (\ie filter,
+exposure time, etc).
+Objects in the cleaned, summed image are detected and a specific set
+of object parameters are measured from these detections.
+Table~\ref{APdetections} gives a list of the different detection
+stages and the object parameters measured for those stages.  For the
+Phase 4 summed image (P4$\Sigma$), the measured object parameters
+consist of: the object centroid and the position covarience matrix,
+the instrumental PSF magnitude and error, local background level and
+error, a measurement of the star-galaxy separation, a measurement of
+the object shape ($\sigma_x, \sigma_y, \theta$), the Petrosian radius,
+magnitude, axis ratio, and angle; and the S\'ersic radius, magnitude, axis
+ratio, angle, and parameter $\nu$.  The detection threshold must be
+configurable, and be a function of the average background flux or the
+image noise map.  Minimal object classification must be performed to
+distinguish objects which are consistent with a single PSF, objects
+which are inconsistent, and objects which are saturated.  The
+resulting collection of detected objects are saved along with the
+relevant image metadata (\ie filter, exposure time, etc).  In this
+measurement, objects at known positions will also be measured even if
+they have not been detected.
+Objects which are detected in both of the Phase 4$\Sigma$ and Phase
+$\Delta$ images are saved to the AP Database, along with the relevant
+image metadata (\ie filter, exposure time, etc).  In the process of
+adding these objects to the database, the transients which are
+correlated with previous detections of an object (and those which are
+not) will automatically be determined.  An independent process will
+query the AP Database for such transient objects of interest which are
+to be sent, along with their associated metadata, to the MOPS and
+other science client pipelines.  This step must be performed at least
+once per night.
+\subsubsection{Static Sky Update}
+It is essential that the static sky image (which may have been
+painstakingly accumulated over many months) not be corrupted by adding
+in transient sources, or data that is of suspect quality (due, e.g.,
+to an error upstream in the processing).
+Object analysis of the static sky images is {\em not} a part of the
+Phase 4 analysis.  This processing is envisioned to take place
+relatively infrequently (perhaps weekly or even monthly) and is
+scheduled as a separate analysis task, probably run during the day at
+a time when the computing infrastructure is not under significant load.
 \begin{figure}
 …
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\section{System Design : Reference Catalog Processing}
+The IPP reference catalog pipelines use the data in the IPP Metadata
+Database and the IPP Object Database to determined improved
+astrometric and photometric calibration references.
+\section{System Design : Miscellaneous Tasks}
+In this section, we discuss the design of the science analysis stages
+which perform the fundamental image analysis steps of the IPP.
+\subsection{Retrieval}
+The retrieval stages simply retrieve pixel data from an external
+source (ordinarily OATS at the Summit, but it could conceivably be
+some other external source) and store it on the nodes.
+\subsection{Static Sky Analysis}
+The IPP is responsible for performing object detection and analysis on
+the static sky.  This analysis is performed continuously (every day or
+week) on those portions of the sky within 15\degree\ of the sun.  In
+this analysis, the object measurement is much more detailed than those
+performed in the real-time analysis.  The currently envisioned
+parameters to be measured for every object are listed in
+Table~\ref{APdetections}.  The parameters include the object centroid
+and the position covarience matrix, the instrumental PSF magnitude and
+error, local background level and error, a measurement of the
+star-galaxy separation, a measurement of the object shape ($\sigma_x,
+\sigma_y, \theta$), the Petrosian radius, magnitude, axis ratio, and
+angle; the S\'ersic radius, magnitude, axis ratio, angle, and
+parameter $\nu$, and a collection of annular aperture flux
+measurements, all of which are also measured for the P4$\Sigma$
+images.  In addition, the galaxy-shape parameters $Gamma_1 \&
+\Gamma_2$, along with the complete `polarization' terms are measured,
+as well as a collection of annular aperture flux and variance
+measurements.  Another unique feature of the static sky analysis is
+that the object detection may be performed simultaneously on all
+filters, in which case the locations and other parameters may be more
+strongly constrained by simultaneously fitting between all bands.  The
+analysis to be performed may be substantially more complex than the
+real-time analysis because of the relatively low data rate.  Instead
+of needing to process thousands of images per night ($\sim 350$Mpix
+per second), it is only necessary to process the complete sky in a
+year, or an average rate of $\sim$2 Mpix per second, or $< 1$\% of the
+object analysis in the other analysis stages.
 \subsection{AstroRef: Astrometric Reference Catalog creation}
+\tbd{needs to be fleshed out substantially}
 This processing stage shall use many observations over a given time
 …
 \subsection{PhotoRef: Photometric Reference Catalog creation}
+\tbd{needs to be fleshed out substantially}
 This processing stage shall use many observations over a given time
 period to fit a consistent global photometric solution, resulting in a
 …
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\section{System Design : Miscellaneous Tasks}
-In this section, we discuss the design of the science analysis stages
-which perform the fundamental image analysis steps of the IPP.
-\subsection{Retrieval}
-The retrieval stages simply retrieve pixel data from an external
-source (ordinarily OATS at the Summit, but it could conceivably be
-some other external source) and store it on the nodes.
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 \section{Software Hierarchy}
 …
 flexibility, the IPP will be built in a layered fashion.  The lowest
 level functions will be written in C and collected together into a
+Pan-STARRS library.  These library functions will be used to write more
+complex modules.  The modules will be written in C but will make use
+of the SWIG tool to make their functionality available within other
+frameworks.  In particular, the modules can be tied together with a
+simple framework (an `engine') or with detailed flow-control through
+the use of a high-level language such as Perl, Python, or Tcl
+employing the SWIG interfaces.  For the high-level functions in the
+operational system, the IPP will make use of \tbd{Python} as the
+scripting language to provide the required flow-control to tie the
+modules together.
+Pan-STARRS library.  These library functions will be used to write
+more complex modules.  The modules will be written in C but will make
+use of SWIG to make their functionality available within other
+languages.  In particular, the modules can be tied together with a
+program in C or through the use of a high-level language such as Perl,
+Python, or Tcl employing the SWIG interfaces.  For the high-level
+functions in the operational system, the IPP will make use of
+\tbd{Python} as the scripting language to provide the required
+flow-control to tie the modules together.
 This approach satisfies the requirement that complicated low-level
 …
 \subsection{Internal Interfaces}
+\tbd{To be updated and expanded.}
+Internal interfaces consist of queries to the IMD or IPS, insertion of
+data in the IMD, IPS, or IOD, or processing configuration files.  The
+science and calibration image processing pipelines make requests for
+images from the IPS, metadata from the IMD, and push their results
+back onto the IPS and IMD.  The reference catalog pipelines make
+requests on the IMD and the IOD and push their results back to the
+IOD.  The scheduler creates input processing configuration files for
+the processing pipelines and queries the IMD and IPS and pushes
+results back to the IIS.
+FITS Images
+FITS Tables
+XML
+SQL queries
+C:DB interactions
+Internal interfaces consist of interactions between the analysis
+scripts and the IPP Metadata Database, Image Server or AP Database.
+There are also interfaces between the IPP Scheduler, Controller, and
+the Metadata Database.
+The science and calibration image processing pipelines make requests
+for images from the Image Server, metadata from the Metadata Database,
+and push their results back onto the Image Server and Metadata
+Database.  The Scheduler specifies analysis tasks and sends them to
+the Controller, and determines the next action based on the contents
+of the Metadata Database.  The various subsystems specify the API for
+client / server interactions with them.  Commands will be sent using
+either text-based commands, SOAP or an equivalent protocol.  The
+format of the exchanged data may be in one of several forms discussed
+below.
+FITS Images will be used to transport images between the components of
+the IPP.  Non-standard FITS images representing triangular collections
+of pixels may be used to store the static sky images.
+FITS Tables will be used to store and transport tabular data,
+especially large queries from database subsystems.  The Autocoding
+technique discussed in Appendix~\ref{Autocode} is used to define many
+different table interactions.
+XML files will be used to store and transport data which is not
+well-suited to the rectangular form of FITS Tables.  Hierarchical data
+concepts and variable-length structures fall in this class.  Examples
+include mosaic astrometry description information and configuration
+information.
+SQL queries and C wrappers of SQL queries will be used as the direct
+interface to the databases.
 \subsection{External Interfaces}
-\tbd{This whole section to be updated.}
 This subsection describes the interfaces between the IPP and other
 Pan-STARRS systems and the external clients.  The interfaces are
+illustrated in Figure~\ref{fig:functionalities}.  Incoming data is
+received by either the IPS (pixels), the IMD (metadata), or the IOD
+(objects).  Requests for data by external clients are also made to
+these three databases.  Requests for data made by the IPP are
+generated by the IPP Scheduler or the science processing pipelines.
+illustrated in Figure~\ref{overview}.
+Incoming data is received by
+either the IPS (pixels), the IMD (metadata), or the IOD (objects).
+Requests for data by external clients are also made to these three
+databases.  Requests for data made by the IPP are generated by the IPP
+Scheduler or the science processing pipelines.
+\subsubsection{OTIS}
+The IPP Scheduler may query OTIS for a list of new images and
+metadata.  The locations of those images in the Summit Pixel Server is
+sent back as a table, and all metadata may be sent to the IPP as a
+collection of FITS Tables.  The IPP also may send quality assessment
+information for each FPA and major frame by writing out FITS tables
+and notifying OTIS of the presence of the new tables.
 \subsubsection{Camera}
+\subsubsection{OTIS}
+The Summit Pixel Server (SPS) sends raw image data, image metadata,
+and enviromental metadata to the IPP.  The IPP provides an interface
+mechanism by which the SPS can register new images with the IPP, which
+sends them to the appropiate subsystem: The image pixel data is sent
+to the IPS while the metadata is sent to the IMD.
+The Pan-STARRS Telescope Scheduler (PTS) sends information about the
+telescope schedule to the IPP: observing plan for the night, or longer
+time scales.  The IPP scheduler sends telescope schedule requests to
+the PTS (i.e.\ calibration needs).
+Images are pulled from the Summit Pixel Server, part of the Camera
+team's purview.  The locations of the files are sent by OTIS.  IPP may
+grab these via {\tt http} commands or via {\tt NFS} or another network
+file exchange protocol.  The IPP notifies OTIS (and Camera) when each
+image has been received.
 \subsubsection{PSPS}
+The Static Image Server provides segments of the current static sky
+image to the IPP on demand.  IPP subsystems which require this data
+will block until it is available or timeout if it is not.  The IPP
+provides updated static sky images to the SIS when available.
+The details of the transfer mechanism have \tbd{not been worked out}.
+The data to be transfered include:
+\begin{itemize}
+\item Static Sky images
+\item Postage Stamps
+\item Metadata tables
+\item Detections \& Object associations.
+\end{itemize}
 \subsubsection{MOPS}
+The Moving Object Processing System interfaces with the IPP to receive
+the objects detected in the difference images via queries to the IOD.
+The MOPS may interface with the IMD as needed.
+The details of the transfer mechanism have \tbd{not been worked out}.
+The data to be transfered include:
+\begin{itemize}
+\item Image Metadata tables
+\item Orphaned Detections
+\end{itemize}
 \subsubsection{Other Preferred Client Science Pipelines}
+The client science pipelines may interface with the IPP via requests
+for data from the IMD, IOD, or IPS.  \tbd{how many clients max? / how
+much data?}
+The details of the transfer mechanism have \tbd{not been worked out}.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 …
 \subsection{Software Runtime Configuration Issues}
+\label{RuntimeConfig}
 The IPP Software requires extensive runtime configuration information.
 …
 \end{document}
-\section{Notes}
-\subsection{Cell vs Chip vs FPA vs Major Frame}
-There are several levels of input data pixel groups: Cell, Chip, Focal
-Plane Array (FPA), and Major Frame.  It is necessary to make the
-association between the data of one level and that of the next in a
-way that is reliable and robust to missing elements.  If a specific
-cell is missing from a chip, that information is known by the
-controller an needs to be represented in the metadata.  Similarly if a
-chip is missing from a mosaic camera, that information is also known
-and must be carried though the metadata.  A more difficult association
-is that between the telescopes to define the major frame.  Some
-possibilities:
-\begin{itemize}
-\item exposures in a major frame are always synchronized; the
-telescopes are required to take exposures in a coordinated fashion and
-these linked exposures are identified as being part of a specific
-major frame by the TCS or PTS.
-\item exposures may be taken in a coordinated fashion, and identified
-by the TCS or PTS as part of a specific major frame, but not all
-exposures are required to be taken in this fashion.  Independent
-images are handled by the IPP differently (Phase 3 and Phase 4 are not
-appropriate, some varient is required).
-\item exposure links are defined more generally on the basis of the
-resulting image metadata.  The telescopes may have images requested
-at the same coordinates and time, and are defined as a major frame on
-the basis of the observed time and coordinates.  The TCS or PTS might
-not be the entity which defines these major-frame associations; this
-may be the role of some component in the IPP.  Different types of
-major frames may be defined depending on the correlation period in
-time or space.  For example, a major frame in which the telescopes are
-pointing at the same position in the sky to within a few pixels and
-with exposures taken within a second can be treated with more special
-assumptions (minimal differential distortion; moving objects
-coincident) than a major frame in which the offsets are larger in
-either dimension.
-\end{itemize}
-A decisions between these possibilities will drive some requirements
-either on the IPP side or on the PTS/TCS side.
-\subsection{Identifying ghosts, spikes, etc}
-One of the functions currenly defined for Phase 1 is the prediction of
-the location of the bright star spikes, ghost images, and regions of
-complex astronomical background.  Elsewhere in the IPP, these
-identifications are used to excise or mark image pixels.  How these
-regions are defined and saved are is not very clear.  I propose that
-we use the mask image to mark as bit-flags all of these cosmetic pixel
-flagging issues.  If we need to save this information, for the short
-period that the input science images are kept, then it is only a small
-addition of data.
-\subsection{Pending Sky-cell / Detector table}
-Define a pending sky-cell / detector table to define the overlaps and to
-give something which the scheduler can query to decide when to
-initiate phase 4.

Note: See TracChangeset for help on using the changeset viewer.

Context Navigation

Changeset 2176

Legend:

trunk/doc/design/ippSDRS.tex

Download in other formats: