Changeset 2504 for trunk/doc/modules/ModulesSDRS.tex

Timestamp:

Nov 26, 2004, 3:48:49 PM (22 years ago)

Author:

eugene

Message:

added header / camera / config modules

File:

• trunk/doc/modules/ModulesSDRS.tex (modified) (8 diffs)

trunk/doc/modules/ModulesSDRS.tex

@@ -1 @@
-%%% $Id: ModulesSDRS.tex,v 1.21 2004-11-11 04:01:43 price Exp $
+%%% $Id: ModulesSDRS.tex,v 1.22 2004-11-27 01:48:49 eugene Exp $
 \documentclass[panstarrs]{panstarrs}
 
 % basic document variables
-\title{Pan-STARRS Image Processing Pipeline Modules}
+\title{Pan-STARRS PS-1 Image Processing Pipeline Modules}
 \subtitle{Supplementary Design Requirements}
 \shorttitle{Modules SDRS}
@@ -11 +11 @@
 \project{Pan-STARRS Image Processing Pipeline}
 \organization{Institute for Astronomy}
-\version{01}
+\version{02}
 \docnumber{PSDC-430-012}
 
@@ -34 +34 @@
 
 \DocumentsInternal
-PSCD-130-001  &   PS-1 Design Reference Mission \\ \hline
-PSCD-430-004  &   Pan-STARRS IPP C Code Conventions \\ \hline
-PSCD-430-005  &   Pan-STARRS IPP SRS \\ \hline
-PSCD-430-006  &   Pan-STARRS IPP ADD \\ \hline
-PSCD-430-008  &   Pan-STARRS IPP Architecture SDR \\ 
+PSCD-230-001  &   PS-1 Design Reference Mission \\ \hline
+PSDC-430-004  &   Pan-STARRS PS-1 IPP C Code Conventions \\ \hline
+PSDC-430-005  &   Pan-STARRS PS-1 IPP Software Requirements Specification \\ \hline
+PSDC-430-006  &   Pan-STARRS PS-1 IPP Algorithm Design Document \\ \hline
+PSDC-430-011  &   Pan-STARRS PS-1 IPP System/Subsystem Design Description \\ \hline
 \DocumentsExternal
 Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\
@@ -52 +52 @@
 
 This document describes the Pan-STARRS Image Processing Pipeline (IPP)
-image processing modules.  The modules use the functionality of the
-\PS{} Library (PSLib) to perform more complex tasks.  They will
-eventually be tied together, employing a script written in a
-high-level language to form a ``processing stage''.  Modules will be
-constructed to support each of the required processing stages, and are
-listed in this document according to the particular processing stage
-for which they will primarily be used.
+data analysis Modules.  The Modules use the functionality of the
+Pan-STARRS Library (PSLib) to perform more complex tasks, especially
+tasks which require assumptions of astronomical analysis or the data
+organization.  Within the IPP, the Modules are tied together into
+programs which perform complete data analysis tasks (an ``analysis
+stage'').  The modules may be tied together within a C framework or
+using a high-level scripting language.  Bindings of the Modules are
+made available to the scripting language using the program SWIG.
 
 In order to preserve name space, globally-visible structures and
-functions shall be prefixed with \code{pm}, for ``\PS{} Modules''.
-
+functions shall be prefixed with \code{pm}, for ``Pan-STARRS
+Modules''.
+
+\section{Runtime Configuration Data}
+
+PSLib defines a \code{psMetadata} structure which can carry labeled
+data of arbirtary types.  The associated functions implemented by
+PSLib consist of tools to manipulate and extract data from
+\code{psMetadata} collections.  Within PSLib, the \code{psMetadata}
+stucture is used to carry the data from a FITS header.  In addition,
+functions are available to fill a \code{psMetadata} collection from a
+text-based configuration file using a particular syntax, and to fill a
+\code{psMetadata} collection from a properly formatted XML document.  
+
+In the IPP Modules, we use \code{psMetadata} collections to carry
+run-time configuration data used by the data analysis modules.  Below,
+in the discussion of the various modules, this configuration
+information is defined by specifying the name of the data item of
+interest, the conceptual meaning of that data item, and the allowed
+values for the data item.  In this section, we discuss top-level
+concepts related to the configuration information, including the
+sources of the run-time configuration data and special
+operations used to extract information from the configuration system.
+
+\subsection{Configuration Data Sources}
+
+All modules need to load some configuration information defining
+parameters which may be configured at run-time.  Some module require
+only a basic amount of information, while others require configuration
+information defining the camera or details of an analysis (the {\it
+recipe}).  We define three utility functions to load these files from
+standard locations, with optional command-line override values.
+
+\begin{verbatim}
+bool pmConfigLoadSite   (psMetadata **site, int *argc, char **argv);
+bool pmConfigLoadCamera (psMetadata **camera, psMetadata *site, psMetadata *header, int *argc, char **argv);
+bool pmConfigLoadRecipe (psMetadata **recipe, psMetadata *camera, int *argc, char **argv, char *script);
+\end{verbatim}
+
+\code{pmConfigLoadSite} loads basic configuration information from a
+general configuration file.  By default, this file has the name
+\code{~/.ipprc} in the user's home directory.  This file may be
+overridden on the command line with the optional flag \code{-site
+(file)}.
+
+% eventually, we should define the complete sequence:
+% $FOO, ~/.foorc, /prefix/config/foorc
+
+\code{pmConfigLoadCamera} loads camera configuration information from
+a file which depends on the camera of interest.  In some cases, the
+camera may be determined from the header of an image currently in
+hand.  In other cases, the camera must be specified without reference
+to a file.  This function attempts to determine the appropriate camera
+configuration file by examining, in order, the command-line options,
+the header data, and the site configuration data.  The camera config
+file may be specified on the command line with the option
+\code{-camera (file)}.  If this does not exist, the
+\code{pmConfigLoadCamera} examines the provided header to determine
+the camera by using the function \code{pmCameraFromHeader} (defined
+below).  If this fails, the function attempts to determine the camera
+from the provided site config data, looking for the keyword
+\code{CAMERA} in the site config data.  The camera name is used to
+select the camera configuration file, defined in the site
+configuration file with the keyword \code{CAMERA.name}, where
+\code{name} is substituted with the name of the camera.
+
+\code{pmConfigLoadRecipe} loads the recipe configuration information
+from a file which depends on the camera of interest and the particular
+analysis task.  The camera configuration file must define recipe files
+for each of the analysis tasks of interest.  The function attempts to
+load the recipe file using the keyword \code{RECIPE.name}, where
+\code{name} is substituted with the name of the analysis task, for
+example \code{PHASE2}.  The choice of the recipe file may be
+overridden on the command line with the option \code{-recipe (file)}.
+
+\subsection{Indirect Configuration Data}
+
+Some configuration data is not provided directly by the configuration
+files.  Instead, the configuration system provides a mechanism to
+define indirect references to configuration data.  Three indirect
+configuration data sources are currently defined: data from a FITS
+header, data from the Metadata Database, and data from the Pan-STARRS
+Status Server.  The configuration data may specify that a certain
+value is provided by one of these mechanisms.  
+
+In order to specify that the configuration data should be derived from
+a FITS header, the data value is given in the form: \code{HD:KEYWORD},
+where \code{KEYWORD} specifies the keyword to be used in search the
+FITS header.  
+
+If the data is to be extracted from the Metadata Database, the data
+value is given in the form: \code{MD:TABLE,FIELD,KEY}.  In this case,
+\code{TABLE} specifies the Metadata Database table to be used to find
+the value, \code{FIELD} specifies the field in that table to be used,
+and \code{KEY} specifies the value of the primary key which gives the
+value of interest.
+
+If the data is to be extracted from the Status Server, the data value
+is given in the form: \code{SS:PATH}.  In this case, \code{PATH}
+specifies the Status Server path to the data value of interest.  
+
+{\bf \it note that the Metadata Database and Status Server
+  interactions require us to define the APIs for these interactions.
+  These two methods are not available until the MD and SS APIs are
+  defined.}
+
+As an example, consider the definition of the data region of an image.
+This value is frequenty represented in the FITS header with the
+keyword \code{DATASEC}.  However, in some cameras, this value may not
+be defined, or another value may be used.  We must be able to flexibly
+define both the appropriate keyword, or the actual value, if known.
+These two cases may be written as follows, using the configuration
+file format parsed by \code{psMetadataParseConfig}:
+\begin{verbatim}
+DATA.REGION  STR  [1:100,2:400]
+DATA.REGION  STR  HD:DATASEC
+\end{verbatim}
+In the first case, the data provided in the configuration file is the
+data of interest.  (\code{[1:100,2:400]}).  In the second case, the
+three-character code \code{HD:} specifies that the value of interest
+may be determined by searching for the value associated with the
+header keyword \code{DATASEC}.
+
+We provide utilities functions to extract this type of potentially
+indirect configuration information.  We provide four functions
+returning four data types.  The first three return the value from the
+appropriate location assuming the data type to be \code{char*},
+\code{psS32}, and \code{psF64}.  The fourth version searches for a
+string which is passed to the function \code{psRegionFromString}
+before being returned as a \code{psRegion} pointer.
+
+\begin{verbatim}
+psSTR     pmConfigLookupSTR    (psMetadata *config, psMetadata *header, char *name);
+psS32     pmConfigLookupS32    (psMetadata *config, psMetadata *header, char *name);
+psF64     pmConfigLookupF64    (psMetadata *config, psMetadata *header, char *name);
+psRegion *pmConfigLookupRegion (psMetadata *config, psMetadata *header, char *name);
+\end{verbatim}
+
+\section{Camera Data Organization \& Camera Geometry}
+
+We require several utility functions to define the geometry of the
+detectors in a camera and to specify the organization of the camera
+data in real FITS images.  The camera data organization is defined in
+a set of \code{psMetadata} structures, and may be stored on disk in
+the file format parsed by \code{psMetadataParseConfig}.
+
+PSLib defines a hierarchy of data structures related to the
+organization of the pixels in a camera.  These structures, starting
+from the top level, follow the sequence \code{psFPA}, \code{psChip},
+\code{psCell}, \code{psReadout}, \code{psImage}.  Each lower level
+structure is carried as an array in the higher level.  The containers
+as defined by PSLib include functions which specify the astrometric
+relationships between these levels, and provide a single
+\code{psMetadata} container pointer for each level.  In this section,
+we define the type of metadata is stored in these metadata containers
+and how the image headers are parse to define the data hierarchy in
+memory.  
+
+\subsection{Camera Config Data \& the {\tt psFPA}}
+
+A relevant collection of metadata is the externally supplied camera
+definition metadata loaded with the function
+\code{pmConfigLoadCamera}.  This metadata defines the expected layout
+of a specific camera along with information detailing how to interpret
+the headers for an image from that camera.  The camera metadata
+consists of keyword / value pairs which are relevant to the entire
+camera, and a set of metadata collections specified for each cell.
+This top-level metadata is added to the \code{psFPA} with the name
+'CAMERA.LAYOUT'.  For example:
+\begin{verbatim}
+fpa->metadata = psMetadataAdd (fpa->metadata, PS_LIST_TAIL, "CAMERA", PS_META_META, "camera", camera);
+\end{verbatim}
+The specific cell-level entries are also placed on the correpsonding
+\code{psCell} elements.  These entries are added to the metadata with
+the name 'CELL':
+\begin{verbatim}
+cell->metadata = psMetadataAdd (cell->metadata, PS_LIST_TAIL, "CELL", PS_META_META, "cell", cellMD);
+\end{verbatim}
+
+\subsection{FITS File Data Representations}
+
+Within the FITS data representation, there are various choices which
+can and have been made for the placement of the pixels in a data file.
+In the simplest case, the camera consists of a single chip consisting
+of a single cell always read with a single readout.  In this case, the
+image data could be written as part of the primary data unit.  In a
+more complex case with multiple chips and multiple cells, the data may
+be organized in various ways.  The data may be distributed into
+multiple files or in multiple FITS data extensions.  A single camera
+image may be written as a collection of files for individual chips
+with separate extensions for each cell (CFH12K.split, GPC).  Another
+camera may write a single file with multiple extensions for each cell
+(Megacam.raw), or multiple extensions per chip, which each cell
+representing portions of the chip image (Megacam.splice, CFHT-IR).
+
+In all of these representations, there are only two principal
+distinctions in how the pixel data is stored.  Either a single data
+block (a single FITS image) represents only a single cell, or it
+represents a collection of cells.  This affects the way in which an
+image is read in, the way the header is associated with a level of the
+data hierarchy, and the way in which the header keywords are
+interpretted to define the layout of the pixels in the cell.  We
+distinguish these two cases as 'cell-based' and 'chip-based' layout.
+
+The FITS headers are attached to the \code{psFPA} data hierarchy by
+adding them to the \code{psMetadata} entries for the appropriate data
+level.  Each of the data levels \code{psFPA}, \code{psChip}, and
+\code{psCell} have \code{metadata} elements to store relevant metadata
+of any type.  A FITS header metadata collection may be added to one of
+these metadata collections using the metadata name 'HEADER':
+\begin{verbatim}
+fpa->metadata = psMetadataAdd (fpa->metadata, PS_LIST_TAIL, "HEADER", PS_META_META, "header", header);
+\end{verbatim}
+
+In 'cell-based' layout, the headers are attached to the corresponding
+\code{psCell} element.  In 'chip-based' layout, the headers are
+attached to the \code{psChip} element.  This difference is natural
+since a 'cell-based' data file has one FITS extension, and therefore
+one FITS header, for each \code{cell}.  Conversely, in a 'chip-based'
+data file, there is one extension per chip, and therefore multiple
+cells for each FITS header, but only one chip for each FITS header.  
+
+\subsection{Determine the Camera from the Primary Header}
+
+\begin{verbatim}
+char *pmCameraFromHeader (psFitsHeader *header, psMetadata *rules);
+\end{verbatim}
+
+This function examines a primary header unit (\code{header}) and
+determines the camera which provided the data.  The rules which
+identify the camera are defined as a \code{psMetadata} structure
+containing one element for each known camera.  These are generally to
+be provided as part of the site configuration information.  The
+elements of this \code{psMetadata} container are in turn
+\code{psMetadata} containers consisting of the expected header
+keywords and their required values.  The function
+\code{pmCameraFromHeader} tests each of the defined cameras in
+succession.  For each camera, it searches for each of the given
+keywords and compares the value with the value expected.  If the
+keyword does not exist, or if the keyword has the wrong value, the
+camera is rejected.  The name of first camera which matches the header
+is returned as an allocated string.  If no match is found, the
+returned value must be \code{UNKNOWN}.  This condition may be treated
+by calling functions as either an error or only a warning.  If an
+error is encountered in parsing the metadata containers, then
+\code{NULL} is returned.  An example of the metadata keyword / value
+pairs is given below (note that this uses an undefined metadata config
+file representation \code{METADATA}):
+
+\begin{verbatim}
+CFH12K.MEF      METADATA
+  TELESCOP      STR   CFHT
+  INSTRUME      STR   CFH12K
+  EXTEND        BOOL  T
+  NEXTEND       S32   12
+END
+
+CFH12K.SPLIT    METADATA
+  TELESCOP      STR   CFHT
+  INSTRUME      STR   CFH12K
+  EXTEND        BOOL  F
+END
+\end{verbatim}
+
+\begin{figure}
+\begin{center}
+\psfig{file=CameraHierarchy,width=5.0in}
+\caption{Camera Data and Metadata Hierarchy\label{CameraHierarchy}}
+\end{center}
+\end{figure}
+
+\subsection{Validate the Header set \& Construct the FPA}
+
+\begin{verbatim}
+bool  pmCameraValidateHeaders (psMetadata *headers, psMetadata *camera);
+psFPA *pmFPAfromHeader (psMetadata *headers, psMetadata *camera);
+\end{verbatim}
+
+These funtions examine the collection of \code{headers} and compare
+them with the camera definitions in the metadata structure
+\code{camera}.  One function (\code{pmFPAfromHeader}) uses the header
+set and camera definition file to construct a \code{psFPA} with all
+\code{psChip} and \code{psCell} entries allocated, but without any
+\code{psReadout} or pixel data.  The other function validates the
+header set against the camera definition, but does not actually
+construct the \code{psFPA}.  
+
+The camera definition metadata collection defines the relationship
+between chips, cells, and the FITS extensions.  An example of the data
+needed by \code{pmCameraValidateHeaders} and \code{pmFPAfromHeader} is
+given below for several types of cameras in the form of a metadata
+config file.  
+
+\begin{verbatim}
+MEGACAM.RAW     METADATA    
+  NCELL         S32    72
+  CELL.FMT      STR    CELL.%02d
+  EXT.TYPE      STR    CELL
+  EXT.KEY       STR    EXTNAME
+  PHU           STR    FPA
+  #
+  #                    EXT.KEY  CHIP  
+  CELL.00       CELL   amp00    CHIP.00
+  CELL.01       CELL   amp01    CHIP.00
+  CELL.02       CELL   amp02    CHIP.01
+  CELL.03       CELL   amp03    CHIP.01
+  ...
+END
+
+MEGACAM.SPLICE  METADATA      
+  NCELL         S32    36
+  CELL.FMT      STR    CELL.%02d
+  EXT.TYPE      STR    CHIP
+  EXT.KEY       STR    EXTNAME
+  PHU           STR    FPA
+  #
+  #                    EXT.KEY  CHIP  
+  CELL.00       CELL   ccd00    CHIP.00
+  CELL.01       CELL   ccd00    CHIP.00
+  CELL.02       CELL   ccd01    CHIP.01
+  CELL.03       CELL   ccd01    CHIP.01
+  ...
+END
+
+CFH12K.SPLIT    METADATA
+  NCELL         S32    12
+  CELL.FMT      STR    CELL.%02d
+  EXT.TYPE      STR    CELL
+  EXT.KEY       STR    EXTNAME
+  PHU           STR    NONE
+  #
+  #                    EXT.KEY  CHIP   
+  CELL.00       CELL   chip00   CHIP.00
+  CELL.01       CELL   chip01   CHIP.01
+  CELL.02       CELL   chip02   CHIP.02
+  ...
+END
+
+GPC.RAW         METADATA    
+  NCELL         S32    4096
+  CELL.FMT      STR    CELL.%04d
+  EXT.TYPE      STR    CELL
+  EXT.KEY       STR    EXTNAME
+  PHU           STR    CHIP
+  #
+  #                    EXT.KEY  CHIP  
+  CELL.0000     CELL   amp00    CHIP.00
+  CELL.0001     CELL   amp01    CHIP.00
+  CELL.0002     CELL   amp02    CHIP.00
+  CELL.0003     CELL   amp03    CHIP.00
+  ...
+  CELL.0065     CELL   amp03    CHIP.01
+  CELL.0066     CELL   amp03    CHIP.01
+  ...
+END
+\end{verbatim}
+
+Among the important elements of the camera definition information are:
+\begin{itemize}
+\item \code{NCELL} - this defines the possible number of cells from this
+  camera.
+\item \code{CELL.FMT} - this defines the format of the \code{CELL.nn}
+  keywords below in the metadata collection, one for each of the
+  \code{NCELL} values
+\item \code{EXT.TYPE} - this defines the lowest data level
+  corresponding to a single FITS extension: \code{CELL} or
+  \code{CHIP}.  In some cases, a single extension represents a chip,
+  and is subdivided into cells by header keywords specifying certain
+  regions.  In other cases, a single extension represents only the
+  data from a single amplifier, ie, a cell.  These two cases require
+  somewhat different handling.
+\item \code{EXT.KEY} - this entry defines a header keyword which
+  allows the unique identification of a given header with one of the
+  cell or chip entries (depending on the value of \code{EXT.TYPE}).
+\item \code{PHU} - this entry defines the meaning of the FITS file
+  primary header unit. 
+\end{itemize}
+In addition to these generic parameters, the camera definition
+metadata includes an entry for each cell giving specific values
+required to define that cell.  
+
+Both functions \code{pmCameraValidateHeaders} and
+\code{pmFPAfromHeader} must start with the set of headers and attempt
+to identify the corresponding cell or chip.  All available cells and
+chips must be identified, and the cells corresponding to each chip
+must be tracked.  In the case of \code{pmCameraValidateHeaders}, the
+function must only verify the headers contain valid extensions,
+without allocating the correpsonding \code{psFPA}, while
+\code{pmFPAfromHeader} must also allocate the \code{psFPA}, the
+contained \code{psChip} arrays, and the correct number of
+\code{psCell} arrays for each \code{psChip}.
+
+These functions also add to the metadata containers for the chips and
+cells, following the rules discussed above. Each \code{psCell} entry
+should have the \code{CELL} metadata lines from above attached to the
+\code{psCell.metadata} element as an additional metadata collection
+with the name \code{CELL.LAYOUT}.  The complete camera metadata
+collection (including the cells), are attached to the
+\code{psFPA.metadata} element with the name \code{CAMERA.LAYOUT}.  The
+image header data are also attached to metadata entries, at a level
+which depends on the value of \code{EXT.TYPE} in the camera layout
+data above.  If the value is \code{CELL}, the header metadata
+collections are attached to the \code{psCell.metadata} elements with
+the name \code{HEADER}.  If the value of \code{EXT.TYPE} is chip, the
+header metadata is attached to the \code{psChip.metadata} element,
+again with the name \code{HEADER}.  
+
+An image file may also have a primary header unit which is not
+associated with a data block.  This metadata block must also be added
+to the data heirarchy so successive operations may update the metadata
+as needed.  The disposition of the primary header unit is defined by
+the camera layout key \code{PHU}.  If this has a value of \code{NONE},
+there is no primary header unit, or it should be ignored.  If the
+value is \code{FPR}, the primary header data is attached to the
+\code{psFPA.metadata} element with the value PHU.  Alternatively, if
+it has the value \code{CHIP}, then the primary header metadata is
+attached to the \code{psChip.metadata} element.
+
+\subsection{Coordinate Transforms and Header Data}
+\tbd{the algorithms for three functions in this section are not
+  well-defined.  do not code yet}.
+
+Astrometric and geometric information about an image from a camera may
+be represented in a variety of ways.  A crude representation of the
+pixel geometry is specified in many image headers using the IRAF-style
+region keywords \code{DATASEC}, \code{DETSEC}, etc.  These keywords
+are used to define the location of a single image's pixels in the
+frame of the full mosaic of detectors in the assumption that the
+mosaic can be represented as a single uniform grid of pixels.  An
+alternative set of keywords have been used in cases where multiple
+cells are saved together in a single FITS image extension.  More
+sophisticated astrometric representations require elements to define
+projections, scaling, distortion, etc.  Several versions of header
+keywords have been used to represent these astrometric
+transformations.  In this section, we define three functions to
+interpret a collection of image headers and construct the appropriate
+offset and/or astrometry parameters.
+
+\begin{verbatim}
+bool  pmFPADefineOffsets (psFPA fpa);
+bool  pmFPADefineWCS (psFPA fpa);
+bool  pmFPADefineWCSfromOffsets (psFPA fpa);
+\end{verbatim}
+
+The first function takes a \code{psFPA} structure which has been
+populated with header and camera configuration metadata in the
+appropriate locations as discussed above.  Using the information in
+the camera config metadata and the headers, the function sets the
+values for the elements \code{psChip.col0,row0},
+\code{psCell.col0,row0}, \code{psReadout.col0,row0},
+\code{psReadout.colParity,rowParity}, and
+\code{psReadout.colBinning,rowBinning}.  This information is
+determined by examining the regions defined by the following names.
+
+% how do we handle the CCDSUM keyword case?
+\begin{verbatim}
+psCell.metadata:CELL:CCDBIN1 $\rightarrow$ psReadout.colBins
+psCell.metadata:CELL:CCDBIN2 $\rightarrow$ psReadout.rowBins
+
+DETSEC from psCell.metadata:CELL:DETSEC
+CCDSEC from psCell.metadata:CELL:CCDSEC
+DATASEC from psCell.metadata:CELL:DATASEC
+
+if (DETSEC.x0 > DETSEC.x1) then psReadout.colParity = -1
+if (DETSEC.y0 > DETSEC.y1) then psReadout.rowParity = -1
+\end{verbatim}
+
+The second function examines the contents of the headers of the chips
+and cells and constructs the collection of astrometric coordinate
+transformations.. 
+
+The third function uses the offset information and the basic telescope
+pointing information to construct a approximate guess at the
+astrometric coefficients based on the detector geometry.
+
+\begin{figure}
+\begin{center}
+\psfig{file=CameraLayout,width=5.5in}
+\caption{Camera Pixel Layout\label{CameraLayout}}
+\end{center}
+\end{figure}
+
+\begin{figure}
+\begin{center}
+\psfig{file=CameraRegionKeywords.ps,width=5.5in}
+\caption{Camera Region Keyword Definitions\label{CameraRegionKeywords}}
+\end{center}
+\end{figure}
+
+\subsection{Chip \& Cell from FITS File}
+\tbd{the algorithm for the function in this section is not
+  well-defined.  do not code yet}.
+
+When loading data from disk, it is may be necessary to use the
+information in the DATASEC and DETSEC entries to determine which part
+of the image should be read.  If the data is stored in a chip-based
+format, then the data for each cell corresponds to only a fraction of
+the pixels stored in a single image extension.  We specify the
+following function to perform the correct read of data from a FITS
+file into the corresponding \code{psCell} entry respecting the
+boundaries of the cells within chip-based images.
+
+\begin{verbatim}
+psReadout *pmReadoutLoad (psReadout *input, psFits *f, psCell *cell, int plane);
+\end{verbatim}
 
 \section{Phase 2}
@@ -76 +580 @@
 \item Flat-field;
 \item Mask bad pixels;
-\item \tbd{Subtract the background;}
+\item Subtract the background;
 \item \tbd{Mask cosmic rays;}
 \item \tbd{Mask optical defects;}
@@ -358 +862 @@
 image), the function shall generate an error.
 
-
-
-\section{Calibration}
-
-The calibration module essentially consists of combining multiple
-images of a particular type in order to build up signal-to-noise.  For
-this, we require a general purpose image combination module.  We
-forsee this module as only acting upon data from the same detector,
-and so each input image will have the same noise characteristics.
-
-\begin{verbatim}
-typedef struct {
-    psStats *stats;                     // Statistics to use in combining pixels
-    unsigned int maskVal,               // Mask pixels where mask & maskVal == 1
-    float fracHigh;                     // Fraction of high pixels to throw
-    float fracLow;                      // Fraction of low pixels to throw
-    int nKeep;                          // Number of pixels to be sure to keep
-} pmCombineParams;
-
-psImage *
-pmReadoutCombine(psImage *output,       // Output image, or NULL
-                 const psList *inputs,  // List of input readouts
-                 pmCombineParams *params, // Combination parameters
-                 const psVector *zero,  // Offsets to apply for each image
-                 const psVector *scale, // Scales to apply for each image
-                 bool applyZeroScale,   // Are zero and scale for application, or only noise properties?
-                 float gain,            // Gain in e/ADU
-                 float readnoise        // Read noise in e
-                 );
-\end{verbatim}
-
-\code{pmReadoutCombine} combines input images pixel by pixel --- for
-each pixel of the output image, a stack of contributing input pixels
-is formed and combined.  Several of its input parameters are lists or
-vectors, and if these are not all of the same length (or \code{NULL}),
-the module shall generate an error and return \code{NULL}.
-
-If the provided \code{output} is \code{NULL}, then the module shall
-allocate a new image of sufficient size for the input images.  If the
-\code{output} image is non-\code{NULL} and is not of sufficient size
-for the combined image, the module shall generate an error and return
-\code{NULL}.
-
-If the \code{inputs} is \code{NULL}, the module shall generate an
-error and return \code{NULL}.  Otherwise, the \code{inputs} shall be a
-list of \code{psReadout}s.  The images contained within the
-\code{psReadout}s need not all be of the same size, but the module
-shall take into account the offsets (\code{col0,row0}) from the corner
-of the detector when comparing pixels, so that it is the same
-\textit{physical} pixels that are combined.
-
-The parameters used in the combination, including how the pixels are
-to be combined, and how the rejection is performed is contained within
-the \code{params}, which may not be \code{NULL} (otherwise the module
-shall generate an error and return \code{NULL}).  We choose to use
-this structure instead of supplying the values separately in order to
-keep down the number of parameters to \code{pmReadoutCombine}; the
-\code{psCombineParams} may be recycled for subsequent calls to
-\code{pmReadoutCombine} since the values are not dependent upon the
-choice of inputs, but merely specify how the combination is to be
-performed.
-
-The particular statistic specified by \code{stats} shall be used to
-combine each stack of pixels from the input images.  Only one of the
-statistics choices may be specified, otherwise the module shall
-generate an error and return \code{NULL}.
-
-If the \code{maskVal} is non-zero, then pixels in the \code{mask} of
-each \code{psReadout} in the \code{inputs} which satisfy the
-\code{maskVal} shall not have the corresponding pixels placed in the
-stack for combination.
-
-After masking, but before performing the combination, the highest
-\code{fracHigh} fraction and lowest \code{fracLow} fraction of pixels
-in the stack are immediately rejected, unless this would leave less
-than \code{nKeep} pixels in the stack, in which case no immediate
-rejection is performed.
-
-If the \code{zero} vector is non-\code{NULL} and \code{applyZeroScale}
-is \code{true}, then the appropriate values shall be added to the
-\code{inputs} before rejection is performed.  If \code{zero} is
-non-\code{NULL} and \code{applyZeroScale} is false, then the values
-shall only be used in calculating the noise.
-
-If the \code{scale} vector is non-\code{NULL} and
-\code{applyZeroScale} is \code{true}, then the appropriate values
-shall multiply the \code{inputs} before rejection is performed.  If
-\code{scale} is non-\code{NULL} and \code{applyZeroScale} is false,
-then the values shall only be used in calculating the noise.
-
-The purpose of \code{applyZeroScale} is to allow combination of fringe
-frames, where the frames have been deliberately sky-subtracted and
-rescaled (to get the fringes amplitudes running from -1 to 1), which
-actions should not be undone when combining, but yet it is desirable
-to provide the \code{zero} and \code{scale} values so that the correct
-noise properties are used in the combination.
-
-If the \code{gain} and \code{readnoise} are positive and non-negative
-(respectively), then these shall be used to provide weights for the
-combination using Poisson statistics.
-
-The \code{inputs, zero} and \code{scale} may be of S16, S32 and F32
-types, and must all be of the same type.  The \code{output} shall be
-of the same type.
-
-\section{Camera Configutation Information}
-
-\tbd{these function are not yet ready for implementation.  The APIs
-  are fine but the algorithms need to be specified.}
-
-We require several utility functions to define the general layout of a
-camera and to relate the camera definition to the header in a given
-camera.  This information may be passed to the modules by using the
-configuration files as parse by \code{psMetadataParseConfig}.  These 
-
-\begin{verbatim}
-char *pmCameraFromHeader (psFitsHeader *header);
-\end{verbatim}
-This function examines a primary header unit (\code{header}) and
-determines the camera which provided the data.  \tbr{the rules for
-  performing this analysis probably depend on the keywords such as
-  INSTRUME and TELESCOP.  Should this function be a script so we can
-  easily add a new camera or should the rule set be defined by
-  configuration information?}.
-
-\begin{verbatim}
-bool  pmCameraValidateHeaders (psHash *headers, psMetadata *camera);
-\end{verbatim}
-This funtion examines the collection of \code{headers} and, based on
-the camera definitions in the metadata \code{camera}, determines if
-the header set is consistent with the specified camera.
-
-\begin{verbatim}
-psFPA *pmFPAfromHeader (psList *header, psMetadata *camera);
-\end{verbatim}
-This funtion uses the collection of \code{headers} and the camera
-definitions in the metadata \code{camera} to construct a complete
-\code{psFPA} structure which represents the data described by the
-collection of \code{headers}.  The resulting \code{FPA} has all
-possible structure elements defined but no space for pixel data should
-be allocated.  
-
-\section{Revision Change Log}
-\input{ChangeLogSDRS.tex}
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-\bibliographystyle{plain}
-\bibliography{panstarrs}
-
-\end{document}
-
 \subsection{Subtract sky}
-
-\tbd{This is a simple sky subtraction routine.  A more complicated
-routine will be specified in the future, following research into the
-best algorithm.}
 
 Given an input image, a polynomial or spline specifying the order of a
 desired fit, a binning factor and statistics to use for the binning,
-along with a clipping level, \code{psPhase2SubtractSky} shall fit and
+along with a clipping level, \code{pmSubtractSky} shall fit and
 subtract a model for the background of the image.  The API shall be
 the following:
 \begin{verbatim}
-psReadout *psPhase2SubtractSky(psReadout *in, void *fitSpec, psFit fit, int binFactor, psStats *stats,
-                               float clipSD);
+psReadout *pmSubtractSky(psReadout *in, void *fitSpec, psFit fit, int binFactor, 
+                         psStats *stats, float clipSD);
 \end{verbatim}
 
 Note that the input image, \code{in}, shall be subtracted in-place.
 The function shall return the subtracted image, and also update the
-polynomial, chebyshev or spline specified by \code{fitSpec}, to hold
+polynomial, Chebyshev or spline specified by \code{fitSpec}, to hold
 the coefficients used in the subtraction.
 
@@ -560 +909 @@
 
 
+\section{Calibration}
+
+The calibration module essentially consists of combining multiple
+images of a particular type in order to build up signal-to-noise.  For
+this, we require a general purpose image combination module.  We
+forsee this module as only acting upon data from the same detector,
+and so each input image will have the same noise characteristics.
+
+\begin{verbatim}
+typedef struct {
+    psStats *stats;                     // Statistics to use in combining pixels
+    unsigned int maskVal,               // Mask pixels where mask & maskVal == 1
+    float fracHigh;                     // Fraction of high pixels to throw
+    float fracLow;                      // Fraction of low pixels to throw
+    int nKeep;                          // Number of pixels to be sure to keep
+} pmCombineParams;
+
+psImage *
+pmReadoutCombine(psImage *output,       // Output image, or NULL
+                 const psList *inputs,  // List of input readouts
+                 pmCombineParams *params, // Combination parameters
+                 const psVector *zero,  // Offsets to apply for each image
+                 const psVector *scale, // Scales to apply for each image
+                 bool applyZeroScale,   // Are zero and scale for application, or only noise properties?
+                 float gain,            // Gain in e/ADU
+                 float readnoise        // Read noise in e
+                 );
+\end{verbatim}
+
+\code{pmReadoutCombine} combines input images pixel by pixel --- for
+each pixel of the output image, a stack of contributing input pixels
+is formed and combined.  Several of its input parameters are lists or
+vectors, and if these are not all of the same length (or \code{NULL}),
+the module shall generate an error and return \code{NULL}.
+
+If the provided \code{output} is \code{NULL}, then the module shall
+allocate a new image of sufficient size for the input images.  If the
+\code{output} image is non-\code{NULL} and is not of sufficient size
+for the combined image, the module shall generate an error and return
+\code{NULL}.
+
+If the \code{inputs} is \code{NULL}, the module shall generate an
+error and return \code{NULL}.  Otherwise, the \code{inputs} shall be a
+list of \code{psReadout}s.  The images contained within the
+\code{psReadout}s need not all be of the same size, but the module
+shall take into account the offsets (\code{col0,row0}) from the corner
+of the detector when comparing pixels, so that it is the same
+\textit{physical} pixels that are combined.
+
+The parameters used in the combination, including how the pixels are
+to be combined, and how the rejection is performed is contained within
+the \code{params}, which may not be \code{NULL} (otherwise the module
+shall generate an error and return \code{NULL}).  We choose to use
+this structure instead of supplying the values separately in order to
+keep down the number of parameters to \code{pmReadoutCombine}; the
+\code{psCombineParams} may be recycled for subsequent calls to
+\code{pmReadoutCombine} since the values are not dependent upon the
+choice of inputs, but merely specify how the combination is to be
+performed.
+
+The particular statistic specified by \code{stats} shall be used to
+combine each stack of pixels from the input images.  Only one of the
+statistics choices may be specified, otherwise the module shall
+generate an error and return \code{NULL}.
+
+If the \code{maskVal} is non-zero, then pixels in the \code{mask} of
+each \code{psReadout} in the \code{inputs} which satisfy the
+\code{maskVal} shall not have the corresponding pixels placed in the
+stack for combination.
+
+After masking, but before performing the combination, the highest
+\code{fracHigh} fraction and lowest \code{fracLow} fraction of pixels
+in the stack are immediately rejected, unless this would leave less
+than \code{nKeep} pixels in the stack, in which case no immediate
+rejection is performed.
+
+If the \code{zero} vector is non-\code{NULL} and \code{applyZeroScale}
+is \code{true}, then the appropriate values shall be added to the
+\code{inputs} before rejection is performed.  If \code{zero} is
+non-\code{NULL} and \code{applyZeroScale} is false, then the values
+shall only be used in calculating the noise.
+
+If the \code{scale} vector is non-\code{NULL} and
+\code{applyZeroScale} is \code{true}, then the appropriate values
+shall multiply the \code{inputs} before rejection is performed.  If
+\code{scale} is non-\code{NULL} and \code{applyZeroScale} is false,
+then the values shall only be used in calculating the noise.
+
+The purpose of \code{applyZeroScale} is to allow combination of fringe
+frames, where the frames have been deliberately sky-subtracted and
+rescaled (to get the fringes amplitudes running from -1 to 1), which
+actions should not be undone when combining, but yet it is desirable
+to provide the \code{zero} and \code{scale} values so that the correct
+noise properties are used in the combination.
+
+If the \code{gain} and \code{readnoise} are positive and non-negative
+(respectively), then these shall be used to provide weights for the
+combination using Poisson statistics.
+
+The \code{inputs, zero} and \code{scale} may be of S16, S32 and F32
+types, and must all be of the same type.  The \code{output} shall be
+of the same type.
+
+\section{Revision Change Log}
+\input{ChangeLogSDRS.tex}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\bibliographystyle{plain}
+\bibliography{panstarrs}
+
+\end{document}
+
 \subsection{Cosmic rays}
 
@@ -705 +1166 @@
 \code{in}.
 
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\section{Read in an FPA}
-
-\TBD{This section on reading in an FPA is all TBD and should not be
-implemented yet.}
-
-Of course, before any image manipulation may occur, the image must be
-read in.  This involves populating the FPA with structures to
-represent the true focal plane.  However, different instruments have
-different focal planes, and the images are written to disk in
-different ways.  We have proposed a configuration file format which
-appears to be able to document the structure of a focal plane and
-specify how the IPP can read the required information from the FITS
-files \tbd{REFERENCE}.
-
-Given an open \code{FILE} pointer to a configuration file,
-\code{config}, and the name of an FPA image, \code{fpaName},
-\code{psFPAGenerate} shall return a \code{psFPA} of the correct
-structure, with all the metadata entries read from the nominated FPA
-image.  Astrometric information shall be initialised to values
-specified by the configuration file.  The API shall be:
-
-\begin{verbatim}
-psFPA *psFPAGenerate(FILE *config, const char *fpaName);
-\end{verbatim}
-
-Once an FPA has been generated, the pixels may all be read in at once,
-or the pixels may be read in a chip, cell or readout at a time.  The
-corresponding APIs are:
-
-\begin{verbatim}
-psFPA *psFPAReadAll(psFPA *in);
-psFPA *psFPAReadChip(psFPA *in, const char *chipName);
-psChip *psChipReadCell(psChip *in, const char *cellName);
-psCell *psCellReadReadout(psCell *in, const char *readoutName);
-\end{verbatim}
-
-Note that reads of a cell or readout do not act upon an FPA, but
-rather a chip and a cell, respectively.  This saves the user from
-specifying the name of the FPA, chip, and cell just to get a readout,
-since this information is already stored in the generated \code{psFPA}
-struct.
-
-\tbd{Writing the output}
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%