Index: trunk/doc/modules/ModulesSDRS.tex =================================================================== --- trunk/doc/modules/ModulesSDRS.tex (revision 2335) +++ trunk/doc/modules/ModulesSDRS.tex (revision 2504) @@ -1,7 +1,7 @@ -%%% $Id: ModulesSDRS.tex,v 1.21 2004-11-11 04:01:43 price Exp $ +%%% $Id: ModulesSDRS.tex,v 1.22 2004-11-27 01:48:49 eugene Exp $ \documentclass[panstarrs]{panstarrs} % basic document variables -\title{Pan-STARRS Image Processing Pipeline Modules} +\title{Pan-STARRS PS-1 Image Processing Pipeline Modules} \subtitle{Supplementary Design Requirements} \shorttitle{Modules SDRS} @@ -11,5 +11,5 @@ \project{Pan-STARRS Image Processing Pipeline} \organization{Institute for Astronomy} -\version{01} +\version{02} \docnumber{PSDC-430-012} @@ -34,9 +34,9 @@ \DocumentsInternal -PSCD-130-001 & PS-1 Design Reference Mission \\ \hline -PSCD-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline -PSCD-430-005 & Pan-STARRS IPP SRS \\ \hline -PSCD-430-006 & Pan-STARRS IPP ADD \\ \hline -PSCD-430-008 & Pan-STARRS IPP Architecture SDR \\ +PSCD-230-001 & PS-1 Design Reference Mission \\ \hline +PSDC-430-004 & Pan-STARRS PS-1 IPP C Code Conventions \\ \hline +PSDC-430-005 & Pan-STARRS PS-1 IPP Software Requirements Specification \\ \hline +PSDC-430-006 & Pan-STARRS PS-1 IPP Algorithm Design Document \\ \hline +PSDC-430-011 & Pan-STARRS PS-1 IPP System/Subsystem Design Description \\ \hline \DocumentsExternal Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ @@ -52,15 +52,519 @@ This document describes the Pan-STARRS Image Processing Pipeline (IPP) -image processing modules. The modules use the functionality of the -\PS{} Library (PSLib) to perform more complex tasks. They will -eventually be tied together, employing a script written in a -high-level language to form a ``processing stage''. Modules will be -constructed to support each of the required processing stages, and are -listed in this document according to the particular processing stage -for which they will primarily be used. +data analysis Modules. The Modules use the functionality of the +Pan-STARRS Library (PSLib) to perform more complex tasks, especially +tasks which require assumptions of astronomical analysis or the data +organization. Within the IPP, the Modules are tied together into +programs which perform complete data analysis tasks (an ``analysis +stage''). The modules may be tied together within a C framework or +using a high-level scripting language. Bindings of the Modules are +made available to the scripting language using the program SWIG. In order to preserve name space, globally-visible structures and -functions shall be prefixed with \code{pm}, for ``\PS{} Modules''. - +functions shall be prefixed with \code{pm}, for ``Pan-STARRS +Modules''. + +\section{Runtime Configuration Data} + +PSLib defines a \code{psMetadata} structure which can carry labeled +data of arbirtary types. The associated functions implemented by +PSLib consist of tools to manipulate and extract data from +\code{psMetadata} collections. Within PSLib, the \code{psMetadata} +stucture is used to carry the data from a FITS header. In addition, +functions are available to fill a \code{psMetadata} collection from a +text-based configuration file using a particular syntax, and to fill a +\code{psMetadata} collection from a properly formatted XML document. + +In the IPP Modules, we use \code{psMetadata} collections to carry +run-time configuration data used by the data analysis modules. Below, +in the discussion of the various modules, this configuration +information is defined by specifying the name of the data item of +interest, the conceptual meaning of that data item, and the allowed +values for the data item. In this section, we discuss top-level +concepts related to the configuration information, including the +sources of the run-time configuration data and special +operations used to extract information from the configuration system. + +\subsection{Configuration Data Sources} + +All modules need to load some configuration information defining +parameters which may be configured at run-time. Some module require +only a basic amount of information, while others require configuration +information defining the camera or details of an analysis (the {\it +recipe}). We define three utility functions to load these files from +standard locations, with optional command-line override values. + +\begin{verbatim} +bool pmConfigLoadSite (psMetadata **site, int *argc, char **argv); +bool pmConfigLoadCamera (psMetadata **camera, psMetadata *site, psMetadata *header, int *argc, char **argv); +bool pmConfigLoadRecipe (psMetadata **recipe, psMetadata *camera, int *argc, char **argv, char *script); +\end{verbatim} + +\code{pmConfigLoadSite} loads basic configuration information from a +general configuration file. By default, this file has the name +\code{~/.ipprc} in the user's home directory. This file may be +overridden on the command line with the optional flag \code{-site +(file)}. + +% eventually, we should define the complete sequence: +% $FOO, ~/.foorc, /prefix/config/foorc + +\code{pmConfigLoadCamera} loads camera configuration information from +a file which depends on the camera of interest. In some cases, the +camera may be determined from the header of an image currently in +hand. In other cases, the camera must be specified without reference +to a file. This function attempts to determine the appropriate camera +configuration file by examining, in order, the command-line options, +the header data, and the site configuration data. The camera config +file may be specified on the command line with the option +\code{-camera (file)}. If this does not exist, the +\code{pmConfigLoadCamera} examines the provided header to determine +the camera by using the function \code{pmCameraFromHeader} (defined +below). If this fails, the function attempts to determine the camera +from the provided site config data, looking for the keyword +\code{CAMERA} in the site config data. The camera name is used to +select the camera configuration file, defined in the site +configuration file with the keyword \code{CAMERA.name}, where +\code{name} is substituted with the name of the camera. + +\code{pmConfigLoadRecipe} loads the recipe configuration information +from a file which depends on the camera of interest and the particular +analysis task. The camera configuration file must define recipe files +for each of the analysis tasks of interest. The function attempts to +load the recipe file using the keyword \code{RECIPE.name}, where +\code{name} is substituted with the name of the analysis task, for +example \code{PHASE2}. The choice of the recipe file may be +overridden on the command line with the option \code{-recipe (file)}. + +\subsection{Indirect Configuration Data} + +Some configuration data is not provided directly by the configuration +files. Instead, the configuration system provides a mechanism to +define indirect references to configuration data. Three indirect +configuration data sources are currently defined: data from a FITS +header, data from the Metadata Database, and data from the Pan-STARRS +Status Server. The configuration data may specify that a certain +value is provided by one of these mechanisms. + +In order to specify that the configuration data should be derived from +a FITS header, the data value is given in the form: \code{HD:KEYWORD}, +where \code{KEYWORD} specifies the keyword to be used in search the +FITS header. + +If the data is to be extracted from the Metadata Database, the data +value is given in the form: \code{MD:TABLE,FIELD,KEY}. In this case, +\code{TABLE} specifies the Metadata Database table to be used to find +the value, \code{FIELD} specifies the field in that table to be used, +and \code{KEY} specifies the value of the primary key which gives the +value of interest. + +If the data is to be extracted from the Status Server, the data value +is given in the form: \code{SS:PATH}. In this case, \code{PATH} +specifies the Status Server path to the data value of interest. + +{\bf \it note that the Metadata Database and Status Server + interactions require us to define the APIs for these interactions. + These two methods are not available until the MD and SS APIs are + defined.} + +As an example, consider the definition of the data region of an image. +This value is frequenty represented in the FITS header with the +keyword \code{DATASEC}. However, in some cameras, this value may not +be defined, or another value may be used. We must be able to flexibly +define both the appropriate keyword, or the actual value, if known. +These two cases may be written as follows, using the configuration +file format parsed by \code{psMetadataParseConfig}: +\begin{verbatim} +DATA.REGION STR [1:100,2:400] +DATA.REGION STR HD:DATASEC +\end{verbatim} +In the first case, the data provided in the configuration file is the +data of interest. (\code{[1:100,2:400]}). In the second case, the +three-character code \code{HD:} specifies that the value of interest +may be determined by searching for the value associated with the +header keyword \code{DATASEC}. + +We provide utilities functions to extract this type of potentially +indirect configuration information. We provide four functions +returning four data types. The first three return the value from the +appropriate location assuming the data type to be \code{char*}, +\code{psS32}, and \code{psF64}. The fourth version searches for a +string which is passed to the function \code{psRegionFromString} +before being returned as a \code{psRegion} pointer. + +\begin{verbatim} +psSTR pmConfigLookupSTR (psMetadata *config, psMetadata *header, char *name); +psS32 pmConfigLookupS32 (psMetadata *config, psMetadata *header, char *name); +psF64 pmConfigLookupF64 (psMetadata *config, psMetadata *header, char *name); +psRegion *pmConfigLookupRegion (psMetadata *config, psMetadata *header, char *name); +\end{verbatim} + +\section{Camera Data Organization \& Camera Geometry} + +We require several utility functions to define the geometry of the +detectors in a camera and to specify the organization of the camera +data in real FITS images. The camera data organization is defined in +a set of \code{psMetadata} structures, and may be stored on disk in +the file format parsed by \code{psMetadataParseConfig}. + +PSLib defines a hierarchy of data structures related to the +organization of the pixels in a camera. These structures, starting +from the top level, follow the sequence \code{psFPA}, \code{psChip}, +\code{psCell}, \code{psReadout}, \code{psImage}. Each lower level +structure is carried as an array in the higher level. The containers +as defined by PSLib include functions which specify the astrometric +relationships between these levels, and provide a single +\code{psMetadata} container pointer for each level. In this section, +we define the type of metadata is stored in these metadata containers +and how the image headers are parse to define the data hierarchy in +memory. + +\subsection{Camera Config Data \& the {\tt psFPA}} + +A relevant collection of metadata is the externally supplied camera +definition metadata loaded with the function +\code{pmConfigLoadCamera}. This metadata defines the expected layout +of a specific camera along with information detailing how to interpret +the headers for an image from that camera. The camera metadata +consists of keyword / value pairs which are relevant to the entire +camera, and a set of metadata collections specified for each cell. +This top-level metadata is added to the \code{psFPA} with the name +'CAMERA.LAYOUT'. For example: +\begin{verbatim} +fpa->metadata = psMetadataAdd (fpa->metadata, PS_LIST_TAIL, "CAMERA", PS_META_META, "camera", camera); +\end{verbatim} +The specific cell-level entries are also placed on the correpsonding +\code{psCell} elements. These entries are added to the metadata with +the name 'CELL': +\begin{verbatim} +cell->metadata = psMetadataAdd (cell->metadata, PS_LIST_TAIL, "CELL", PS_META_META, "cell", cellMD); +\end{verbatim} + +\subsection{FITS File Data Representations} + +Within the FITS data representation, there are various choices which +can and have been made for the placement of the pixels in a data file. +In the simplest case, the camera consists of a single chip consisting +of a single cell always read with a single readout. In this case, the +image data could be written as part of the primary data unit. In a +more complex case with multiple chips and multiple cells, the data may +be organized in various ways. The data may be distributed into +multiple files or in multiple FITS data extensions. A single camera +image may be written as a collection of files for individual chips +with separate extensions for each cell (CFH12K.split, GPC). Another +camera may write a single file with multiple extensions for each cell +(Megacam.raw), or multiple extensions per chip, which each cell +representing portions of the chip image (Megacam.splice, CFHT-IR). + +In all of these representations, there are only two principal +distinctions in how the pixel data is stored. Either a single data +block (a single FITS image) represents only a single cell, or it +represents a collection of cells. This affects the way in which an +image is read in, the way the header is associated with a level of the +data hierarchy, and the way in which the header keywords are +interpretted to define the layout of the pixels in the cell. We +distinguish these two cases as 'cell-based' and 'chip-based' layout. + +The FITS headers are attached to the \code{psFPA} data hierarchy by +adding them to the \code{psMetadata} entries for the appropriate data +level. Each of the data levels \code{psFPA}, \code{psChip}, and +\code{psCell} have \code{metadata} elements to store relevant metadata +of any type. A FITS header metadata collection may be added to one of +these metadata collections using the metadata name 'HEADER': +\begin{verbatim} +fpa->metadata = psMetadataAdd (fpa->metadata, PS_LIST_TAIL, "HEADER", PS_META_META, "header", header); +\end{verbatim} + +In 'cell-based' layout, the headers are attached to the corresponding +\code{psCell} element. In 'chip-based' layout, the headers are +attached to the \code{psChip} element. This difference is natural +since a 'cell-based' data file has one FITS extension, and therefore +one FITS header, for each \code{cell}. Conversely, in a 'chip-based' +data file, there is one extension per chip, and therefore multiple +cells for each FITS header, but only one chip for each FITS header. + +\subsection{Determine the Camera from the Primary Header} + +\begin{verbatim} +char *pmCameraFromHeader (psFitsHeader *header, psMetadata *rules); +\end{verbatim} + +This function examines a primary header unit (\code{header}) and +determines the camera which provided the data. The rules which +identify the camera are defined as a \code{psMetadata} structure +containing one element for each known camera. These are generally to +be provided as part of the site configuration information. The +elements of this \code{psMetadata} container are in turn +\code{psMetadata} containers consisting of the expected header +keywords and their required values. The function +\code{pmCameraFromHeader} tests each of the defined cameras in +succession. For each camera, it searches for each of the given +keywords and compares the value with the value expected. If the +keyword does not exist, or if the keyword has the wrong value, the +camera is rejected. The name of first camera which matches the header +is returned as an allocated string. If no match is found, the +returned value must be \code{UNKNOWN}. This condition may be treated +by calling functions as either an error or only a warning. If an +error is encountered in parsing the metadata containers, then +\code{NULL} is returned. An example of the metadata keyword / value +pairs is given below (note that this uses an undefined metadata config +file representation \code{METADATA}): + +\begin{verbatim} +CFH12K.MEF METADATA + TELESCOP STR CFHT + INSTRUME STR CFH12K + EXTEND BOOL T + NEXTEND S32 12 +END + +CFH12K.SPLIT METADATA + TELESCOP STR CFHT + INSTRUME STR CFH12K + EXTEND BOOL F +END +\end{verbatim} + +\begin{figure} +\begin{center} +\psfig{file=CameraHierarchy,width=5.0in} +\caption{Camera Data and Metadata Hierarchy\label{CameraHierarchy}} +\end{center} +\end{figure} + +\subsection{Validate the Header set \& Construct the FPA} + +\begin{verbatim} +bool pmCameraValidateHeaders (psMetadata *headers, psMetadata *camera); +psFPA *pmFPAfromHeader (psMetadata *headers, psMetadata *camera); +\end{verbatim} + +These funtions examine the collection of \code{headers} and compare +them with the camera definitions in the metadata structure +\code{camera}. One function (\code{pmFPAfromHeader}) uses the header +set and camera definition file to construct a \code{psFPA} with all +\code{psChip} and \code{psCell} entries allocated, but without any +\code{psReadout} or pixel data. The other function validates the +header set against the camera definition, but does not actually +construct the \code{psFPA}. + +The camera definition metadata collection defines the relationship +between chips, cells, and the FITS extensions. An example of the data +needed by \code{pmCameraValidateHeaders} and \code{pmFPAfromHeader} is +given below for several types of cameras in the form of a metadata +config file. + +\begin{verbatim} +MEGACAM.RAW METADATA + NCELL S32 72 + CELL.FMT STR CELL.%02d + EXT.TYPE STR CELL + EXT.KEY STR EXTNAME + PHU STR FPA + # + # EXT.KEY CHIP + CELL.00 CELL amp00 CHIP.00 + CELL.01 CELL amp01 CHIP.00 + CELL.02 CELL amp02 CHIP.01 + CELL.03 CELL amp03 CHIP.01 + ... +END + +MEGACAM.SPLICE METADATA + NCELL S32 36 + CELL.FMT STR CELL.%02d + EXT.TYPE STR CHIP + EXT.KEY STR EXTNAME + PHU STR FPA + # + # EXT.KEY CHIP + CELL.00 CELL ccd00 CHIP.00 + CELL.01 CELL ccd00 CHIP.00 + CELL.02 CELL ccd01 CHIP.01 + CELL.03 CELL ccd01 CHIP.01 + ... +END + +CFH12K.SPLIT METADATA + NCELL S32 12 + CELL.FMT STR CELL.%02d + EXT.TYPE STR CELL + EXT.KEY STR EXTNAME + PHU STR NONE + # + # EXT.KEY CHIP + CELL.00 CELL chip00 CHIP.00 + CELL.01 CELL chip01 CHIP.01 + CELL.02 CELL chip02 CHIP.02 + ... +END + +GPC.RAW METADATA + NCELL S32 4096 + CELL.FMT STR CELL.%04d + EXT.TYPE STR CELL + EXT.KEY STR EXTNAME + PHU STR CHIP + # + # EXT.KEY CHIP + CELL.0000 CELL amp00 CHIP.00 + CELL.0001 CELL amp01 CHIP.00 + CELL.0002 CELL amp02 CHIP.00 + CELL.0003 CELL amp03 CHIP.00 + ... + CELL.0065 CELL amp03 CHIP.01 + CELL.0066 CELL amp03 CHIP.01 + ... +END +\end{verbatim} + +Among the important elements of the camera definition information are: +\begin{itemize} +\item \code{NCELL} - this defines the possible number of cells from this + camera. +\item \code{CELL.FMT} - this defines the format of the \code{CELL.nn} + keywords below in the metadata collection, one for each of the + \code{NCELL} values +\item \code{EXT.TYPE} - this defines the lowest data level + corresponding to a single FITS extension: \code{CELL} or + \code{CHIP}. In some cases, a single extension represents a chip, + and is subdivided into cells by header keywords specifying certain + regions. In other cases, a single extension represents only the + data from a single amplifier, ie, a cell. These two cases require + somewhat different handling. +\item \code{EXT.KEY} - this entry defines a header keyword which + allows the unique identification of a given header with one of the + cell or chip entries (depending on the value of \code{EXT.TYPE}). +\item \code{PHU} - this entry defines the meaning of the FITS file + primary header unit. +\end{itemize} +In addition to these generic parameters, the camera definition +metadata includes an entry for each cell giving specific values +required to define that cell. + +Both functions \code{pmCameraValidateHeaders} and +\code{pmFPAfromHeader} must start with the set of headers and attempt +to identify the corresponding cell or chip. All available cells and +chips must be identified, and the cells corresponding to each chip +must be tracked. In the case of \code{pmCameraValidateHeaders}, the +function must only verify the headers contain valid extensions, +without allocating the correpsonding \code{psFPA}, while +\code{pmFPAfromHeader} must also allocate the \code{psFPA}, the +contained \code{psChip} arrays, and the correct number of +\code{psCell} arrays for each \code{psChip}. + +These functions also add to the metadata containers for the chips and +cells, following the rules discussed above. Each \code{psCell} entry +should have the \code{CELL} metadata lines from above attached to the +\code{psCell.metadata} element as an additional metadata collection +with the name \code{CELL.LAYOUT}. The complete camera metadata +collection (including the cells), are attached to the +\code{psFPA.metadata} element with the name \code{CAMERA.LAYOUT}. The +image header data are also attached to metadata entries, at a level +which depends on the value of \code{EXT.TYPE} in the camera layout +data above. If the value is \code{CELL}, the header metadata +collections are attached to the \code{psCell.metadata} elements with +the name \code{HEADER}. If the value of \code{EXT.TYPE} is chip, the +header metadata is attached to the \code{psChip.metadata} element, +again with the name \code{HEADER}. + +An image file may also have a primary header unit which is not +associated with a data block. This metadata block must also be added +to the data heirarchy so successive operations may update the metadata +as needed. The disposition of the primary header unit is defined by +the camera layout key \code{PHU}. If this has a value of \code{NONE}, +there is no primary header unit, or it should be ignored. If the +value is \code{FPR}, the primary header data is attached to the +\code{psFPA.metadata} element with the value PHU. Alternatively, if +it has the value \code{CHIP}, then the primary header metadata is +attached to the \code{psChip.metadata} element. + +\subsection{Coordinate Transforms and Header Data} +\tbd{the algorithms for three functions in this section are not + well-defined. do not code yet}. + +Astrometric and geometric information about an image from a camera may +be represented in a variety of ways. A crude representation of the +pixel geometry is specified in many image headers using the IRAF-style +region keywords \code{DATASEC}, \code{DETSEC}, etc. These keywords +are used to define the location of a single image's pixels in the +frame of the full mosaic of detectors in the assumption that the +mosaic can be represented as a single uniform grid of pixels. An +alternative set of keywords have been used in cases where multiple +cells are saved together in a single FITS image extension. More +sophisticated astrometric representations require elements to define +projections, scaling, distortion, etc. Several versions of header +keywords have been used to represent these astrometric +transformations. In this section, we define three functions to +interpret a collection of image headers and construct the appropriate +offset and/or astrometry parameters. + +\begin{verbatim} +bool pmFPADefineOffsets (psFPA fpa); +bool pmFPADefineWCS (psFPA fpa); +bool pmFPADefineWCSfromOffsets (psFPA fpa); +\end{verbatim} + +The first function takes a \code{psFPA} structure which has been +populated with header and camera configuration metadata in the +appropriate locations as discussed above. Using the information in +the camera config metadata and the headers, the function sets the +values for the elements \code{psChip.col0,row0}, +\code{psCell.col0,row0}, \code{psReadout.col0,row0}, +\code{psReadout.colParity,rowParity}, and +\code{psReadout.colBinning,rowBinning}. This information is +determined by examining the regions defined by the following names. + +% how do we handle the CCDSUM keyword case? +\begin{verbatim} +psCell.metadata:CELL:CCDBIN1 $\rightarrow$ psReadout.colBins +psCell.metadata:CELL:CCDBIN2 $\rightarrow$ psReadout.rowBins + +DETSEC from psCell.metadata:CELL:DETSEC +CCDSEC from psCell.metadata:CELL:CCDSEC +DATASEC from psCell.metadata:CELL:DATASEC + +if (DETSEC.x0 > DETSEC.x1) then psReadout.colParity = -1 +if (DETSEC.y0 > DETSEC.y1) then psReadout.rowParity = -1 +\end{verbatim} + +The second function examines the contents of the headers of the chips +and cells and constructs the collection of astrometric coordinate +transformations.. + +The third function uses the offset information and the basic telescope +pointing information to construct a approximate guess at the +astrometric coefficients based on the detector geometry. + +\begin{figure} +\begin{center} +\psfig{file=CameraLayout,width=5.5in} +\caption{Camera Pixel Layout\label{CameraLayout}} +\end{center} +\end{figure} + +\begin{figure} +\begin{center} +\psfig{file=CameraRegionKeywords.ps,width=5.5in} +\caption{Camera Region Keyword Definitions\label{CameraRegionKeywords}} +\end{center} +\end{figure} + +\subsection{Chip \& Cell from FITS File} +\tbd{the algorithm for the function in this section is not + well-defined. do not code yet}. + +When loading data from disk, it is may be necessary to use the +information in the DATASEC and DETSEC entries to determine which part +of the image should be read. If the data is stored in a chip-based +format, then the data for each cell corresponds to only a fraction of +the pixels stored in a single image extension. We specify the +following function to perform the correct read of data from a FITS +file into the corresponding \code{psCell} entry respecting the +boundaries of the cells within chip-based images. + +\begin{verbatim} +psReadout *pmReadoutLoad (psReadout *input, psFits *f, psCell *cell, int plane); +\end{verbatim} \section{Phase 2} @@ -76,5 +580,5 @@ \item Flat-field; \item Mask bad pixels; -\item \tbd{Subtract the background;} +\item Subtract the background; \item \tbd{Mask cosmic rays;} \item \tbd{Mask optical defects;} @@ -358,174 +862,19 @@ image), the function shall generate an error. - - -\section{Calibration} - -The calibration module essentially consists of combining multiple -images of a particular type in order to build up signal-to-noise. For -this, we require a general purpose image combination module. We -forsee this module as only acting upon data from the same detector, -and so each input image will have the same noise characteristics. - -\begin{verbatim} -typedef struct { - psStats *stats; // Statistics to use in combining pixels - unsigned int maskVal, // Mask pixels where mask & maskVal == 1 - float fracHigh; // Fraction of high pixels to throw - float fracLow; // Fraction of low pixels to throw - int nKeep; // Number of pixels to be sure to keep -} pmCombineParams; - -psImage * -pmReadoutCombine(psImage *output, // Output image, or NULL - const psList *inputs, // List of input readouts - pmCombineParams *params, // Combination parameters - const psVector *zero, // Offsets to apply for each image - const psVector *scale, // Scales to apply for each image - bool applyZeroScale, // Are zero and scale for application, or only noise properties? - float gain, // Gain in e/ADU - float readnoise // Read noise in e - ); -\end{verbatim} - -\code{pmReadoutCombine} combines input images pixel by pixel --- for -each pixel of the output image, a stack of contributing input pixels -is formed and combined. Several of its input parameters are lists or -vectors, and if these are not all of the same length (or \code{NULL}), -the module shall generate an error and return \code{NULL}. - -If the provided \code{output} is \code{NULL}, then the module shall -allocate a new image of sufficient size for the input images. If the -\code{output} image is non-\code{NULL} and is not of sufficient size -for the combined image, the module shall generate an error and return -\code{NULL}. - -If the \code{inputs} is \code{NULL}, the module shall generate an -error and return \code{NULL}. Otherwise, the \code{inputs} shall be a -list of \code{psReadout}s. The images contained within the -\code{psReadout}s need not all be of the same size, but the module -shall take into account the offsets (\code{col0,row0}) from the corner -of the detector when comparing pixels, so that it is the same -\textit{physical} pixels that are combined. - -The parameters used in the combination, including how the pixels are -to be combined, and how the rejection is performed is contained within -the \code{params}, which may not be \code{NULL} (otherwise the module -shall generate an error and return \code{NULL}). We choose to use -this structure instead of supplying the values separately in order to -keep down the number of parameters to \code{pmReadoutCombine}; the -\code{psCombineParams} may be recycled for subsequent calls to -\code{pmReadoutCombine} since the values are not dependent upon the -choice of inputs, but merely specify how the combination is to be -performed. - -The particular statistic specified by \code{stats} shall be used to -combine each stack of pixels from the input images. Only one of the -statistics choices may be specified, otherwise the module shall -generate an error and return \code{NULL}. - -If the \code{maskVal} is non-zero, then pixels in the \code{mask} of -each \code{psReadout} in the \code{inputs} which satisfy the -\code{maskVal} shall not have the corresponding pixels placed in the -stack for combination. - -After masking, but before performing the combination, the highest -\code{fracHigh} fraction and lowest \code{fracLow} fraction of pixels -in the stack are immediately rejected, unless this would leave less -than \code{nKeep} pixels in the stack, in which case no immediate -rejection is performed. - -If the \code{zero} vector is non-\code{NULL} and \code{applyZeroScale} -is \code{true}, then the appropriate values shall be added to the -\code{inputs} before rejection is performed. If \code{zero} is -non-\code{NULL} and \code{applyZeroScale} is false, then the values -shall only be used in calculating the noise. - -If the \code{scale} vector is non-\code{NULL} and -\code{applyZeroScale} is \code{true}, then the appropriate values -shall multiply the \code{inputs} before rejection is performed. If -\code{scale} is non-\code{NULL} and \code{applyZeroScale} is false, -then the values shall only be used in calculating the noise. - -The purpose of \code{applyZeroScale} is to allow combination of fringe -frames, where the frames have been deliberately sky-subtracted and -rescaled (to get the fringes amplitudes running from -1 to 1), which -actions should not be undone when combining, but yet it is desirable -to provide the \code{zero} and \code{scale} values so that the correct -noise properties are used in the combination. - -If the \code{gain} and \code{readnoise} are positive and non-negative -(respectively), then these shall be used to provide weights for the -combination using Poisson statistics. - -The \code{inputs, zero} and \code{scale} may be of S16, S32 and F32 -types, and must all be of the same type. The \code{output} shall be -of the same type. - -\section{Camera Configutation Information} - -\tbd{these function are not yet ready for implementation. The APIs - are fine but the algorithms need to be specified.} - -We require several utility functions to define the general layout of a -camera and to relate the camera definition to the header in a given -camera. This information may be passed to the modules by using the -configuration files as parse by \code{psMetadataParseConfig}. These - -\begin{verbatim} -char *pmCameraFromHeader (psFitsHeader *header); -\end{verbatim} -This function examines a primary header unit (\code{header}) and -determines the camera which provided the data. \tbr{the rules for - performing this analysis probably depend on the keywords such as - INSTRUME and TELESCOP. Should this function be a script so we can - easily add a new camera or should the rule set be defined by - configuration information?}. - -\begin{verbatim} -bool pmCameraValidateHeaders (psHash *headers, psMetadata *camera); -\end{verbatim} -This funtion examines the collection of \code{headers} and, based on -the camera definitions in the metadata \code{camera}, determines if -the header set is consistent with the specified camera. - -\begin{verbatim} -psFPA *pmFPAfromHeader (psList *header, psMetadata *camera); -\end{verbatim} -This funtion uses the collection of \code{headers} and the camera -definitions in the metadata \code{camera} to construct a complete -\code{psFPA} structure which represents the data described by the -collection of \code{headers}. The resulting \code{FPA} has all -possible structure elements defined but no space for pixel data should -be allocated. - -\section{Revision Change Log} -\input{ChangeLogSDRS.tex} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\bibliographystyle{plain} -\bibliography{panstarrs} - -\end{document} - \subsection{Subtract sky} - -\tbd{This is a simple sky subtraction routine. A more complicated -routine will be specified in the future, following research into the -best algorithm.} Given an input image, a polynomial or spline specifying the order of a desired fit, a binning factor and statistics to use for the binning, -along with a clipping level, \code{psPhase2SubtractSky} shall fit and +along with a clipping level, \code{pmSubtractSky} shall fit and subtract a model for the background of the image. The API shall be the following: \begin{verbatim} -psReadout *psPhase2SubtractSky(psReadout *in, void *fitSpec, psFit fit, int binFactor, psStats *stats, - float clipSD); +psReadout *pmSubtractSky(psReadout *in, void *fitSpec, psFit fit, int binFactor, + psStats *stats, float clipSD); \end{verbatim} Note that the input image, \code{in}, shall be subtracted in-place. The function shall return the subtracted image, and also update the -polynomial, chebyshev or spline specified by \code{fitSpec}, to hold +polynomial, Chebyshev or spline specified by \code{fitSpec}, to hold the coefficients used in the subtraction. @@ -560,4 +909,116 @@ +\section{Calibration} + +The calibration module essentially consists of combining multiple +images of a particular type in order to build up signal-to-noise. For +this, we require a general purpose image combination module. We +forsee this module as only acting upon data from the same detector, +and so each input image will have the same noise characteristics. + +\begin{verbatim} +typedef struct { + psStats *stats; // Statistics to use in combining pixels + unsigned int maskVal, // Mask pixels where mask & maskVal == 1 + float fracHigh; // Fraction of high pixels to throw + float fracLow; // Fraction of low pixels to throw + int nKeep; // Number of pixels to be sure to keep +} pmCombineParams; + +psImage * +pmReadoutCombine(psImage *output, // Output image, or NULL + const psList *inputs, // List of input readouts + pmCombineParams *params, // Combination parameters + const psVector *zero, // Offsets to apply for each image + const psVector *scale, // Scales to apply for each image + bool applyZeroScale, // Are zero and scale for application, or only noise properties? + float gain, // Gain in e/ADU + float readnoise // Read noise in e + ); +\end{verbatim} + +\code{pmReadoutCombine} combines input images pixel by pixel --- for +each pixel of the output image, a stack of contributing input pixels +is formed and combined. Several of its input parameters are lists or +vectors, and if these are not all of the same length (or \code{NULL}), +the module shall generate an error and return \code{NULL}. + +If the provided \code{output} is \code{NULL}, then the module shall +allocate a new image of sufficient size for the input images. If the +\code{output} image is non-\code{NULL} and is not of sufficient size +for the combined image, the module shall generate an error and return +\code{NULL}. + +If the \code{inputs} is \code{NULL}, the module shall generate an +error and return \code{NULL}. Otherwise, the \code{inputs} shall be a +list of \code{psReadout}s. The images contained within the +\code{psReadout}s need not all be of the same size, but the module +shall take into account the offsets (\code{col0,row0}) from the corner +of the detector when comparing pixels, so that it is the same +\textit{physical} pixels that are combined. + +The parameters used in the combination, including how the pixels are +to be combined, and how the rejection is performed is contained within +the \code{params}, which may not be \code{NULL} (otherwise the module +shall generate an error and return \code{NULL}). We choose to use +this structure instead of supplying the values separately in order to +keep down the number of parameters to \code{pmReadoutCombine}; the +\code{psCombineParams} may be recycled for subsequent calls to +\code{pmReadoutCombine} since the values are not dependent upon the +choice of inputs, but merely specify how the combination is to be +performed. + +The particular statistic specified by \code{stats} shall be used to +combine each stack of pixels from the input images. Only one of the +statistics choices may be specified, otherwise the module shall +generate an error and return \code{NULL}. + +If the \code{maskVal} is non-zero, then pixels in the \code{mask} of +each \code{psReadout} in the \code{inputs} which satisfy the +\code{maskVal} shall not have the corresponding pixels placed in the +stack for combination. + +After masking, but before performing the combination, the highest +\code{fracHigh} fraction and lowest \code{fracLow} fraction of pixels +in the stack are immediately rejected, unless this would leave less +than \code{nKeep} pixels in the stack, in which case no immediate +rejection is performed. + +If the \code{zero} vector is non-\code{NULL} and \code{applyZeroScale} +is \code{true}, then the appropriate values shall be added to the +\code{inputs} before rejection is performed. If \code{zero} is +non-\code{NULL} and \code{applyZeroScale} is false, then the values +shall only be used in calculating the noise. + +If the \code{scale} vector is non-\code{NULL} and +\code{applyZeroScale} is \code{true}, then the appropriate values +shall multiply the \code{inputs} before rejection is performed. If +\code{scale} is non-\code{NULL} and \code{applyZeroScale} is false, +then the values shall only be used in calculating the noise. + +The purpose of \code{applyZeroScale} is to allow combination of fringe +frames, where the frames have been deliberately sky-subtracted and +rescaled (to get the fringes amplitudes running from -1 to 1), which +actions should not be undone when combining, but yet it is desirable +to provide the \code{zero} and \code{scale} values so that the correct +noise properties are used in the combination. + +If the \code{gain} and \code{readnoise} are positive and non-negative +(respectively), then these shall be used to provide weights for the +combination using Poisson statistics. + +The \code{inputs, zero} and \code{scale} may be of S16, S32 and F32 +types, and must all be of the same type. The \code{output} shall be +of the same type. + +\section{Revision Change Log} +\input{ChangeLogSDRS.tex} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\bibliographystyle{plain} +\bibliography{panstarrs} + +\end{document} + \subsection{Cosmic rays} @@ -705,50 +1166,2 @@ \code{in}. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Read in an FPA} - -\TBD{This section on reading in an FPA is all TBD and should not be -implemented yet.} - -Of course, before any image manipulation may occur, the image must be -read in. This involves populating the FPA with structures to -represent the true focal plane. However, different instruments have -different focal planes, and the images are written to disk in -different ways. We have proposed a configuration file format which -appears to be able to document the structure of a focal plane and -specify how the IPP can read the required information from the FITS -files \tbd{REFERENCE}. - -Given an open \code{FILE} pointer to a configuration file, -\code{config}, and the name of an FPA image, \code{fpaName}, -\code{psFPAGenerate} shall return a \code{psFPA} of the correct -structure, with all the metadata entries read from the nominated FPA -image. Astrometric information shall be initialised to values -specified by the configuration file. The API shall be: - -\begin{verbatim} -psFPA *psFPAGenerate(FILE *config, const char *fpaName); -\end{verbatim} - -Once an FPA has been generated, the pixels may all be read in at once, -or the pixels may be read in a chip, cell or readout at a time. The -corresponding APIs are: - -\begin{verbatim} -psFPA *psFPAReadAll(psFPA *in); -psFPA *psFPAReadChip(psFPA *in, const char *chipName); -psChip *psChipReadCell(psChip *in, const char *cellName); -psCell *psCellReadReadout(psCell *in, const char *readoutName); -\end{verbatim} - -Note that reads of a cell or readout do not act upon an FPA, but -rather a chip and a cell, respectively. This saves the user from -specifying the name of the FPA, chip, and cell just to get a readout, -since this information is already stored in the generated \code{psFPA} -struct. - -\tbd{Writing the output} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%