Index: /tags/ipp-docs/MSDR-08/doc/modules/.cvsignore =================================================================== --- /tags/ipp-docs/MSDR-08/doc/modules/.cvsignore (revision 22201) +++ /tags/ipp-docs/MSDR-08/doc/modules/.cvsignore (revision 22201) @@ -0,0 +1,2 @@ +*.log *.dvi *.aux *.toc *.log *.out +configProposal.pdf ModulesSDRS.pdf Index: /tags/ipp-docs/MSDR-08/doc/modules/CameraGeometry.tex =================================================================== --- /tags/ipp-docs/MSDR-08/doc/modules/CameraGeometry.tex (revision 22201) +++ /tags/ipp-docs/MSDR-08/doc/modules/CameraGeometry.tex (revision 22201) @@ -0,0 +1,109 @@ + +\section{Camera Data Organization \& Camera Geometry} + +We require several utility functions to define the geometry of the +detectors in a camera and to specify the organization of the camera +data in real FITS images. The camera data organization is defined in +a set of \code{psMetadata} structures, and may be stored on disk in +the file format parsed by \code{psMetadataParseConfig}. + +PSLib defines a hierarchy of data structures related to the +organization of the pixels in a camera. These structures, starting +from the top level, follow the sequence \code{psFPA}, \code{psChip}, +\code{psCell}, \code{psReadout}, \code{psImage}. Each lower level +structure is carried as an array in the higher level. The containers +as defined by PSLib include functions which specify the astrometric +relationships between these levels, and provide a single +\code{psMetadata} container pointer for each level. In this section, +we define the type of metadata is stored in these metadata containers +and how the image headers are parse to define the data hierarchy in +memory. + +\subsection{Coordinate Transforms and Header Data} +\tbd{The algorithms for three functions in this section are not + well-defined. Do not code yet.} + +Astrometric and geometric information about an image from a camera may +be represented in a variety of ways. A crude representation of the +pixel geometry is specified in many image headers using the IRAF-style +region keywords \code{DATASEC}, \code{DETSEC}, etc. These keywords +are used to define the location of a single image's pixels in the +frame of the full mosaic of detectors in the assumption that the +mosaic can be represented as a single uniform grid of pixels. An +alternative set of keywords have been used in cases where multiple +cells are saved together in a single FITS image extension. More +sophisticated astrometric representations require elements to define +projections, scaling, distortion, etc. Several versions of header +keywords have been used to represent these astrometric +transformations. In this section, we define three functions to +interpret a collection of image headers and construct the appropriate +offset and/or astrometry parameters. + +\begin{prototype} +bool pmFPADefineOffsets(psFPA fpa); +bool pmFPADefineWCS(psFPA fpa); +bool pmFPADefineWCSfromOffsets(psFPA fpa); +\end{prototype} + +The first function takes a \code{psFPA} structure which has been +populated with header and camera configuration metadata in the +appropriate locations as discussed above. Using the information in +the camera config metadata and the headers, the function sets the +values for the elements \code{psChip.col0,row0}, +\code{psCell.col0,row0}, \code{psReadout.col0,row0}, +\code{psReadout.colParity,rowParity}, and +\code{psReadout.colBinning,rowBinning}. This information is +determined by examining the regions defined by the following names. + +% how do we handle the CCDSUM keyword case? +\begin{verbatim} +psCell.metadata:CELL:CCDBIN1 $\rightarrow$ psReadout.colBins +psCell.metadata:CELL:CCDBIN2 $\rightarrow$ psReadout.rowBins + +DETSEC from psCell.metadata:CELL:DETSEC +CCDSEC from psCell.metadata:CELL:CCDSEC +DATASEC from psCell.metadata:CELL:DATASEC + +if (DETSEC.x0 > DETSEC.x1) then psReadout.colParity = -1 +if (DETSEC.y0 > DETSEC.y1) then psReadout.rowParity = -1 +\end{verbatim} + +The second function examines the contents of the headers of the chips +and cells and constructs the collection of astrometric coordinate +transformations.. + +The third function uses the offset information and the basic telescope +pointing information to construct a approximate guess at the +astrometric coefficients based on the detector geometry. + +\begin{figure} +\begin{center} +\psfig{file=CameraLayout,width=5.5in} +\caption{Camera Pixel Layout\label{CameraLayout}} +\end{center} +\end{figure} + +\begin{figure} +\begin{center} +\psfig{file=CameraRegionKeywords.ps,width=5.5in} +\caption{Camera Region Keyword Definitions\label{CameraRegionKeywords}} +\end{center} +\end{figure} + +\subsection{Chip \& Cell from FITS File} +\tbd{the algorithm for the function in this section is not + well-defined. do not code yet}. + +When loading data from disk, it is may be necessary to use the +information in the DATASEC and DETSEC entries to determine which part +of the image should be read. If the data is stored in a chip-based +format, then the data for each cell corresponds to only a fraction of +the pixels stored in a single image extension. We specify the +following function to perform the correct read of data from a FITS +file into the corresponding \code{psCell} entry respecting the +boundaries of the cells within chip-based images. + +\begin{prototype} +psReadout *pmReadoutLoad(psReadout *input, psFits *f, psCell *cell, int plane); +\end{prototype} + Index: /tags/ipp-docs/MSDR-08/doc/modules/CameraImages.tex =================================================================== --- /tags/ipp-docs/MSDR-08/doc/modules/CameraImages.tex (revision 22201) +++ /tags/ipp-docs/MSDR-08/doc/modules/CameraImages.tex (revision 22201) @@ -0,0 +1,725 @@ +\section{Focal Plane} +\label{sec:focalplane} + +\subsection{Overview} + +In PSLib, we have defined a basic container for a single 2D collection +of pixels (\code{psImage}), along with basic operations to manipulate +the image pixels. For astronomical applications, this data structure +is insufficient for two reasons. First, it does not provide +sufficient additional metadata to describe the data in detail. +Second, astronomy applications frequently involve multiple, related +images. For Pan-STARRS, and for general astronomical applications, we +require a richer collection of data structures which describe a very +general image concept. We have defined several layers in the +hierarchy which are necessary to describe the image data which will be +produced by the Pan-STARRS GigaPixel Cameras as well as other standard +astronomical images. + +A simple 2D image is the basic data unit for much of astronomical +imaging: if we consider various optical and IR array cameras, a single +readout of the detector produces a collection of pixel measurements +which is well represented as a single 2D image. We define our +lowest-level astronomical image structure, \code{pmReadout}, to +contain the pixels produced by a single readout of the detector, along +with metadata needed to define that readout: the origin and binning of +the image relative to the original detector pixels explicitly in the +structure, and pointers to the general metadata and derived objects, +if any. + +A single detector may be read multiple times in sequence. For +example, infrared detectors frequently produce an image immediately +after the detector is reset followed by an image after the basic +exposure is complete. Both readouts correspond to the same pixels, +though the binning or rastering may be different between the two +readouts. Another example is the video sequence produced by the +Pan-STARRS GigaPixel Camera guide cells, each of which represents a +series of many images from a subraster of pixels in the detector +readout portion. The second level of our image container hierarchy, +\code{pmCell}, consists of a collection of readouts from a single +detector. + +In the Pan-STARRS GigaPixel camera, the basic readout region is a +fraction of the full imaging area of a single CCD chip. The chip is +divided into 64 cells, any fraction of which may have been readout for +a given exposure. In other cameras, such as Megacam at CFHT, the +individual CCDs have multiple amplifiers addressing contiguous +portions of the detector. In such cameras, each amplifier produces a +separate collection of pixels. In the third level of our image +container hierarchy, the data structure \code{pmChip} represents a +collection of different cells. + +The top level of our image container hierarchy is a complete focal +plane array (\code{pmFPA}). This structure represents the collection +of chips in the camera, all of which are read out in a given +exposure. + +For example, take a mosaic camera consisting of eight $2k\times 4k$ +CCDs, each of which is read out through two amplifiers. Then there +would be sixteen cells in total, each of which is presumably $1k\times +4k$. There would be eight chips, each consisting of two cells, and +the focal plane consists of these eight chips. + +As another example, consider an observation by PS-1. The focal plane +would consist of 60 chips, each of which consist of 64 cells (or less; +a few cells may be dead). Some cells (those containing guide stars +for the orthogonal transfer) will contain multiple readouts. + +These data structures represent containers with which to carry around +the collection of related image data. There is no requirement on the +functions or the structures that each instance of one of these data +structures represent the physical hardware. For example, it is not +necessary that an instance of \code{pmFPA} always carry the data for +all 60 GigaPixel Camera OTAs. The usage of these structures is such +that all astronomical operations which apply to a CCD image should be +performed on an instance of \code{pmFPA}. If a particular +circumstance only requires a single 2D image, then that is represented +by an instance of \code{pmFPA} with one \code{pmChip}, which in turn +has one \code{pmCell}, which in turn has one \code{pmReadout}. + +The data structures defined below provide two additional features +beyond the hierarchy of relationships. First, each level of the +hierarchy includes hooks for carrying metadata to provide the PS +concepts and analysis metadata that would be appropriate for that +level. The functions within PSLib do not specify the contents of +those metadata containers. + +While the \code{psMetadata} pointers provide a mechanism to carry +generic information about the image, the hierarchy of data structures +also provides an explicit set of information defining the geometrical +relationships between the levels of the hierarchy. Two types of +information are provided. In the first case, basic offsets (and in +the case of the readouts, binning and flips) are defined to specify +the location of a given \code{pmCell} with respect to its containing +\code{pmChip} in the assumption that the pixels in the entire focal +plane array are laid out on a uniform grid. This is a crude +approximation, and cannot be assumed for careful astrometric analysis, +but it can be used as a starting point or to place the the pixels in a +test image. For higher precision, detailed astrometric +transformations between one frame and the next are also provided. + +\tbd{In the future, it may be worthwhile to migrate all of these +additional pieces to the psMetadata since there is no pressing need to +have them visible in the data structures} + +\subsection{Image Data Container Hierarchy} + +Here we specify the contents of the focal plane hierarchy: +\code{pmReadout}, \code{pmCell}, \code{pmChip} and \code{pmFPA}. Many +of the components of these are similar. All but the \code{pmFPA} +contain offsets from the level above (\code{col0,row0}), and a link to +the parent. All but the \code{pmReadout} contain a private pointer to +FITS data and more detailed astrometric transforms. Each contains an +\code{analysis} metadata container which is intended to store results +of analyses (e.g., the r.m.s. of the overscan fit). + +\tbd{At what stage are the offsets ({\tt col0,row0}) set, and how are +they known?} + +\subsubsection{A Readout} + +A readout is the result of a single read of a cell (or a portion +thereof). It contains the offset from the lower-left corner of the +chip, in the case that the CCD was windowed, as well as the binning +factors and parity (if the binning value is negative, then the parity +is reversed). It also contains the pixel data (with corresponding +mask and weight), metadata container for the analysis, and a link to +the parent. + +\begin{verbatim} +typedef struct { + // Position on the cell + int col0; // Offset from the left of cell. + int row0; // Offset from the bottom of cell. + int colBins; // Amount of binning in x-dimension and parity (from sign) + int rowBins; // Amount of binning in y-dimension and parity (from sign) + // Information + psImage *image; // Imaging area of readout + psImage *mask; // Mask for image + psImage *weight; // Weight for image + psList *bias; // List of bias section (sub-)images + psMetadata *analysis; // Readout-level analysis metadata + pmCell *parent; // Parent cell +} pmReadout; +\end{verbatim} + +The constructor for \code{pmReadout} shall be: +\begin{verbatim} +pmReadout *pmReadoutAlloc(pmCell *cell); +\end{verbatim} +The constructor shall make an empty \code{pmReadout}. If the parent +\code{cell} is not \code{NULL}, the \code{parent} link is made and the +readout shall be placed in the parent's array of \code{readouts}. The +metadata containers shall be allocated. All other pointers in the +structure shall be initialized to \code{NULL}. + +\subsubsection{A Cell} + +A cell consists of one or more readouts (usually only one except in +the case that the cell has been used for fast guiding, or similar +situations). It has values which specifies the position of the cell +on the chip for rough positioning, along with more precise coordinate +transforms from the cell to the chip and, as a convenience, from the +cell directly to the focal plane. It is expected that these +transforms will consist of two first-order 2D polynomials, simply +specifying a translation, rotation and magnification; hence they are +easily inverted, and there is no need to add reverse transformations. +We also add an additional transformation, which is intended to provide +a ``quick and dirty'' transform from the cell coordinates to the sky; +this transformation not guaranteed to be as precise as the +``standard'' transformation of Cell $\rightarrow$ Chip $\rightarrow$ +Focal Plane $\rightarrow$ Tangent Plane $\rightarrow$ Sky, but will be +faster. The cell also contains metadata containers for the concepts +and analysis, a link to the parent, and a container for the FITS data, +if that corresponds to this level. A boolean indicates whether the +cell is of interest, allowing it to be excluded from analysis. + +\begin{verbatim} +typedef struct { + // Offset specifying position on chip + int col0; // Offset from the left of chip. + int row0; // Offset from the bottom of chip. + // Astrometric transformations + psPlaneTransform* toChip; // Transformations from cell to chip coordinates + psPlaneTransform* toFPA; // Transformations from cell to FPA coordinates + psPlaneTransform* toSky; // Transformations from cell to sky coordinates + // Information + psMetadata *concepts; // Cache for PS concepts + psMetadata *camera; // Camera information + psMetadata *analysis; // Cell-level analysis metadata + psArray *readouts; // The readouts (referred to by number) + pmChip *parent; // Parent chip + bool valid; // Do we bother about reading and working with this cell? + p_pmHDU *private; // FITS data +} pmCell; +\end{verbatim} + +The constructor for \code{pmCell} shall be: +\begin{verbatim} +pmCell *pmCellAlloc(pmChip *chip, psMetadata *cameraData, psString name); +\end{verbatim} +The constructor shall make an empty \code{pmCell}. If the parent +\code{chip} is not \code{NULL}, the \code{parent} link is made and the +cell shall be placed in the parent's array of \code{cells}. The +\code{readouts} array shall be allocated with a zero size, and the +metadata containers constructed. The cell's \code{camera} pointer +shall be set to the provided \code{cameraData}, and the \code{name} +shall be used to set \code{CELL.NAME} in the \code{concepts}. All +other pointers in the structure shall be initialized to \code{NULL}. + +\subsubsection{A Chip} + +A chip consists of one or more cells (according to the number of +amplifiers on the device). The chip contains metadata containers for +the concepts and analysis, a link to the parent, and pointers to the +pointers to the various FITS data, if that corresponds to this level. +For astrometry, in addition to the rough positioning information, it +contains a coordinate transform from the chip to the focal plane. It +is expected that this transform will consist of two second-order 2D +polynomials; hence we think that it is prudent to include a reverse +transformation which will be derived from numerically inverting the +forward transformation. A boolean indicates whether the chip is of +interest, allowing it to be excluded from analysis. + +\begin{verbatim} +typedef struct { + // Offset specifying position on focal plane + int col0; // Offset from the left of FPA. + int row0; // Offset from the bottom of FPA. + // Astrometric transformations + psPlaneTransform* toFPA; // Transformation from chip to FPA coordinates + psPlaneTransform* fromFPA; // Transformation from FPA to chip coordinates + // Information + psMetadata *concepts; // Cache for PS concepts + psMetadata *analysis; // Chip-level analysis metadata + psArray *cells; // The cells (referred to by name) + pmFPA *parent; // Parent FPA + bool valid; // Do we bother about reading and working with this chip? + p_pmHDU *private; // FITS data +} pmChip; +\end{verbatim} + +The constructor for \code{pmChip} shall be: +\begin{verbatim} +pmChip *pmChipAlloc(pmFPA *fpa, psString name); +\end{verbatim} +The constructor shall make an empty \code{pmChip}. If the parent +\code{fpa} is not NULL, the \code{parent} link is made and the chip +shall be placed in the parent's array of \code{chips}. The +\code{cells} array shall be allocated with a zero size, and the +metadata containers constructed. The \code{name} shall be used to set +\code{CHIP.NAME} in the \code{concepts}. All other pointers in the +structure shall be initialized to \code{NULL}. + +\subsubsection{A Focal Plane} + +A focal plane consists of one or more chips (according to the number +of pieces of contiguous silicon). It contains metadata containers for +the concepts and analysis, a link to the parent, and pointers to the +FITS header, if that corresponds to this level (the FPA may be the +PHU, but will not ever contain pixels). For astrometry, it contains a +transformation from the focal plane to the tangent plane and the fixed +pattern residuals. It is expected that the transformation will +consist of two 4D polynomials (i.e.\ a function of two coordinates in +position, the magnitude of the object, and the color of the object) in +order to correct for optical distortions and the effects of the +atmosphere; hence we think that it is prudent to include a reverse +transformation which will be derived from numerically inverting the +forward transformation. + +\begin{verbatim} +typedef struct { + // Astrometric transformations + psPlaneDistort* fromTangentPlane; // Transformation from tangent plane to focal plane + psPlaneDistort* toTangentPlane; // Transformation from focal plane to tangent plane + psProjection *projection; // Projection from tangent plane to sky + // Information + psMetadata *concepts; // Cache for PS concepts + psMetadata *analysis; // FPA-level analysis metadata + const psMetadata *camera; // Camera configuration + psArray *chips; // The chips + p_pmHDU *private; // FITS data + psMetadata *phu; // Primary Header +} pmFPA; +\end{verbatim} + +The constructor for \code{pmFPA} shall be: +\begin{verbatim} +pmFPA *pmFPAAlloc(const psMetadata *camera); +\end{verbatim} +The constructor shall make an empty \code{pmFPA}. The \code{chips} +array shall be allocated with a zero size, the \code{camera} and +\code{db} pointers set to the values provided, and the \code{concepts} +metadata constructed. All other pointers in the structure shall be +initialized to \code{NULL}. + + +The inclusion of hierarchical links pointing both down (via the +arrays) and up (via the \code{parent}) could make for difficulties. +For this reason, we specify a utility function to manage the +collection of upward-pointing links: +\begin{verbatim} +bool pmFPACheckParents(pmFPA *fpa); +\end{verbatim} +This function checks the validity of the \code{parent} links in the +FPA hierarchy. If a \code{parent} link is not set (or not set +correctly), it is corrected, and the function shall return +\code{false}. If all the \code{parent} pointers were correct, the +function shall return \code{true}. + +\subsection{Detector Coordinate Transformations} + +\begin{figure} +\psfig{file=CameraLayout,width=5.5in} +\caption{Camera Pixel Layout\label{CameraLayout}} +\end{figure} + +These container levels also include in their definition the +information needed to transform the coordinates in one of the levels +to the coordinate system relevant at the higher levels. + +The data structures define the basic coordinate relationships between +all of these data elements. A set of offsets for each level in the +data hierarchy specifies the location of the particular set of pixels +in the next level of the hierarchy. This is illustrated in +Figure~\ref{CameraLayout}. These offsets may be used to define the +complete camera layout in the approximating assumption that all pixels +in the camera are laid out on a single linear pixel grid. This +approximate is sufficient for many basic operations. For more detail, +the precise astrometric relationship between each level of the +hierarchy may also be made available in the metadata of the data +structures. + +In practice, a single readout from a detector may represent only a +subset of the complete set of pixels addressed by the {\it cell}. The +readout may also have binning applied in both of the two dimensions. +There may also be overscan and pre-scan regions in the set of pixels. +Finally, the readout direction is not always the same for all detector +amplifiers. As shown in Figure~\ref{CameraLayout}, these different +concepts are represented in the data hierarchy. The coordinate of the +origin of the data grid for one level of the hierarchy in the grid of +the containing hierarchy is defined for each data level. For example, +the origin of the coordinates for a single chip are located in the +camera grid at \code{pmChip.cell0,row0}. The \code{pmReadout} data +level has additional information to specify the details of the readout +process. The elements \code{pmReadout.colBins,rowBins} specify the +binning factor in the two dimensions, while the sign indicates the +parity of the specific readout (readout direction). Note that the +value of \code{pmReadout.col0,row0} must be assigned in such a way +that it represents the coordinate of the origin pixel in the actual +image: the overscan or pre-scan pixels must be accounted for. Putting +all of these element together, we can see that the pixel coordinates +in the camera grid may be determined from the pixel coordinates in the +image grid from the following relationship: + +\begin{verbatim} +pmFPA(cell,row) = pmChip.cell0,row0 + pmCell.cell0,row0 + pmReadout.cell0,row0 + + pmReadout.cellParity,rowParity * pmReadout.cellBins,rowBins * + psImage.data(cell,row) +\end{verbatim} + +\subsection{Input/Output of a Focal Plane Hierarchy} + +We specify two functions to construct a focal plane hierarchy from a +camera configuration and read from a FITS file. These two operations +are decoupled so that the big investment of memory from the read only +occurs when it is necessary. + +\begin{prototype} +pmFPA *pmFPAConstruct(const psMetadata *camera, psDB *db); +bool pmFPARead(pmFPA *fpa, psFits *fits); +\end{prototype} + +\code{pmFPAConstruct} shall construct a focal plane hierarchy from a +\code{camera} configuration. A \code{db} handle is also provided so +that may be set in the \code{pmFPA}. The resultant \code{pmFPA} and +its lower-down components shall be ready for to read a FITS file into +it by setting the \code{extname} pointers at the appropriate levels to +the appropriate FITS extension name. + +\code{pmFPARead} shall read a \code{fits} file (the contents of which +are described by the previous \code{camera} configuration) into an +extant \code{fpa}. This involves reading the headers and pixels, as +well as ingesting all the concepts. + +\begin{prototype} +bool pmFPASelectChip(pmFPA *fpa, int chipNum); +int pmFPAExcludeChip(pmFPA *fpa, int chipNum); +\end{prototype} + +These functions are provided to set the \code{valid} booleans within +an \code{fpa} so that only certain chips within the FITS file are read +in. + +\code{pmFPASelectChip} shall set \code{valid} to \code{true} for the +specified chip number (\code{chipNum}), and all other chips shall have +\code{valid} set to \code{false}. In the event that the specified +chip number does not exist within the \code{fpa}, the function shall +return \code{false}. + +\code{pmFPAExcludeChip} shall set \code{valid} to \code{false} only +for the specified chip number (\code{chipNum}). In the event that the +specified chip number does not exist within the \code{fpa}, the +function shall generate a warning, and perform no action. The +function shall return the number of chips within the \code{fpa} that +have \code{valid} set to \code{true}. + +\tbd{make these functions richer: select by extention, extname, cell, +options to invalidate all / validate all, etc} + +\begin{prototype} +bool pmFPAMorph(pmFPA *toFPA, pmFPA *fromFPA, bool positionDependent, int chipNum, int cellNum); +\end{prototype} + +\code{pmFPAMorph} shall morph the \code{fromFPA} focal plane hierarchy +to the \code{toFPA} focal plane hierarchy. This allows us to write +the pixels out using a different (though consistent) camera +configuration. In the event that the \code{toFPA} has different +levels than the \code{fromFPA}, only the chip and cell specified by +\code{chipNum} and \code{cellNum} shall be written; if the levels are +the same, these numbers are ignored. This function shall break apart +pixel regions or splice them together where required in order to +satisfy the demands of the \code{toFPA}. How the image and overscan +regions are spliced together depends on the value of +\code{positionDependent}: If \code{positionDependent} is \code{true}, +then the overscan regions go to the low end of the spliced image if $i +< N/2$, and to the high end of the spliced image if $i >= N/2$, where +$i$ (zero-offset) is the cell number, and $N$ is the total number of +cells; if \code{positionDependent} is \code{false}, then the overscan +regions all go to the high end of the spliced image. Care should be +taken to check the \code{CELL.READDIR}, so the orientation of the +overscan regions is known. If the bias and trim sections are +specified by headers in the \code{toFPA}, these shall be updated +appropriately; otherwise, the function is permitted to fail with a +suitable error message, in which case it shall return \code{false}. + +\begin{prototype} +bool pmFPAWrite(psFits *fits, pmFPA *fpa); +\end{prototype} + +\code{pmFPAWrite} shall write the focal plane hierarchy, \code{fpa}, +to the specified \code{fits} file, returning \code{true} upon success +and \code{false} otherwise. The \code{fpa} should contain sufficient +information with which to write the FITS images. + +\begin{prototype} +pmFPAWriteMask(psFits *fits, pmFPA *fpa); +\end{prototype} + +\code{pmFPAWriteMask} is very similar to \code{pmFPAWrite}, but it +shall write the \code{mask} elements of the \code{pmReadout}s +comprising the \code{fpa}. + +\begin{prototype} +pmFPAWriteWeight(psFits *fits, pmFPA *fpa); +\end{prototype} + +\code{pmFPAWriteWeight} is very similar to \code{pmFPAWrite}, but it +shall write the \code{weight} elements of the \code{pmReadout}s +comprising the \code{fpa}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Astrometry} + +Astrometry is a basic functionality required for the IPP that will be +used repeatedly, both for low-precision (roughly where is my favorite +object?) and high-precision (what is the proper motion of this star?). +As such, it must be flexible, yet robust. + +\subsection{Coordinate frames} +\label{sec:coordinateFrames} + +There are five coordinate frames that we need to worry about for the +purposes of astrometry: +\begin{itemize} +\item Cell: $(x,y)$ in pixels --- raw coordinates; +\item Chip: $(X,Y)$ in pixels --- the location on the silicon; +\item Focal Plane: $(p,q)$ in microns --- the location on the focal plane; +\item Tangent Plane: $(l,m)$ in arcsec from the telescope boresight; and +\item Sky: (RA,Dec) --- ICRS. +\end{itemize} + +The following steps are required to convert from the cell coordinates to +the sky: +\begin{itemize} +\item Cell $\longleftrightarrow$ Chip: two 2D polynomials, $(X,Y) = f(x,y)$; +\item Chip $\longleftrightarrow$ FP: two 2D polynomials, $(p,q) = g(X,Y)$; +\item FP $\longleftrightarrow$ TP: two 4D polynomials, $(l,m) = +h(p,q,m,c)$, where $m$ and $c$ are the magnitude and color of the +object, respectively; and +\item TP $\longleftrightarrow$ Sky: transformation to the sky using +pre-computed coefficients for each pointing. +\end{itemize} + +Note that the transformation between the Focal Plane and the Tangent +Plane is a four-dimensional polynomial, in order to account for any +possible dependencies in the astrometry on the stellar magnitude and +color; the former serves as a check for charge transfer +inefficiencies, while the latter will correct chromatic refraction, +both through the atmosphere and the corrector lenses. + +We require structures to contain each of the above transformations as +well as the pixel data. + +\subsection{Position Finding} + +We require functions to return the structure containing given +coordinates. For example, we want the chip that corresponds to the +focal plane coordinates $(p,q) = (-1.234,+5.678)$. These routines +handle the one-to-many problem --- i.e., for one given focal plane +coordinate, there are many chips that this coordinate may be +correspond to; these functions will select the correct one. +% +\begin{verbatim} +pmCell *pmCellInFPA (const psPlane *coord, const pmFPA *fpa); +pmChip *pmChipInFPA (const psPlane *coord, const pmFPA *fpa); +pmCell *pmCellInChip(const psPlane *coord, const pmChip *chip); +\end{verbatim} + +\subsection{Conversion Functions} + +We require functions to convert between the various coordinate frames +(Section~\ref{sec:coordinateFrames}). The hierarchy of the coordinate +frames and the transformations between each are shown in +Figure~\ref{fig:coco}. The functions that employ the transformations +are shown in Figure~\ref{fig:cocoFunc}. In addition to +transformations between each adjoining coordinate frame in the +hierarchy, we also require higher-level functions to convert between +the Cell and Sky coordinate frames; these will simply perform the +intermediate steps. + +\begin{figure} +\psfig{file=../pslib/coordinateFrames,height=7in,angle=-90} +\caption{The coordinate systems in the \PS{} IPP, and the relation +between each by transformations contained in the appropriate +structures.} +\label{fig:coco} +\end{figure} + +\begin{figure} +\psfig{file=../pslib/coordinateConv,height=7in,angle=-90} +\caption{Conversion between coordinate systems by PSLib.} +\label{fig:cocoFunc} +\end{figure} + +We specify the following functions to convert between coordinates in +one type of frame to another type of frame. The first group consist +of unambiguous transformations: from the coordinates in a low-level +frame to the coordinates in the containing higher-level frame, of +which only one exists. In all of these functions, the output +coordinate structure may be \code{NULL} or may be supplied by the +calling function. In the former case, the structure must be +allocated; in the latter case, the supplied structure must be used. + +\begin{verbatim} +psPlane *pmCoordCellToChip (psPlane *out, const psPlane *in, const pmCell *cell); +% astrometry comes from cell (no need for parent) +\end{verbatim} +which converts coordindates \code{in} on the specified \code{cell} to +the coordinates on the parent chip. + +\begin{verbatim} +psPlane *pmCoordChipToFPA (psPlane *out, const psPlane *in, const pmChip *chip); +% astrometry comes from chip (no need for parent) +\end{verbatim} +which converts the coordinates \code{in} on the specified \code{chip} +to the coordinates on the parent FPA. + +\begin{verbatim} +psPlane *pmCoordFPAToTP(psPlane *out, const psPlane *in, float color, float mag, const pmFPA *fpa); +% astrometry comes from FPA (no need for parent) +\end{verbatim} +which converts coordinates \code{in} on the specified focal plane +\code{fpa} to tangent plane coordinates, applying the appropriate +distortion terms. The \code{color} and magnitude (\code{mag}) of the +source is necessary in order to perform the distortion between the +focal plane and the tangent plane. + +\begin{verbatim} +psSphere *pmCoordTPToSky(psSphere *out, const psPlane *in, const psProjection *projection); +\end{verbatim} +which converts the tangent plane coordinates \code{in} to (RA,Dec) on +the sky, using the specified \code{projection}. + +% astrometry comes from cell +\begin{verbatim} +psPlane *pmCoordCellToFPA(psPlane *out, const psPlane *in, const pmCell *cell); +\end{verbatim} +which performs the single-step conversion between Cell coordinates +\code{in} and FPA coordinates. + +% astrometry comes from cell,chip,fpa (PARENT IS NEEDED HERE) +\begin{verbatim} +psSphere *pmCoordCellToSky(psSphere *out, const psPlane *in, float color, float mag, const pmCell *cell); +\end{verbatim} +which converts coordinates on the specified cell to (RA,Dec). This +transformation must be performed using the intermediate stage +transformations of Cell to Chip, Chip to FPA, FPA to Tangent Plane, +Tangent Plane to Sky. The information needed for each of these +transformations is available in the \code{.parent} elements of +\code{pmCell} and \code{pmChip}, and the \code{pmFPA.projection}. The +\code{color} and magnitude (\code{mag}) of the source is necessary in +order to perform the distortion between the focal plane and the +tangent plane. + +% astrometry comes from cell (no need for parent) +\begin{verbatim} +psSphere *pmCoordCellToSkyQuick(psSphere *out, const psPlane *in, const pmCell *cell); +\end{verbatim} +which uses the 'quick-and-dirty' transformation to convert coordinates +on the specified cell to (RA,Dec). This transformation should use the +locally linear transformation specified by the element +\code{pmCell.toTP}. Although the accuracy of this transformation +is lower than the complete transformation above, the calculation is +substantially faster as it only involves linear transformations. + +The following functions convert from high-level frames to the +coordinates of contained lower-level frames. + +\begin{verbatim} +psPlane *pmCoordSkyToTP(psPlane *out, const psSphere *in, const psProjection *projection); +\end{verbatim} +which converts (RA,Dec) coordinates \code{in} to tangent plane coords +using the specified \code{projection}. + +\begin{verbatim} +psPlane *pmCoordTPToFPA(psPlane *out, const psPlane *in, float color, float mag, const pmFPA *fpa); +\end{verbatim} +which converts the tangent plane coordinates \code{in} to focal plane +coordinates. The \code{color} and magnitude (\code{mag}) of the +source is necessary in order to perform the distortion between the +focal plane and the tangent plane. + +\begin{verbatim} +psPlane *pmCoordFPAToChip (psPlane *out, const psPlane *in, const pmChip *chip); +\end{verbatim} +which converts the specified FPA coordinates \code{in} to the +coordinates on the given Chip. The specified chip need not contain +the input coordinate. To find the chip which contains a particular +coordinate, the function \code{pmChipInFPA}, defined above, should be +used. + +\begin{verbatim} +psPlane *pmCoordChipToCell (psPlane *out, const psPlane *in, const pmCell *cell); +\end{verbatim} +which converts the specified Chip coordinate \code{in} to the +coordinate on the given Cell. The specified Cell need not contain the +input coordinate. To find the cell which contains a particular +coordinate, the function \code{pmCellInChip}, defined above, should be +used. + +\begin{verbatim} +psPlane *pmCoordSkyToCell(psPlane *out, const psSphere *in, float color, float mag, pmCell *cell); +\end{verbatim} +which directly converts (RA,Dec) \code{in} to coordinates on the +specified cell. The specified cell need not contain the input +coordinates. The \code{color} and magnitude (\code{mag}) of the +source is necessary in order to perform the distortion between the +focal plane and the tangent plane. + +\begin{verbatim} +psPlane *pmCoordSkyToCellQuick(psPlane *out, const psSphere *in, pmCell *cell); +\end{verbatim} +which directly converts (RA,Dec) \code{in} to coordinates on the +specified cell. The specified cell need not contain the input +coordinates. This transformation should use the locally linear +transformation specified by the element \code{pmCell.toTP}. +Although the accuracy of this transformation is lower than the +complete transformation above, the calculation is substantially faster +as it only involves linear transformations. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{FITS World Coordinate System} + +The FITS World Coordinate System (WCS) headers are commonly employed +with astronomical images in order to relate pixels to celestial (or +otherwise) coordinates. Since it is a FITS standard, we must be able +to read and write from WCS into our internal format. For the time +being, we will consider only celestial WCS (i.e., no spectral +wavelength calibrations, etc). Because WCS does not support the +multiple layers that we have built for \PS{}, we will use a simple +internal representation: a transformation, which handles any +distortions (i.e., goes directly from the coordinate frame of the +image to the tangent plane); and the projection. + +\begin{verbatim} +bool pmAstrometryReadWCS(psPlaneTransform **transform, // Output transformation + psProjection **projection, // Output projection + psMetadata *header // Input FITS header + ); +bool pmAstrometryWriteWCS(psMetadata *header, // Output FITS header + psPlaneTransform *transform, // Input transformation + psProjection *projection, // Input projection + double color, // Mean color to use + double magnitude, // Mean magnitude to use + ); +bool pmAstrometrySimplify(psPlaneTransform **transform, // Output transformation + psProjection **projection, // Output projection + pmCell *cell // Cell for which to generate transform and projection + ); +\end{verbatim} + +\code{pmReadAstrometry} shall parse the specified FITS \code{header}, +returning new instances of the \code{transform} and \code{projection} +that represent the WCS. The function shall return \code{true} if it +was able to successfully generate the outputs; otherwise it shall +return \code{false}. + +\code{pmWriteAstrometry} shall add WCS keywords to the supplied FITS +\code{header} that implement the given \code{transform} and +\code{projection}. The function shall return \code{true} if it was +able to successfully generate the output; otherwise it shall return +\code{false}. + +\code{pmSimplifyAstrometry} shall take a \code{cell} and simplify the +internal astrometric representation (\code{cell->toFPA} or equivalent, +\code{cell->parent->parent->toTangentPlane} and +\code{cell->parent->parent->grommit}) to a single \code{transform} and +\code{projection}. This allows the subsequent use of +\code{pmWriteAstrometry} in the case that we have only the +multi-layered \PS{} internal astrometric representation. The function +shall return \code{true} if it was able to successfully generate the +output; otherwise it shall return \code{false}. + Index: /tags/ipp-docs/MSDR-08/doc/modules/ChangeLogSDRS.tex =================================================================== --- /tags/ipp-docs/MSDR-08/doc/modules/ChangeLogSDRS.tex (revision 22201) +++ /tags/ipp-docs/MSDR-08/doc/modules/ChangeLogSDRS.tex (revision 22201) @@ -0,0 +1,138 @@ +%%% $Id: ChangeLogSDRS.tex,v 1.32 2005-09-13 21:05:23 eugene Exp $ + +\subsection{Changes from version 00 (16 August 2004) to version 01 (12 October 2004)} + +\begin{itemize} +\item clarified the image offsets for pmFlatField () +\item changed return value to bool. +\item added \code{pmCameraFromHeader} +\item added \code{pmCameraValidateHeaders} +\item added \code{pmFPAfromHeader} +\item Added \code{pmReadoutCombine} +\end{itemize} + +\subsection{Changes from version 01 (12 October 2004) to version 02 (30 November 2004)} + +\begin{itemize} +\item \code{nBin} in \code{pmSubtractBias} is also interpreted as the + number of spline pieces if spline fitting is specified. +\item Refined \code{pmReadoutCombine} specification in response to bug 227. +\item added details to the functions \code{pmCameraFromHeader}, \code{pmCameraValidateHeaders}, and \code{pmFPAfromHeader}. + +\item reorganiztion: placed configuration section up front, camera layout next, etc +\item added details about configuration system +\item added utility modules \code{pmConfigLoadSite}, \code{pmConfigLoadCamera}, \code{pmConfigLoadRecipe} +\item added utility modules \code{pmConfigLookupSTR}, \code{pmConfigLookupS32}, \code{pmConfigLookupF64}, \code{pmConfigLookupRegion}, +\item added discussion about Coordinate transforms +\item added discussion about \code{pmReadoutLoad} +\item added module \code{pmSubtractSky} +\end{itemize} + +\subsection{Changes from version 02 (30 November 2004) to version 03 (21 January 2005) } + +\begin{itemize} +\item Fixed up specification of \code{fitSpec} for \code{pmSubtractSky}. +\item Added a \code{mask} to \code{pmSubtractSky}, and specified that binned pixels which are clipped may be interpolated over, or simply ignored. +\item Added further explanation for \code{pmReadoutCombine}. +\item Added Object Detection section +\item Added PSPhot pseudo-C example +\end{itemize} + +\subsection{Changes from version 03 (21 January 2005) to version 04 (14 February 2005)} + +\begin{itemize} +\item changed entries of the form psXXX to pmXXX (object section) +\item added enum psSourceType +\item Specified appropriate image types for the phase 2 modules (bug 258). +\item clarified pmSourceRoughClass +\item clarified pmSourceAddModel, pmSourceSubModel +\end{itemize} + +\subsection{Changes from version 04 (14 February 2005) to version 05 (21 March 2005)} + +\begin{itemize} +\item Added section on image combination +\item Added section on image subtraction +\end{itemize} + +\subsection{Changes from version 05 (21 March 2005) to version 06 (27 April 2005)} + +\begin{itemize} +\item changed \code{pmFindImagePeaks} to return an array, not a list +\item replaced \code{pmCullPeaks} with \code{pmPeaksSubset} which returns a new array +\item changed \code{pmModel} to use vectors for params and dparms. +\item added nDOF and nIter to pmModel +\item changed models to return psF64, not psF32, to match psMinimizeLMChi2Func +\end{itemize} + +\subsection{Changes from version 06 (27 April 2005) to version 07 (15 July 2005)} + +\begin{itemize} +\item Changed \code{psRegion *region} to \code{psRegion region} in + prototypes (passed by value instead of by reference). +\item \code{pmSourceMoments} does not require image parameter. +\item Added \code{masks} to \code{pmRejectPixels}. +\item Added \code{size} and \code{spatialOrder} to \code{pmSubtractionKernels}. +\item added Image Hierarchy section from psLibSDRS +\item added photometry section from psLibSDRS +\item added object function abstractions to Objects +\item modified pmSource to include modelPSF and modelFLT +\item Major changes to configuration. Modifyied \code{pmConfig} + functions, and \code{pmCameraFromHeader}, and added various other + functions. +\item Modifying \code{psReadout, psCell, psChip, psFPA} structures. +\item Removing \code{psObservatory, psExposure, psGrommit} which were + centered on slalib. +\item Added \code{pmFPAConstruct, pmFPARead, pmFPAMorph, pmFPAWrite}. +\end{itemize} + +\subsection{Changes from version 07 (15 July 2005) to version 08 (13 Sept 2005)} + +\begin{itemize} +\item Added \code{bias} sections to \code{pmReadout}. +\item Added \code{CELL.READDIR} to specify read direction; \code{pmCellGetReaddir}. +\item \code{pmFPAMorph} specified, ready for coding. +\item \code{pmConfigRead} shall call \code{psTimeInitialize}, \code{psLogSetLevel}, + \code{psLogSetFormat}, \code{psTraceSetLevel}. +\item Added \code{pmConfigDB}. +\item Changed \code{pmNonLinearityLookup} to read a \code{psLookupTable}. +\item Changed input types for \code{pmMaskBadPixels}. +\item Added \code{pmFPAWriteMask}. +\item \code{pmMaskBadPixels} shall grow the mask for saturated by 1, in addition + to the explicit grow. +\item Added section on ``Paper Trail'' to Phase 2 functions. +\item Adding log destination to \code{pmConfigRead}. +\item Changing details of focal plane hierarchy. +\item Concepts are evaluated at ingest by \code{pmFPARead}. +\item Modified \code{pmSubtractBias}. +\item Remove mask from \code{pmFlatField} (mask is contained in the readout). + +\item cleaned up / extended discussion of FITS and the FPA hierarchy. +\item added \code{CONCEPT.DEFAULT} to \code{DEFAULTS} table. +\item added \code{psArgumentVerbosity} requirement to \code{pmConfigRead}. + +\item substantial changes to the Objects section: +\begin{itemize} + \item added \code{psphotMaskValues}? + \item added SN to \code{pmMoments} + \item added \code{pmPSFClump} + \item modified \code{pmSourceType} enum list + \item added radius to \code{pmModel} + \item added \code{pmModelGroup} + \item added several Model Group functions + \item added \code{pmPSF} and related functions + \item added \code{pmPSFtry} and related functions + \item added \code{pmSourceDefinePixels} + \item modified \code{pmSourceLocalSky} API + \item modified \code{pmSourceMoments} API + \item added \code{pmSourcePSFClump} + \item modified \code{pmSourceRoughClass} API + \item dropped \code{pmSourceSetPixelsCircle} (replaced with + \code{pmSourceDefinePixels} and the image mask functions. + \item added \code{pmModelFitStatus} + \item added \code{pmSourcePhotometry} + \item added \code{pmSourceDophotType} + \item added \code{pmSourceSextractType} + \item moved discussion of the object models to an appendix +\end{itemize} +\end{itemize} Index: /tags/ipp-docs/MSDR-08/doc/modules/Makefile =================================================================== --- /tags/ipp-docs/MSDR-08/doc/modules/Makefile (revision 22201) +++ /tags/ipp-docs/MSDR-08/doc/modules/Makefile (revision 22201) @@ -0,0 +1,21 @@ +# $Id: Makefile,v 1.5 2005-07-15 00:51:18 price Exp $ + +PDFLATEX = env TEXINPUTS=../../latex/inputs:$(TEXINPUTS):.: pdflatex +PSLATEX = env TEXINPUTS=../../latex/inputs:$(TEXINPUTS):.: latex + +all : ModulesSDRS.pdf + +ModulesSDRS.pdf: ChangeLogSDRS.tex CameraImages.tex + +%.pdf: %.tex + $(PSLATEX) $*.tex + $(PSLATEX) $*.tex + dvips -z -t letter -o $*.ps $*.dvi + ps2pdf $*.ps $*.pdf + @rm -f $*.ps $*.dvi $*.aux $*.log $*.tbr $*.tbd $*.lof $*.toc body.tmp head.tmp + +clean : + $(RM) *.log *.dvi *.aux *.toc *.tbd *.tbr *.lof *.out *~ core body.tmp head.tmp + +empty : clean + Index: /tags/ipp-docs/MSDR-08/doc/modules/ModulesSDRS.tex =================================================================== --- /tags/ipp-docs/MSDR-08/doc/modules/ModulesSDRS.tex (revision 22201) +++ /tags/ipp-docs/MSDR-08/doc/modules/ModulesSDRS.tex (revision 22201) @@ -0,0 +1,3578 @@ +%%% $Id: ModulesSDRS.tex,v 1.58 2005-09-13 21:05:23 eugene Exp $ +\documentclass[panstarrs,spec]{panstarrs} + +% basic document variables +\title{Pan-STARRS PS-1 Image Processing Pipeline Modules} +\subtitle{Supplementary Design Requirements} +\shorttitle{Modules SDRS} +\author{Paul Price, Eugene Magnier} +\audience{Pan-STARRS PMO} +\group{Pan-STARRS Algorithm Group} +\project{Pan-STARRS Image Processing Pipeline} +\organization{Institute for Astronomy} +\version{08} +\docnumber{PSDC-430-012} + +\begin{document} +\maketitle +\sloppy + +% -- Revision History -- +% provide explicit values for the old versions +% use '\theversion' for the current version (set above) +% use \hline between each table row +\RevisionsStart +% version Date Description +DR & 2004 Jun 7 & Draft \\ \hline +00 & 2004 Aug 16 & final for cycle 3 \\ \hline +01 & 2004 Oct 12 & draft for cycle 4 \\ \hline +02 & 2004 Nov 30 & final for cycle 4 \\ \hline +03 & 2005 Jan 21 & draft for cycle 5 \\ \hline +04 & 2005 Feb 14 & final for cycle 5 \\ \hline +05 & 2005 Mar 21 & draft for cycle 6 \\ \hline +06 & 2005 Apr 27 & final for cycle 6 \\ \hline +07 & 2005 Jul 15 & final for cycle 7 \\ \hline +08 & 2005 Sep 13 & final for cycle 8 \\ \hline +\RevisionsEnd + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\DocumentsInternal +PSCD-230-001 & PS-1 Design Reference Mission \\ \hline +PSDC-430-004 & Pan-STARRS PS-1 IPP C Code Conventions \\ \hline +PSDC-430-005 & Pan-STARRS PS-1 IPP Software Requirements Specification \\ \hline +PSDC-430-006 & Pan-STARRS PS-1 IPP Algorithm Design Document \\ \hline +PSDC-430-011 & Pan-STARRS PS-1 IPP System/Subsystem Design Description \\ \hline +\DocumentsExternal +Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ +\DocumentsEnd + +\tableofcontents +\pagebreak +\pagenumbering{arabic} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Introduction} + +This document describes the Pan-STARRS Image Processing Pipeline (IPP) +data analysis Modules. The Modules use the functionality of the +Pan-STARRS Library (PSLib) to perform more complex tasks, especially +tasks which require assumptions of astronomical analysis or the data +organization. Within the IPP, the Modules are tied together into +programs which perform complete data analysis tasks (an ``analysis +stage''). The modules may be tied together within a C framework or +using a high-level scripting language. Bindings of the Modules are +made available to the scripting language using the program SWIG. + +In order to preserve name space, globally-visible structures and +functions shall be prefixed with \code{pm}, for ``Pan-STARRS +Modules''. + +\section{Runtime Configuration Data} + +PSLib defines a \code{psMetadata} structure which can carry labeled +data of arbitrary types. The associated functions implemented by +PSLib consist of tools to manipulate and extract data from +\code{psMetadata} collections. A particular application of the +\code{psMetadata} structure within PSLib is to carry the data from a +FITS header. Other general-purpose information is also carried with +the structure. Functions are available to fill a \code{psMetadata} +collection from a text-based configuration file using a human-readable +syntax, and to fill a \code{psMetadata} collection from a properly +formatted XML document. + +In the IPP Modules, we use \code{psMetadata} collections to carry +run-time configuration data used by the data analysis modules. Below, +in the discussion of the various modules, this configuration +information is defined by specifying the name of the data item of +interest, the conceptual meaning of that data item, and the allowed +values for the data item. In this section, we discuss top-level +concepts related to the configuration information, including the +sources of the run-time configuration data and special +operations used to extract information from the configuration system. + +\subsection{Configuration Data Sources} + +All modules need to load some configuration information defining +parameters which may be configured at run-time. We break these +parameters down into three levels: +\begin{itemize} +\item Options for the particular site installation of the + pipeline: the {\it site}; +\item Options specifying the instrument setup, and in particular the + format of the FITS file: the {\it camera}; and +\item Options specifying the particular parameter choices that affect + the details of an analysis: the {\it recipe}. +\end{itemize} +Note that these are arranged in an hierarchical order, with the site +configuration being the most general, and the recipe configuration the +most specific. For example, not all sites will have to deal with all +cameras, and different cameras may require different recipes at +different times according to their particular quirks, analysis +experimentations, or their evolution. + +Each of the levels will have a metadata configuration file. In the +case of the site configuration, the filename shall be that specified +by the \code{-site} option on the command line if provided, the +environment variable \code{PS_SITE}, if defined, or \code{~/.ipprc} +otherwise. The camera configuration shall be specified by the +\code{-camera} option on the command line if provided, or shall be +inferred from a FITS header (more detail below). The recipe +configuration shall be specified by the \code{-recipe} option on the +command line if provided, or from the the camera configuration (more +detail below). + +\subsection{Configuration Files} + +\subsubsection{Site Configuration} + +The site configuration file must contain the following: +\begin{itemize} +\item The database configuration: + \begin{itemize} + \item \code{DBSERVER} of type \code{STR}: The database host name for + \code{psDBInit}. + \item \code{DBUSER} of type \code{STR}: The database user name for + \code{psDBInit}. + \item \code{DBPASSWORD} of type \code{STR}: The corresponding + database password for \code{psDBInit}. + \end{itemize} +\item \code{CAMERAS} of type \code{METADATA}: A list of instruments + that the system understands. Cameras are specified as separate + metadata entries, with the name of the camera as the key, and the + filename of the camera configuration file (of type \code{STR}) as + the data. +\end{itemize} +and may also contain the following psLib configuration options: +\begin{itemize} +\item \code{TIME} of type \code{STR}: The time configuration file (for + \code{psTimeInitialize}). +\item \code{LOGLEVEL} of type \code{S32}: The log level for + \code{psLogSetLevel}. +\item \code{LOGFORMAT} of type \code{STR}: The log format for + \code{psLogSetFormat}. +\item \code{LOGDEST} of type \code{STR}: The log destination for + \code{psLogSetDestination}. +\item \code{TRACE} of type \code{METADATA}: A list of components with + the desired trace level (of type \code{S32}) for each. +\end{itemize} + +\tbd{No doubt there is a need for better security than storing the +database password directly in the file, but we push this problem onto +the stack for now.} + +\tbd{We will add other data sources in the future, e.g., file paths, +configuration for Nebulous and DVO, etc.}. + + +An example site configuration file: + +\begin{verbatim} +### Example .ipprc file + +### Database configuration +DBSERVER STR ippdb.ifa.hawaii.edu # Database host name (for psDBInit) +DBUSER STR ipp # Database user name (for psDBInit) +DBPASSWORD STR password # Database password (for psDBInit) + +### Setups for each camera system +CAMERAS METADATA + MEGACAM_RAW STR megacam_raw.config + MEGACAM_SPLICE STR megacam_splice.config + GPC1_RAW STR gpc1_raw.config + LRIS_BLUE STR lris_blue.config + LRIS_RED STR lris_red.config +END + +### psLib setup +TIME STR time.config # Time configuration file +LOGLEVEL S32 3 # Logging level; 3=INFO +LOGFORMAT STR THLNM # Log format +LOGDEST STR STDOUT # Log destination +TRACE METADATA # Trace levels + psLib.math.psPolynomial S32 6 + psLib.image.psImageConvolve S32 2 +END +\end{verbatim} + +\subsubsection{Camera Configuration} + +The camera configuration is somewhat complicated and involved, since +it must not only specify how to translate the pixels from a FITS file +into a focal plane hierarchy (\S\ref{sec:focalplane}), but it must +also specify how to derive the various values the IPP needs +(\S\ref{sec:concepts}). Moreover, it must be able to do these for the +great variety of cameras in use in the astronomical community. + +Example camera configuration files are included in an appendix, but +below we explain the components. + +\paragraph{FITS File to Focal Plane Hierarchy} + +The Focal Plane hierarchy (\code{pmFPA, pmChip, pmCell, pmReadout}) is +explained in more detail in \S\ref{sec:focalplane}. The top level, an +FPA contains one or more chips, which correspond to a contiguous piece +of silicon. A chip contains one or more cells, which correspond to a +single amplifier. A cell contains one or more readouts, which +correspond to individual reads of the detector. + +The FITS data storage formation is a standard in the astronomical +community for storing astronomical images. A FITS file consists of an +arbitrary number of coupled human readable \code{ASCII} header +segments and binary data segments. The headers describe the format +and layout of the data segments. The first of these groups is +traditionally called the 'primary header unit' (PHU) and the rest are +referred to as 'extensions'. The header segments may contain +extensive documentary information related to the interpretation of the +data. Although the FITS format defines a standard representation of +the data, the header metadata is not so consistently defined within +the astronomical community. Also, the flexibility of the data format +means that different representations are possible for the same +fundamental collection of data. The tools presented in this section +provide a method to define and constrain the wide range of possible +FITS representations of astronomical images. + +Within the FITS data representation, there are various choices which +can and have been made for the placement of the pixels in the file. +In the simplest case, the camera consists of a single chip consisting +of a single cell always read with a single readout. In this case, the +image data could be written as part of the primary data unit. In a +more complex case with multiple chips and multiple cells, the data may +be organized in several ways. The data may be distributed into +multiple files or in multiple FITS data extensions. A single camera +image may be written as a collection of files for individual chips +with separate extensions for each cell (CFH12K.split, GPC). Another +camera may write a single file with multiple extensions for each cell +(Megacam.raw), or multiple extensions per chip, with each cell +representing portions of the chip image (Megacam.splice, CFHT-IR). + +In all of these representations, there are only two basic distinctions +in how the pixel data is stored: what level in the hierarchy the +entire FITS file corresponds to (FPA, chip, or cell), and what level +the extensions correspond to (chip, cell or no extensions at all). +Knowing these, and having a list of the components, we can construct +the focal plane hierarchy. + +Note that a single data extension, consisting of a uniform grid of +pixels, can only naturally represent a cell or a chip. In order to +represent the entire focal plane array as a single grid, some +artificial choices would be made to fill-in or ignore the gaps between +chips and their relative rotations. Within our framework, a complete +focal plane mosaic of multiple chips could be represented as a single +extension by treating the collection of pixels as if they were from a +single chip. + +To define the hierarchy, we specify the following keywords: +\begin{itemize} + +\item \code{PHU} of type \code{STR}: May be one of \code{FPA}, + \code{CHIP}, or \code{CHIP}. This specifies the focal plane level + of the Primary Header Unit, and hence the entire FITS file (the + 'class' of the file) . + +\item \code{EXTENSIONS} of type \code{STR}: May be one of \code{CHIP}, + \code{CELL} or \code{NONE}, though not of a level higher than that + specified by the \code{PHU}. This specifies what each extension + represents. + +\item \code{CONTENTS} which may be of type \code{METADATA} or + \code{STR}, depending upon the \code{PHU} and \code{EXTENSIONS}, + specifies what the contents of the FITS file are: + \begin{itemize} + \item \code{PHU=FPA, EXTENSIONS=CHIP}: Type \code{METADATA} with + the component keywords being the extension names and the values + the names of the cells, separated by commas or whitespace. + \item \code{PHU=FPA, EXTENSIONS=CELL}: Type \code{METADATA} with + the component keywords being the extension names and the values + the chip name and the cell type, separated by a colon. + \item \code{PHU=FPA, EXTENSIONS=NONE}: Type \code{METADATA} with + the component keywords being the chip names and the values the + names of the cells, separated by commas or whitespace. + \item \code{PHU=CHIP, EXTENSIONS=CELL}: Type \code{METADATA} with + the component keywords being the extension names and the values + the corresponding cell type. + \item \code{PHU=CHIP, EXTENSIONS=NONE}: Type \code{STR} with the + value being the cell types separated by commas or whitespace. + \end{itemize} +\item \code{CELLS} of type \code{METADATA} with the component keywords + being the cell names or types, each of type \code{METADATA}. Within + each cell should be specified various PS concept values appropriate + for each cell. +\end{itemize} + +An example: + +\begin{verbatim} +# How to read this data +PHU STR FPA # The FITS file represents an entire FPA +EXTENSIONS STR CELL # The extensions represent cells + +# What's in the FITS file? +CONTENTS METADATA + # Extension name, chip name:type + amp00 STR ccd00:left + amp01 STR ccd00:right + amp02 STR ccd01:left + amp03 STR ccd01:right + amp04 STR ccd02:left +END + +# Specify the cell data +CELLS METADATA + left METADATA # Left amplifier + CELL.BIASSEC STR BIASSEC + CELL.TRIMSEC STR DATASEC + CELL.PARITY S32 1 + END + right METADATA # Right amplifier + # This cell is read out in the opposite direction + CELL.BIASSEC STR BIASSEC + CELL.TRIMSEC STR [1025:2048,1:2048] + CELL.PARITY S32 -1 + END +END +\end{verbatim} + +Observe how the \code{CONTENTS} specifies the extension name, which we +know from the \code{EXTENSIONS} is a cell, and that each extension is +associated with a chip, and has a cell type. + +\paragraph{Deriving concept values} + +The PS concepts are described in more detail in \S\ref{sec:concepts}. +Basically, astronomical cameras generally store the important details +(``concepts'') in different ways. This is generally manifested in the +choice of different FITS header keywords to describe the same concept, +but one can also imagine deriving values from a database or a known +default. + +We therefore specify the following keywords: +\begin{itemize} +\item \code{TRANSLATION} of type \code{METADATA} is a translation + table for understanding PS concepts in terms of FITS headers. The + PS concept (keyword) is derived from the FITS header given in the + value. +\item \code{DATABASE} of type \code{METADATA} is a formula for + obtaining a PS concept from the database. Each component is of a + user-specified type containing \code{TABLE}, \code{COLUMN}, + \code{GIVENDBCOL} and \code{GIVENPS}. The idea is that to obtain + the value of a PS concept, one refers to a particular \code{COLUMN} + in a particular \code{TABLE}, where the value of certain PS concepts + (\code{GIVENPS}; multiple values separated by a comma or semicolon) + match certain database columns (\code{GIVENDBCOL}; multiple values + separated by a comma or semicolon). +\item \code{DEFAULTS} of type \code{METADATA} is a set of default + values of PS concepts for the camera. The PS concept (keyword) is + assigned the value. There is also limited dependency allowed; see + \S\ref{sec:concepts}. +\end{itemize} + +An example: +\begin{verbatim} +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.NAME STR EXPNUM + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR ROTANGLE + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + FPA.MJD STR MJD-OBS + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.XBIN STR CCDBIN1 + CELL.YBIN STR CCDBIN2 + CELL.SATURATION STR SATURATE +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + CELL.BAD S32 0 + CELL.PARITY.DEPEND STR CHIP.NAME + CELL.PARITY METADATA + amp00 S32 1 + amp01 S32 -1 + amp02 S32 1 + amp03 S32 -1 + END +END + +# How to translate PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS + CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME + CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME +END +\end{verbatim} + +The \code{.DEPEND} entry in the \code{DEFAULTS} will be explained in +\S\ref{sec:concepts}. + +\paragraph{Indentification by rule} +\label{sec:camerarule} + +The function \code{pmConfigCameraFromHeader} requires that the camera +configuration also contains a rule on how to recognise that a FITS +header comes from that camera. + +We therefore specify another keyword: \code{RULE} of type +\code{METADATA}: Contains a list of FITS headers keywords and values +(of the appropriate type) against which actual headers are compared to +determine if it matches the camera type. + +An example is: +\begin{verbatim} +# How to identify this type +RULE METADATA + TELESCOP STR CFHT 3.6m + DETECTOR STR MegaCam + EXTEND BOOL T + NEXTEND S32 72 +END +\end{verbatim} + +\paragraph{Recipes} + +The camera configuration file must also contain filenames for the +recipe configuration files. We include \code{RECIPES} of type +\code{METADATA} with component keywords being the various recipe names +and the values (of type \code{STR}) the corresponding recipe +configuration filename. + +An example: +\begin{verbatim} +# Recipes for LRIS +RECIPES METADATA + PHASE1 STR lris_phase1.config + PHASE2 STR lris_phase2.config + PHASE4 STR lris_phase4.config +END +\end{verbatim} + +\subsubsection{Recipe Configuration} + +\tbd{The contents of the recipe configuration file are dependent upon +the particular module, and hence are not specified here at this time.} + + +\subsection{PS Concepts} +\label{sec:concepts} + +Each image has associated with it what we will call {\it concepts} +(for want of a better word). These are values corresponding to +general quantities and qualities necessary to understand and interpret +the data such as airmass, date, read noise and filter. The values of +each of the below concepts shall be determined when the FPA is read +into memory (via \code{pmFPARead}), and stored at the appropriate +level in the focal plane hierarchy. + +Below is a list of concepts that the IPP requires, with the +expected type and a short description. + +\begin{itemize} +\item \code{FPA.AIRMASS} (F32): Airmass at which the observation is made (boresight). +\item \code{FPA.FILTER} (STR): Filter used in observation +\item \code{FPA.POSANGLE} (F32): Position angle for camera +\item \code{FPA.RA} (F64): Right Ascension of boresight in radians +\item \code{FPA.DEC} (F64): Declination of boresight in radians +\item \code{FPA.RADECSYS} (STR): System of RA,Dec (e.g., J2000 or ICRS) +\item \code{FPA.NAME} (STR): An identifier (e.g., observation number) for the FPA instance +\item \code{CHIP.NAME} (STR): The name of the chip (unique within the FPA) --- set at FITS read +\item \code{CELL.NAME} (STR): The name of the cell (unique within the parent chip) --- set at FITS read +\item \code{CELL.TIME} (\code{psTime}): Time of observation start +\item \code{CELL.READDIR} (S32): Read direction: line (1) or column (2) +\item \code{CELL.BIASSEC} (STR): Overscan region(s) +\item \code{CELL.TRIMSEC} (STR): Trim region +\item \code{CELL.GAIN} (F32): CCD gain (e/ADU) +\item \code{CELL.READNOISE} (F32): CCD read noise (e) +\item \code{CELL.SATURATION} (F32): CCD saturation point (ADU) +\item \code{CELL.BAD} (F32): CCD bad pixel point (ADU) +\item \code{CELL.XBIN} (S32): CCD binning in x +\item \code{CELL.YBIN} (S32): CCD binning in y +\item \code{CELL.XPARITY} (S32): Direction of CCD readout in x relative to the rest of the chip +\item \code{CELL.YPARITY} (S32): Direction of CCD readout in y relative to the rest of the chip +\item \code{CELL.EXPOSURE} (F32): Exposure time of image (sec) +\item \code{CELL.DARKTIME} (F32): Dark time for image (sec) +\end{itemize} + +\tbd{Note that CELL.EXPOSURE, CELL.DARKTIME and CELL.TIME should +actually be specified at the readout level. However, at this present +time, we're not sure how these should be specified, and so we move +them up to the cell level and assume that all readouts are of the same +exposure and dark time.} + +For different camera systems, these concepts are not always known by +the same name, nor are they generally obtained in the same manner, and +so their source or value must be specified in the camera configuration +file. The value of a concept shall be found by searching in the +following order: +\begin{itemize} +\item The cell data from the \code{CELLS} metadata in the camera configuration. +\item The FITS header via the \code{TRANSLATION} table. +\item The \code{DATABASE} lookup. +\item The \code{DEFAULTS} value. +\end{itemize} + +After ingest (performed by \code{pmFPARead}, the user may safely +assume that all of the above concepts exist (defined at the +appropriate level), is of the specified type and in the specified +format. + +\subsubsection{Dependencies for defaults} + +In the \code{DEFAULTS} table in the camera configuration, we allow the +specification of the concept with an additional suffix, \code{DEPEND}. +The value (of type \code{STR}) of the \code{CONCEPT.DEPEND} is the +name of a concept on which the first concept depends. For example, it +might depend on the chip name. Then the first concept becomes of type +\code{METADATA}, with the component keywords being the value of the +second concept (on which the first depends). To avoid infinite +recursion, no further dependency is permitted. We also allow an entry +\code{CONCEPT.DEFAULT} specifiying the default value of the concept if +a match is not made with the dependcency list. An example of the +dependency: + +\begin{verbatim} +# Default PS concepts that may be specified by value +DEFAULTS METADATA + CELL.GAIN.DEPEND STR CHIP.NAME + CELL.GAIN.DEFAULT STR 1.0 + CELL.GAIN METADATA + ccd00 F32 1.2 + ccd01 F32 3.4 + ccd02 F32 5.6 + END +END +\end{verbatim} + +\subsubsection{FORMATS} + +Because of the variety of methods for specifying these concepts +(especially in FITS headers), we must also specify additional +information in the camera configuration that specifies how to +interpret the data provided. These are provided in an entry +\code{FORMATS} (of type \code{METADATA}) in the camera configuration. +Within the \code{FORMATS} metadata, there is a string for each of the +concepts that requires a format to be specified. + +\paragraph{CELL.TIME} + +The time at which the shutter opens is represented in a variety of +ways in FITS files, so care must be taken to specify what the format +is in the file under consideration. Permitted values of +\code{CELL.TIME.FORMAT} are: + +\begin{itemize} +\item \code{JD}: The value pointed to by \code{CELL.TIME} is to be + interpreted as a Julian Date. +\item \code{MJD}: The value pointed to by \code{CELL.TIME} is to be + interpreted as a Modified Julian Date. +\item \code{ISO}: The value pointed to by \code{CELL.TIME} is to be + interpreted as an ISO date-time (yyyy-mm-ddThh:mm:ss.ss). +\item \code{SEPARATE}: The date and time are specified separately, and + the \code{CELL.TIME} contains the headers for the date and the time + separated by whitespace or a comma. Then it is necessary to add + additional qualifiers to specify the formats of these: + \begin{itemize} + \item \code{PRE2000}: The year is in the old style two-digit format + popular before the year 2000, and it should be assumed that the + date is in the twentieth century. + \item \code{BACKWARDS}: The date is in the format dd-mm-yyyy or + dd/mm/yyyy. + \item \code{SOD}: The time is specified as seconds-of-day. + \end{itemize} +\end{itemize} + +Note that the FITS standard is that the time in the header refers to +the {\it start} of the observation. + +\tbd{the PRE2000 and BACKWARDS qualifiers should be replace with +explicit format definitions in the form YYYY/MM/DD} + +\tbd{In the future, we might add additional qualifiers that calculate +the start time of the observation based on someone foolishly putting +the end- or mid-time in the header.} + +\tbd{Should we move CELL.TIMESYS into the format as well?} + +\paragraph{FPA.RA and FPA.DEC} + +The RA and Declination of the boresight might be specified in a few +ways. We need to specify both how the value is interpreted and the +units. \code{FPA.RA.FORMAT} and \code{FPA.DEC.FORMAT} should be one +of the following: + +\begin{itemize} +\item \code{HOURS}: The value pointed to by the concept should be + interpreted as being in hours. +\item \code{DEGREES}: The value pointed to by the concept should be + interpreted as being in degrees. +\item \code{RADIANS}: The value pointed to by the concept should be + interpreted as being in radians. +\end{itemize} + +How the value is interpreted can be determined from the type of the +header: if it is of type \code{STR}, then we can reasonably assume +that it is in sexagesimal format with colons or spaces as separators; +and if it is of type \code{F32} (or \code{F64}), then we can assume +that it is in decimal format. + +\subsubsection{Implicit format information} + +While details like the units of the right ascension in the header must +be specified explicitly, some other details can be determined from +implicit information. + +\begin{itemize} +\item \code{FPA.RA} and \code{FPA.DEC}: if the value on ingest is of +type \code{STRING}, then it may be interpreted as sexagesimal +notation, ``\code{dd:mm:ss.ss}'', or ``\code{dd:mm.mmm}''. A space +may be used instead of a colon to separate the values. Otherwise, if +the value is of a numerical type (\code{F32} or \code{F64}), then that +is the appropriate value. +\item \code{CELL.XBIN} and \code{CELL.YBIN}: if the value on ingest is +of type \code{STRING}, then it may be interpreted as ``\code{x,y}'', +where \code{x} is the binning in x, and \code{y} is the binning in y. +A space may be used instead of a comma, and there may even be a space +before or after the comma (or both). Otherwise, if the value is of a +numerical type (\code{S32}, etc), then that is the appropriate value. +\item \code{CELL.BIASSEC} and \code{CELL.TRIMSEC}: These values on +ingest should always be of type \code{STRING}. If they contain a +square bracket, then they may be interpreted as a list of standard +region specifications, ``\code{[x0:x1,y0:y1];[x2:x3,y2:y3];...}'', +where the semi-colon may be replaced by spaces. Otherwise, the string +may be interpreted as a FITS header (or headers, separated by spaces, +commas or semi-colons) that contains the appropriate values. +\end{itemize} + +\tbd{the use of implicit interpretation of formats should be + discouraged: format interpretation guides should be provided} + +\subsection{Configuration APIs} + +\begin{prototype} +bool pmConfigRead(psMetadata **site, psMetadata **camera, psMetadata **recipe, + int *argc, char **argv, const char *recipeName); +psMetadata *pmConfigCameraFromHeader(const psMetadata *site, const psMetadata *header); +psMetadata *pmConfigRecipeFromCamera(const psMetadata *camera, const char *recipeName); +\end{prototype} + +\code{pmConfigRead} shall load the \code{site} configuration +(according to the above rule for determining the source). The +\code{camera} configuration shall also be loaded if it is specified on +the command line (\code{argc, argv}); otherwise it shall be set to +\code{NULL}. The \code{recipe} shall also be loaded from the command +line (if specified) or, if the camera configuration has been loaded, +from the camera configuration and recipe specification therein (see +below). In dealing with the command line parameters, the functions +shall use the appropriate functions in psLib to retrieve and remove +the relevant options from the argument list; this simplifies +assignment of the mandatory arguments, since all the optional command +line arguments are removed leaving only the mandatory arguments. The +following psLib setups shall also be performed if they are specified +in the site configuration: +\begin{itemize} +\item the function shall call \code{psTimeInitialize} with the + configuration file specified by \code{TIME}. +\item the function shall call \code{psLogSetLevel} with the logging + level specified by \code{LOGLEVEL}. +\item the function shall call \code{psLogSetFormat} with the log + format specified by \code{LOGFORMAT}. +\item the function shall call \code{psTraceSetLevel} with the component names and + trace levels specified by the \code{TRACE}. +\end{itemize} +Note that additional log/trace command-line options may be specified +and interpretted using the \code{psArgumentVerbosity} function from +psLib. These options should (in the case of logging) override the +configuration-supplied information or (in the case of tracing) +supplement it. + +\code{pmConfigCameraFromHeader} shall load the \code{camera} +configuration based on the contents of the FITS \code{header}, using +the list of known cameras contained in the \code{site} configuration. +If more than one camera matches the FITS header, a warning shall be +generated and the first matching camera returned. + +\code{pmConfigRecipeFromCamera} shall load the \code{recipe} +configuration based on the \code{recipeName} and the list of known +recipes contained in the \code{camera} configuration (details below). + +\begin{prototype} +bool pmConfigValidateCamera(const psMetadata *camera, const psMetadata *header); +\end{prototype} + +This function, used by \code{pmConfigCameraFromHeader}, shall return +\code{true} if the FITS \code{header} matches the rule contained in +the \code{camera} configuration (see \S\ref{sec:camerarule}); +otherwise it shall return \code{false}. + +\begin{prototype} +psDB *pmConfigDB(psMetadata *site); +\end{prototype} + +\code{pmConfigDB} shall use the \code{site} configuration data to open +a database handle. \tbd{This is fairly straightforward at the moment, +but will change when we beef up security.} + +\subsubsection{Example usage} + +The following is provided as an example of how the above functions +are envisioned in use. + +\begin{verbatim} +int main(int argc, char *argv[]) +{ + // Parse other command-line arguments here + psMetadata *site = NULL; // Site configuration + psMetadata *camera = NULL; // Camera configuration + psMetadata *recipe = NULL; // Recipe configuration + if (! pmConfigRead(&site, &camera, &recipe, &argc, argv, "moduleName")) { + psLogMsg("moduleName", PS_LOG_ERROR, "Can't find site configuration!\n"); + exit(EXIT_FAILURE); + } + // Parse other command-line arguments here + + // The command-line argument list now contains only mandatory arguments + // Assume the first of these is an input image + char *imageName = argv[1]; // Name of FITS file + psFits *imageFH = psFitsOpen(imageName, "r"); // File handle for FITS file + if (! imageFH) { + psLogMsg("moduleName", PS_LOG_ERROR, "Can't open input image %s\n", imageName); + exit(EXIT_FAILURE); + } + psMetadata *header = psFitsReadHeader(NULL, imageFH); // FITS header + + if (!camera && !(camera = pmConfigCameraFromHeader(site, header))) { + psLogMsg("moduleName", PS_LOG_ERROR, "Can't find camera configuration!\n"); + exit(EXIT_FAILURE); + } + + if (! recipe && !(recipe = pmConfigRecipeFromCamera(camera, "moduleName"))) { + psLogMsg("moduleName", PS_LOG_ERROR, "Can't find recipe configuration!\n"); + exit(EXIT_FAILURE); + } + + // Now go on and do stuff + .... +} +\end{verbatim} + +\input{CameraImages.tex} + +%\input{CameraGeometry.tex} + +\section{Photometry} + +\tbd{This section is to be deferred, and for now consists only of +place holders, with no functional items.} + +Photometric observations are performed in an instrumental photometric +system, and must be related to other photometric systems. We +require a data structure which defines a photometric system, as well +as a structure to define the transformation between photometric +systems. + +The photometric system is defined by the psPhotSystem structure. +A photometric system is identified by a human-readable \code{name} +(ie, SDSS.g, Landolt92.B, GPC1.OTA32.r). Each photometric system is +given a unique identifier \code{ID}. Observations taken with a +specific camera, detector, and filter represent their own photometric +system, and it may be necessary to perform transformations between +these systems. Photometric systems associated with observations from +a specific camera/detector/filter combination can be associated with +those components. +\begin{datatype} +typedef struct { + const int ID; ///< ID number for this photometric system + const char *name; ///< Name of photometric system + const char *camera; ///< Camera for photometric system + const char *filter; ///< Filter used for photometric system + const char *detector; ///< Detector used for photometric system +} psPhotSystem; +\end{datatype} + +The following structure defines the transformation between two +photometric systems. +\begin{datatype} +typedef struct { + psPhotSystem src; ///< Source photometric system + psPhotSystem dst; ///< Destination photometric system + psPhotSystem pP, pM; ///< Primary color reference + psPhotSystem sP, sM; ///< Secondary color reference + float pA, sA; ///< Color offset for references + psPolynomial3D transform; ///< Transformation from source to destination +} psPhotTransform; +\end{datatype} + +The transformation between two photometric systems may depend on the +airmass of the observation and on the colors of the object of +interest. For a specific observation, such a transformations can be +defined as a polynomial function of the color of the star and the +airmass of the observations. If sufficient data exists, the +transformation between the photometric systems may include more than +one color, constraining the curvature of the stellar spectral energy +distributions. This latter term may be significant for stars which +are highly reddened, for example. Derived photometric quantities may +have been corrected for airmass variations, in which case only color +terms may be measurable. The structure defines the transformation +between a source photometric system (\code{src}) and a target +photometric system (\code{dst}). The photometric system of a primary +color is defined by \code{pP, pM} such that the color is constructed +as $pP - pM$. A secondary color is defined by \code{sP, sM}. For +both, a reference color is specified (\code{pA, sA}): the polynomial +transformation terms refer to colors in the form $pP - pM - pA$. The +transformation is specified as a 3D polynomial. For a star of +magnitude $M_{\rm src}$ in the source photometric system, with +additional magnitude information in the other systems $M_{\rm pP}$, +$M_{\rm pM}$, $M_{\rm sP}$, $M_{\rm sM}$, observed at an airmass of +$z$, the magnitude of the star in the target system $M_{\rm dst}$ is +given by: $M_{\rm dst} = M_{\rm src} + transform(z, M_{\rm pP} - +M_{\rm pM} - pA, M_{\rm sP} - M_{\rm sM} - sA)$. + +\section{Image Detrending} + +Image Detrending is the image analysis process wherein the +instrumental signatures are removed from the individual images. This +section discusses the modules used for image detrending. The basic +image detrending steps are: +\begin{itemize} +\item Subtract bias; +\item Correct for non-linearity; +\item Flat-field; +\item Mask bad pixels; +\item Subtract the background; +\item Mask cosmic rays; +\item Mask optical defects; +\end{itemize} + +\subsection{Bias subtraction} +\label{sec:bias} + +The bias subtraction module provides a facility to correct detector +images for the electronic pedestal introduced by the readout +electronics. + +Given an input image and various other parameters, +\code{pmSubtractBias} shall subtract the bias from the image. +The API shall be the following: +\begin{prototype} +pmReadout *pmSubtractBias(pmReadout *in, void *fitSpec, pmFit fit, bool overscan, + const psStats *stat, int nBin, + const pmReadout *bias, const pmReadout *dark); +\end{prototype} + +Three types of bias correction may optionally be performed on the +input image, \code{in}. The first is the subtraction of an overscan. +Multiple overscan regions may be specified and fit as a function of +row (or column). The second is the subtraction of a full-frame bias +image. The third is the subtraction of a suitably scaled full-frame +dark image. + +The input image, \code{in}, shall have the bias subtracted in-place. +The input image may be of type U16, S32, or F32. + +Overscan subtraction is controlled by the \code{overscan} boolean. If +it is \code{true}, then overscan subtraction is performed. The +prescan and/or overscan regions to be used are specified in +\code{in->bias} (a linked list of subimages). These shall be reduced +to a single vector (in the dimension specified by \code{CELL.READDIR}, +which is accessed via the parent of the \code{pmReadout}). + +If the overscan is not defined for each row/column, then the function +shall generate a warning and then interpolate using the provided +functional form if \code{fit} is not \code{PM_FIT_NONE} (see below); +otherwise, the function shall generate an error. + +The statistic to use in combining multiple pixels in the +prescan/overscan regions is specified by \code{stat}. \code{stat} is +of type \code{psStats} instead of simply \code{psStatsOptions} so that +clipping levels may be specified, if desired. In the event that +multiple options are specified by \code{stats}, a warning shall be +generated, and the option with the highest priority shall be used, +according to the following priority order: \code{PS_STAT_SAMPLE_MEAN}, +\code{PS_STAT_SAMPLE_MEDIAN}, \code{PS_STAT_CLIPPED_MEAN}, +\code{PS_STAT_ROBUST_MEAN}, \code{PS_STAT_ROBUST_MEDIAN}, +\code{PS_STAT_ROBUST_MODE}. + +If \code{nBin} is positive and less than the size of the vector, then +the vector shall subsequently be binned into \code{nBin} bins, using +the specified statistic (\code{stat}). For example, the whole +overscan region can be taken as a unity if \code{nBin=1}. If +\code{fit} is \code{PM_FIT_SPLINE}, then \code{nBin} also serves as +the number of spline pieces. + +\code{fit} is an enumerated type which specifies the type of fit to +employed on the overscan vector (and hence the type of \code{fitSpec}): +\begin{datatype} +typedef enum { + PM_FIT_NONE, ///< No fit + PM_FIT_POLYNOMIAL, ///< Fit polynomial + PM_FIT_SPLINE ///< Fit cubic splines +} pmFit; +\end{datatype} + +If \code{fit} is \code{PM_FIT_POLYNOMIAL} or \code{PM_FIT_SPLINE}, +then \code{fitSpec} shall be interpreted to be a structure of the +appropriate type (\code{psPolynomial1D} for \code{PM_FIT_POLYNOMIAL}, +and \code{psSpline1D} for \code{PM_FIT_SPLINE}), and the overscan +vector shall be fit using the specified functional form. + +In cases where \code{fitSpec} is \code{NULL} and \code{fit} is not +\code{PM_FIT_NONE} or \code{overscan} is \code{true}, the function +shall generate an error. If \code{overscan} is \code{false} and +\code{fit} is not \code{PM_FIT_NONE}, then the function shall generate +a warning, and no overscan subtraction shall be performed. Upon +return, the \code{fitSpec} shall contain the coefficients of the +overscan fit. + +If \code{fitSpec} is \code{NULL}, or \code{fit} is \code{PM_FIT_NONE}, +then no fit shall be performed to the overscan. + +A bias frame shall be subtracted pixel-by-pixel from the input image +if \code{bias} is non-NULL. If \code{dark} is non-\code{NULL}, then +the dark image, scaled by the ratio of dark times (from +\code{CELL.DARKTIME}) shall be subtracted pixel-by-pixel from the +input image. Note that the input image, +\code{in}, and the \code{bias} and \code{dark} frames need not be the +same size, but the function shall use the offsets in the image +(\code{in->x0} and \code{in->y0}) to determine the appropriate offsets +to obtain the correct pixel on the \code{bias}. In the event that the +\code{bias} image is too small (i.e., pixels on the input image refer +to pixels outside the range of the \code{bias} image), the function +shall generate an error. Any pixels masked in the \code{bias} or +\code{dark} shall also be masked in the output. The bias and dark +images may be copied to the same type as the input image if required. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Non-linearity} + +We here specify two functions to perform the non-linearity correction, +since either (or both) might be used to specify the correction. + +The first, \code{pmNonLinearityPolynomial} shall correct the input +image for non-linearity by replacing the flux in each pixel of the +input image, \code{in}, with the result of the specified polynomial, +\code{coeff}, acting on the flux. The API shall be the following: + +\begin{prototype} +pmReadout *pmNonLinearityPolynomial(pmReadout *in, const psPolynomial1D *coeff); +\end{prototype} + +The polynomial coefficients, \code{coeff}, will be supplied by the +caller, likely from the image metadata. + +The second function, \code{pmNonLinearityLookup} shall correct +the input image for non-linearity by using a lookup table. The API +shall be the following: + +\begin{prototype} +pmReadout *pmNonLinearityLookup(pmReadout *in, const char *filename); +\end{prototype} + +For each pixel in the input image, the function shall replace the flux +with the corresponding value from the supplied lookup table, specified +by the \code{filename}. The lookup table file shall consist of two +columns of data, the first being the original flux value and the +second being the replaced flux value. The file shall be in a format +suitable for reading by \code{psLookupTableRead}. + +Both \code{pmNonLinearityPolynomial} and \code{pmNonLinearityLookup} +shall modify the input image in-place. The input image may be of +type U16, S32, or F32. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Flat-fielding} + +Given an input image and a flat-field image, \code{pmFlatField} shall +divide the input image by the flat-field image and return it in place, +updating the mask contained within the input image as appropriate. +The API shall be the following: +\begin{prototype} +bool pmFlatField(pmReadout *in, const pmReadout *flat); +\end{prototype} + +Note that the input image, \code{in}, and the flat-field image, +\code{flat}, need not be the same size, since the input image may +already have been trimmed (following overscan subtraction), but the +function shall use the offsets of the readout (\code{in->col0, +in->row0}) and the image subarray (\code{in->image->x0, +in->image->y0}) to determine the appropriate offsets to obtain the +correct detector pixels in the flat-field image. Note that the image +offset is relative to its parent, so this offset must be followed to +the top level image which is not a child of another image and the +offsets summed. The detector pixel coordinates of pixel \code{x,y} in +a top-level image are thus \code{x + in->image->x0 + in->col0, y + +in->image->y0 + in->row0}. In the event that the \code{flat} image is +too small (i.e., pixels on the input image refer to pixels outside the +range of the \code{flat} image), the function shall generate an error. + +Pixels which are negative or zero in the \code{flat} shall be masked +in the input image with the value \code{PM_MASK_FLAT} (see +\S\ref{sec:maskValues}). Negative pixels in the \code{flat} may be +set to zero so that they are treated identically to zeroes. Any +pixels masked in the \code{flat} shall be masked with corresponding +values in the \code{output}. + +The function shall not normalize the \code{flat}; this responsibility +is left to the caller. This function is basically equivalent to a +divide (with \code{psImageOp}), but with care for the region that is +divided, checking for zero and negative pixels, and copying of the +mask from the \code{flat} to the output. + +The images in the input and flat-field readouts must both be of type +F32. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Masking} + +\subsubsection{Mask values} +\label{sec:maskValues} + +We define several mask values for use in the detrend processing: +\begin{datatype} +/** Mask values */ +typedef enum { + PM_MASK_TRAP = 0x0001, ///< The pixel is a charge trap + PM_MASK_BADCOL = 0x0002, ///< The pixel is a bad column + PM_MASK_SAT = 0x0004, ///< The pixel is saturated + PM_MASK_FLAT = 0x0008, ///< The pixel is non-positive in the flat-field +} pmMaskValue; +\end{datatype} + +Of these, masks for the charge traps need to be grown by the extent of +the OT convolution kernel. For other pixel types, orthogonal transfer +of the flux in this pixel will not (necessarily) affect the flux in +neighbouring pixels. + +\subsubsection{Bad pixels} + +Given an input image, \code{in}, a bad pixel \code{mask}, a +corresponding value in the bad pixel mask to mask in the input image, +\code{maskVal}, a saturation level, and a growing radius, +\code{pmMaskBadPixels} shall mask in the input image those +pixels in the bad pixel mask that match the value to mask. The API +shall be the following: +\begin{prototype} +pmReadout *pmMaskBadPixels(pmReadout *in, const pmReadout *mask, unsigned int maskVal, + float sat, unsigned int growVal, int grow); +\end{prototype} + +Note that the input image, \code{in}, is modified in-place. All +pixels in the \code{mask} which satisfy the \code{maskVal} shall have +their corresponding pixels masked in the input image, \code{in}. All +pixels which satisfy the \code{growVal} shall have their corresponding +pixels, along with all pixels within the \code{grow} radius masked. +Pixels which have flux greater than \code{sat} shall also be masked, +and grown by a single pixel (in addition to the \code{grow} done on +the \code{growVal}). + +\tbd{In the future, may change {\tt grow} to a convolution kernel}. + +Note that the input image, \code{in}, and the \code{mask} need not be +the same size, since the input image may already have been trimmed +(following overscan subtraction), but the function shall use the +offsets in the image (\code{in->x0} and \code{in->y0}) to determine +the appropriate offsets to obtain the correct pixel on the mask. In +the event that the \code{mask} image is too small (i.e., pixels on the +input image correspond to pixels outside the range of the \code{mask} +image), the function shall generate an error. + +The input image may be of type U16, S32 or F32. The mask image +must be of type U8. + +\subsection{Subtract sky} + +\tbd{This may be deferred.} + +Given an input image, a polynomial or spline specifying the order of a +desired fit, a binning factor and statistics to use for the binning, +along with a clipping level, \code{pmSubtractSky} shall fit and +subtract a model for the background of the image. The API shall be +the following: +\begin{prototype} +pmReadout *pmSubtractSky(pmReadout *in, psPolynomial2D *poly, psImage *mask, psU8 maskVal, + int binFactor, psStats *stats, float clipSD); +\end{prototype} + +Note that the input image, \code{in}, shall be subtracted in-place. +The function shall return the subtracted image, and also update the +polynomial, Chebyshev or spline specified by \code{fitSpec}, to hold +the coefficients used in the subtraction. + +The polynomial, \code{poly}, specifies the order of the polynomial, +and on return shall contain the coefficients of the fit. If +\code{poly} is \code{NULL}, then no fit shall be performed, and the +function shall generate a warning and return. + +When fitting the polynomial, the function shall first bin the input +image by \code{binFactor} in order to reduce the required processing +time. In the binning, pixels in the \code{mask} (if non-\code{NULL}) +which satisfy the \code{maskVal} shall be excluded. The statistic to +use in this binning is specified by \code{stat}. \code{stat} is of +type \code{psStats} instead of simply \code{psStatsOptions} so that +clipping levels may be specified, if desired. In the event that +multiple options are specified by \code{stats}, a warning shall be +generated, and the option with the highest priority shall be used, +according to the following priority order: \code{PS_STAT_SAMPLE_MEAN}, +\code{PS_STAT_SAMPLE_MEDIAN}, \code{PS_STAT_CLIPPED_MEAN}, +\code{PS_STAT_ROBUST_MEAN}, \code{PS_STAT_ROBUST_MEDIAN}, +\code{PS_STAT_ROBUST_MODE}. If the \code{binFactor} is non-positive, +or \code{stats} is \code{NULL} or fails to specify an option, a +warning shall be generated, and the fit shall be performed on the +entire image. + +Binned pixels deviating more than \code{clipSD} standard deviations +from the mean of the binned pixels shall be clipped in a single +clipping iteration before polynomial fitting. These pixels may be +interpolated over, or may be simply ignored in the fitting, according +to the choice of algorithm. If the \code{clipSD} is non-positive, +then the function shall generate a warning and not perform any +clipping. + +The \code{mask} shall be of type U8, and the input image, +\code{in}, must be of type F32. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Paper Trail} + +The elements of the focal plane hierarchy each contain an +\code{analysis} member, intended to log the results of the detrend +tasks. The detrend tasks shall add to the \code{analysis} members as +follows: + +\begin{itemize} +\item \code{pmMaskBadPixels}: + \begin{itemize} + \item \code{MASK.DONE} (STR): The time at which masking was + completed. + \item \code{MASK.SAT} (S32): The number of saturated pixels masked + in the image + \item \code{MASK.SAT.GROW} (S32): The number of additional pixels + masked by growing the saturated pixels. + \item \code{MASK.BAD} (S32): The number of pixels masked in the + image + \item \code{MASK.BAD.GROW} (S32): The number of additional pixels + masked by growing the specified bad pixels. + \end{itemize} +\item \code{pmNonLinearityPolynomial} and \code{pmNonLinearityLookup}: + \begin{itemize} + \item \code{NONLIN.DONE} (STR): The time at which the non-linearity + correction was completed. + \item \code{NONLIN.POLY} (STR): The polynomial coefficients used (if + applicable). + \item \code{NONLIN.LOOKUP} (STR): The filename for the lookup table + (if applicable). + \end{itemize} +\item \code{pmSubtractBias}: + \begin{itemize} + \item \code{BIAS.DONE} (STR): The time at which the bias-subtraction + was completed. + \item \code{BIAS.OVERSCAN.AXIS} (STR): Overscan axis used. + \item \code{BIAS.OVERSCAN.FIT.TYPE} (STR): Fit type applied to + overscan. + \item \code{BIAS.OVERSCAN.FIT.COEFF} (STR): Coefficients of overscan + fit. + \item \code{BIAS.OVERSCAN.REGION} (STR): Overscan regions (from + \code{x0,y0,numCols,numRows}). + \item \code{BIAS.OVERSCAN.BIN} (S32): Number of pixels per bin used + in overscan. + \item \code{BIAS.OVERSCAN.MEAN} (F32): The mean of the binned + overscan pixels after subtracting the fit. + \item \code{BIAS.OVERSCAN.SD} (F32): The standard deviation of the + binned overscan pixels after subtracting the fit. + \end{itemize} +\item \code{pmFlatField}: + \begin{itemize} + \item \code{FLAT.DONE} (STR): The time at which the flat-fielding + was completed. + \item \code{FLAT.BAD} (S32): Number of non-positive flat-field + pixels. + \end{itemize} +\end{itemize} + +To be added by higher-levels: +\begin{itemize} +\item \code{BIAS.NAME} (STR): Name of bias image +\item \code{DARK.NAME} (STR): Name of dark image +\item \code{FLAT.NAME} (STR): Name of flat image +\item \code{MASK.NAME} (STR): Name of mask image +\end{itemize} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Calibration} + +The calibration module essentially consists of combining multiple +images of a particular type in order to build up signal-to-noise. For +this, we require a general purpose image combination module. We +forsee this module as only acting upon data from the same detector, +and so each input image will have the same noise characteristics. + +\begin{datatype} +typedef struct { + psStats *stats; // Statistics to use in combining pixels + unsigned int maskVal, // Mask pixels where mask & maskVal == 1 + float fracHigh; // Fraction of high pixels to throw + float fracLow; // Fraction of low pixels to throw + int nKeep; // Number of pixels to be sure to keep +} pmCombineParams; +\end{datatype} + +\begin{prototype} +psImage * +pmReadoutCombine(psImage *output, // Output image, or NULL + const psList *inputs, // List of input readouts + pmCombineParams *params, // Combination parameters + const psVector *zero, // Offsets to apply for each image + const psVector *scale, // Scales to apply for each image + bool applyZeroScale, // Are zero and scale for application, or only noise properties? + float gain, // Gain in e/ADU + float readnoise // Read noise in e + ); +\end{prototype} + +\code{pmReadoutCombine} combines input images pixel by pixel --- for +each pixel of the output image, a stack of contributing input pixels +is formed and combined. Several of its input parameters are lists or +vectors, and if these are not all of the same length (or \code{NULL}), +the module shall generate an error and return \code{NULL}. + +If the provided \code{output} is \code{NULL}, then the module shall +allocate a new image of sufficient size for the input images. If the +\code{output} image is non-\code{NULL} and is not of sufficient size +for the combined image, the module shall generate an error and return +\code{NULL}. + +If the \code{inputs} is \code{NULL}, the module shall generate an +error and return \code{NULL}. Otherwise, the \code{inputs} shall be a +list of \code{pmReadout}s. The images contained within the +\code{pmReadout}s need not all be of the same size, but the module +shall take into account the offsets (\code{col0,row0}) from the corner +of the detector when comparing pixels, so that it is the same +\textit{physical} pixels that are combined. + +The parameters used in the combination, including how the pixels are +to be combined, and how the rejection is performed is contained within +the \code{params}, which may not be \code{NULL} (otherwise the module +shall generate an error and return \code{NULL}). We choose to use +this structure instead of supplying the values separately in order to +keep down the number of parameters to \code{pmReadoutCombine}; the +\code{pmCombineParams} may be recycled for subsequent calls to +\code{pmReadoutCombine} since the values are not dependent upon the +choice of inputs, but merely specify how the combination is to be +performed. + +The particular statistic specified by \code{stats} shall be used to +combine each stack of pixels from the input images. Only one of the +statistics choices may be specified, otherwise the module shall +generate an error and return \code{NULL}. + +If the \code{maskVal} is non-zero, then pixels in the \code{mask} of +each \code{pmReadout} in the \code{inputs} which satisfy the +\code{maskVal} shall not have the corresponding pixels placed in the +stack for combination. + +After masking, but before performing the combination, the highest +\code{fracHigh} fraction and lowest \code{fracLow} fraction of pixels +in the stack are immediately rejected, unless this would leave less +than \code{nKeep} pixels in the stack, in which case no immediate +rejection is performed. + +If the \code{zero} vector is non-\code{NULL} and \code{applyZeroScale} +is \code{true}, then the appropriate values shall be added to the +\code{inputs} before rejection is performed. If \code{zero} is +non-\code{NULL} and \code{applyZeroScale} is false, then the values +shall only be used in calculating the Poisson variances. + +If the \code{scale} vector is non-\code{NULL} and +\code{applyZeroScale} is \code{true}, then the appropriate values +shall multiply the \code{inputs} before rejection is performed. If +\code{scale} is non-\code{NULL} and \code{applyZeroScale} is false, +then the values shall only be used in calculating the Poisson +variances. + +The purpose of \code{applyZeroScale} is to allow combination of fringe +frames, where the frames have been deliberately sky-subtracted and +rescaled (to get the fringes amplitudes running from -1 to 1), which +actions should not be undone when combining, but yet it is desirable +to provide the \code{zero} and \code{scale} values so that the correct +noise properties are used in the combination. + +If the \code{gain} and \code{readnoise} are positive and non-negative +(respectively), then these shall be used to provide weights for the +combination using Poisson statistics ($\sigma_i$ below). + +In summary, pixels corresponding to the same physical pixel are +combined, having values $x_i \pm \sigma_i$. In the case that +\code{applyZeroScale} is \code{true}, then: +\begin{eqnarray} +x_i & = & s_i f_i + z_i \\ +\sigma_i & = & [g x_i + r^2]^{1/2} / g +\end{eqnarray} +Where $f_i$ is the value of the pixel in image $i$, $s_i$ is the scale +applied to image $i$, $z_i$ is the zero offset applied to image $i$, +$g$ is the gain, and $r$ is the read noise. If scales are not +provided, they are set to unity; if zero offsets are not provided, +they are set to zero. + +If \code{applyZeroScale} is \code{false}, then the values are: +\begin{eqnarray} +x_i & = & f_i \\ +\sigma_i & = & [g (s_i f_i + z_i) + r^2]^{1/2} / g +\end{eqnarray} +where the same symbols are used as above. + +The \code{inputs, zero} and \code{scale} may be of U16, S32 and F32 +types, and must all be of the same type. The \code{output} shall be +of the same type. + +\section{Objects on Images} + +\subsection{Overview} + +The process of finding, measuring, and classifying astronomical +sources on images is one of the critical tasks of the IPP or any +astronomical software system. In this section, we define structures +and functions related to the task of source detection and measurement. +The elements defined in this section are generally low-level +components which can be connected together to construct a complete +object measurement suite. + +We first define the collection of structures needed to carry +information about the detected sources. A major challenge is to +define what we mean by an astronomical object in the context of image +source detection. An astronomical object may be as simple as a +stellar point source, or it may consist of a galaxy which has smooth +extended structure; it may consist of an irregular galaxy or galaxy +group with substantial and complex sub-structure, or it may consist of +complex non-stellar structures such as planetary nebulae, reflection +nebulae, outflows and jets. + +The simplest objects (ie, stars) can be sufficiently modeled by the +point-source function (PSF). More complex objects (such as simple, +smooth galaxies), may have approximate analytical models which +represent their morphology with more-or-less accuracy. In the extreme +cases, the objects are not well modeled at all and must be represented +in other ways. Thus, one aspect of our data structures must be +elements to specify if an object has been represented by a model, what +the model parameters are, and how well it is represented by the model. +Another aspect of the data structures must be a representation of the +pixels associated with the object so complex structures may be +referenced without attempting to supply an analytical model. Finally, +it is often useful to allow a single complex model to be represented +as a collection of simpler contained structures which may be modeled. +Thus, the representation of an object must be capable of identifying +children, or substructures, of that object. + +Two additional aspects must be considered. First, source detection +need not be performed on a single image in isolation: it is necessary +for multiple realizations of the same source in multiple images to be +measured together (whether or not through simultaneous fitting in +multiple bands or via application of the results from one image to +another image). Second, it will be necessary to performed object +measurements on pixels in which no source is actually detected. For +example, this is a convenient way to provide flux upper limits at the +locations of known objects. + +In the discussion that follows, images are of type F32 and masks are +of type U8. + +\subsection{Structures to Describe Sources} + +\subsubsection{pmSource and pmPeak} + +We start by defining a single source detected in a single band: +\begin{datatype} +typedef struct { + pmPeak *peak; // description of peak pixel + psImage *pixels; // rectangular region including object pixels + psImage *mask; // mask to mark pixels associated with object in region + pmMoments *moments; // basic moments measure for the object + pmModel *modelPSF; // PSF model parameters and type + pmModel *modelFLT; // FLT model parameters and type + pmSourceType type; // best identification of object +} pmSource; +\end{datatype} + +In the object analysis process, we will use specific mask values to +mark the image pixels. The following structure defines the relevant +mask values. +\begin{datatype} +enum { + PSPHOT_MASK_CLEAR = 0x00, + PSPHOT_MASK_INVALID = 0x01, + PSPHOT_MASK_SATURATED = 0x02, + PSPHOT_MASK_MARKED = 0x08, +} psphotMaskValues; +\end{datatype} + +A source has the capacity for several types of measurements. The +simplest measurement of a source is the location and flux of the peak +pixel associated with the source: +\begin{datatype} +typedef struct { + int x; // x-coordinate of peak pixel + int y; // y-coordinate of peak pixel + float counts; // value of peak pixel (above sky?) + pmPeakType class; // description of peak +} pmPeak; +\end{datatype} + +A peak pixel may have several features which may be determined when +the peak is found or measured. These are specified by the +\code{pmPeakType} enum. \code{PM_PEAK_LONE} represents a single pixel +which is higher than its 8 immediate neighbors. The +\code{PM_PEAK_EDGE} represents a peak pixel which touching the image +edge. The \code{PM_PEAK_FLAT} represents a peak pixel which has more +than a specific number of neighbors at the same value, within some +tolerance: +\begin{datatype} +typedef enum { + PM_PEAK_LONE; // isolated peak + PM_PEAK_EDGE; // peak on edge + PM_PEAK_FLAT; // peak has equal-value neighbors +} pmPeakType; +\end{datatype} + +\subsubsection{pmMoments and source description} + +The pixels which contain the source may be specified with the +\code{psImage *pixels} element, and the mask image may be used to +exclude any pixels which are not considered part of the source. Note +that the source image may be simply a subimage of the main image or a +separate copy if the pixels are modified (eg, by subtracting flux from +other sources, etc). + +One of the simplest measurements which can be made quickly for an +object are the object moments. We specify a structure to carry the +moment information for a specific source: + +\begin{datatype} +typedef struct { + float x; // x-coord of centroid + float y; // y-coord of centroid + float Sx; // x-second moment + float Sy; // y-second moment + float Sxy; // xy cross moment + float Sum; // pixel sum above sky (background) + float Peak; // peak counts above sky + float Sky; // sky level (background) + float SN; // approx signal-to-noise + int nPixels; // number of pixels used +} pmMoments; +\end{datatype} + +A collection of object moment measurements can be used to determine +approximate object classes. The key to this analysis is the location +and statistics (in the second-moment plane, $\sigma_x$ vs $\sigma_y$) +of the group of objects which are likely PSF objects. We define the +following structure to identify the location and size of the psf clump +in the second-moment plane. +\begin{datatype} +typedef struct { + float X; + float dX; + float Y; + float dY; +} pmPSFClump; +\end{datatype} + +A given source may be identified as most-likely to be one of several +source types. The \code{pmSource} entry \code{pmSourceType} defines +the current best-guess for this source. \tbd{The values given below +are currently illustrative and will require some modification as the +source classification code is developed.} + +\begin{datatype} +typedef enum { + PM_SOURCE_DEFECT, // a cosmic-ray + PM_SOURCE_SATURATED, // random saturated pixels + + PM_SOURCE_SATSTAR, // a saturated star + PM_SOURCE_PSFSTAR, // a PSF star + PM_SOURCE_GOODSTAR, // a good-quality star + + PM_SOURCE_POOR_FIT_PSF, // poor quality PSF fit + PM_SOURCE_FAIL_FIT_PSF, // failed to get a good PSF fit + PM_SOURCE_FAINTSTAR, // below S/N cutoff + + PM_SOURCE_GALAXY, // an extended object (galaxy) + PM_SOURCE_FAINT_GALAXY, // a galaxy below S/N cutoff + PM_SOURCE_DROP_GALAXY, // ? + PM_SOURCE_FAIL_FIT_GAL, // failed on the galaxy fit + PM_SOURCE_POOR_FIT_GAL, // poor quality galaxy fit + + PM_SOURCE_OTHER, // unidentified +} pmSourceType; +\end{datatype} + +\subsubsection{pmModel Source Model and Abstraction} + +An object's flux distribution may be modeled with some analytical +function. The description of the model includes the model parameters +and their errors, along with the fit $\chi^2$. The model type is +identified by code \code{type}, dynamically assigned based on the +available models (see below). We discuss the details of these models +in section~\ref{ObjectModels}. The model parameters have 4 special +elements. The first four elements represent aspects of the source +which are not specified by the image PSF, even for point sources. +These consist of, in order: +\begin{itemize} +\item the local sky +\item the object normalization +\item the x-coordinate +\item the y-coordinate +\end{itemize} + +Every source may have two types of models: a PSF model and a FLT +(floating) model. The PSF model represents the best fit of the image +PSF to the specific object. In this case, the PSF-dependent +parameters are specified for the object by the PSF, not by the fit. +The FLT model represents the best fit of the given model to the +object, with all parameters floating in the fit. + +\begin{datatype} +typedef struct { + pmModelType type; // model to be used + psVector *params; // parameter values + psVector *dparams; // parameter errors + psF32 chisq; // fit chisq + psS32 nDOF; // number of degrees of freedom + psS32 nIter; // number of iterations + float radius; // fit radius actually used +} pmModel; +\end{datatype} + +Every model instance belongs to a class of models, defined by the +value of the \code{pmModelType type} entry. Various functions need +access to information about each of the models. Some of this +information varies from model to model, and may depend on the current +parameter values or other data quantities. In order to keep the code +from requiring the information about each model to be coded into the +low-level fitting routines, we define a collection of functions which +allow us to abstract this type of model-dependent information. These +generic functions take the model type and return the corresponding +function pointer for the specified model. Each +model is defined by creating this collection of specific functions, +and placing them in a single file for each model. We define the +following structure to carry the collection of information about the +models. + +\begin{datatype} +typedef struct { + char *name; + int nParams; + pmModelFunc modelFunc; + pmModelFlux modelFlux; + pmModelRadius modelRadius; + pmModelLimits modelLimits; + pmModelGuessFunc modelGuessFunc; + pmModelFromPSFFunc modelFromPSFFunc; + pmModelFitStatusFunc modelFitStatusFunc; +} pmModelGroup; +\end{datatype} + +Each entry in the \code{pmModelGroup} defines the information needed +by the system to specify a model. The function types define above are +\begin{prototype} +typedef psMinimizeLMChi2Func pmModelFunc; +typedef psF64 (*pmModelFlux)(const psVector *params); +typedef psF64 (*pmModelRadius)(const psVector *params, double flux); +typedef bool (*pmModelLimits)(psVector **beta_lim, psVector **params_min, psVector **params_max); +typedef bool (*pmModelGuessFunc)(pmModel *model, pmSource *source); +typedef bool (*pmModelFromPSFFunc)(pmModel *modelPSF, pmModel *modelFLT, pmPSF *psf); +typedef bool (*pmModelFitStatusFunc)(pmModel *model); +\end{prototype} + +Each of these functions is found for a given model by calling the +corresponding lookup function: +\begin{prototype} +pmModelFunc pmModelFunc_GetFunction (pmModelType type); +pmModelFlux pmModelFlux_GetFunction (pmModelType type); +pmModelRadius pmModelRadius_GetFunction (pmModelType type); +pmModelLimits pmModelLimits_GetFunction (pmModelType type); +pmModelGuessFunc pmModelGuessFunc_GetFunction (pmModelType type); +pmModelFromPSFFunc pmModelFromPSFFunc_GetFunction (pmModelType type); +pmModelFitStatusFunc pmModelFitStatusFunc_GetFunction (pmModelType type); +\end{prototype} + +\code{pmModelFunc} is the function used to determine the value of the +model at a specific coordinate, and is the one used by +\code{psMinimizeLMChi2}. + +\code{pmModelFlux} returns the total integrated flux for the given +input parameters. + +\code{pmModelRadius} returns the scaling radius at which the flux of +the model matches the specified flux. This presumes that the model is +a function of an elliptical contour. + +\code{pmModelLimits} sets the parameter limit vectors for the +function. + +\code{pmModelGuessFunc} generates an initial guess for the model based +on the provided source statistics (moments and pixel values as +needed). + +\code{pmModelFromPSFFunc} takes as input a representation of the psf +and a value for the model and fills in the PSF parameters of the +model. The input primarily relies upon the centroid coordinates of +the input model, thought the normalization may potentially be used. + +\code{pmModelFitStatusFunc} returns a true or false values based on +the success or failure of a model fit. the success is determined by +quantities such as the chisq or the signal-to-noise. + +In addition, the following functions are useful for interacting with +the collection of models: +\begin{prototype} +int pmModelParameterCount (pmModelType type); +\end{prototype} +This function returns the number of parameters used by the listed +function. + +\begin{prototype} +char *pmModelGetType (pmModelType type); +pmModelType pmModelSetType (char *name); +\end{prototype} +These two functions provide translations between the user-space model +names and the internal model type codes. The model type codes are not +necessarily maintained between compilations of the program; the name +should be used to transfer models between programs or systems. + +\subsubsection{pmPSF, pmPSFtry, and PSF model} + +It is useful to generate a model to define the point-spread-function +which describes the flux distribution for unresolved sources in an +image. In general, the PSF varies with position in the image. We +allow any of the source models defined for the \code{pmModel} to +represent the PSF. For a given source model, the 2D spatial variation +of all of the source parameters, except the first four PSF-independent +parameters, are represented as polynomial, stored in a \code{psArray}. +The other elements of the structure define the quality of the PSF +determination. + +\begin{datatype} +typedef struct { + psS32 type; ///< PSF Model in use + psArray *params; ///< Model parameters (psPolynomial2D) + psF32 chisq; ///< PSF goodness statistic + psS32 nPSFstars; ///< number of stars used to measure PSF +} pmPSF; +\end{datatype} + +\begin{prototype} +pmModel *pmModelFromPSF (pmModel *model, pmPSF *psf); +\end{prototype} +This function constructs a \code{pmModel} instance based on the +\code{pmPSF} description of the PSF. The input is a \code{pmModel} +with at least the values of the centroid coordinates (possibly +normalization if this is needed) defined. The values of the +PSF-dependent parameters are specified for the specific realization +based on the coordinates of the object. + +\begin{prototype} +bool pmPSFFromModels (pmPSF *psf, psArray *models, psVector *mask); +\end{prototype} +This function takes a collection of \code{pmModel} fitted models from +across a single image and builds a \code{pmPSF} representation of the +PSF. The input array of model fits may consist of entries to be +ignored (noted by a non-zero \code{mask} entry). The analysis of the +models fits a 2D polynomial for each parameter to the collection of +model parameters as a function of position (and normalization?). In +this process, some of the input models may be marked as outliers and +excluded from the fit. These elements will be marked with a specific +mask value (1 == \code{PSFTRY_MASK_OUTLIER}). + +We have the capability to test several different model functions in an +attempt to build an accurate PSF for an image. The complete set of +data needed to build and test as specific PSF model is carried by the +\code{pmPSFtry} structure: +\begin{datatype} +typedef struct { + pmModelType modelType; + pmPSF *psf; + psArray *sources; // pointers to the original sources + psArray *modelFLT; // model fits, floating parameters + psArray *modelPSF; // model fits, PSF parameters + psVector *mask; + psVector *metric; + psVector *fitMag; + float ApResid; + float dApResid; + float skyBias; +} pmPSFtry; +\end{datatype} +This structure contains a pointer to the collection of \code{sources} +which will be used to test the PSF model form. It lists the +\code{pmModelType type} of model being tests, and contains an element +to store the resulting \code{psf} representation. In addition, this +structure carries the complete collection of FLT (floating parameter) +and PSF (fixed parameter) model fits to each of the sources +\code{modelFLT} and \code{modelPSF}. It also contains a mask which is +set by the model fitting and psf fitting steps. For each model, the +value of the quality metric is stored in the vector \code{metric} and +the fitted instrumental magnitude is stored in \code{fitMag}. The +quality metric for the PSF model is the aperture magnitude minus the +fitted magnitude for each source. This collection of aperture +residuals is examined in the analysis process, and a linear trend of +the residual with the inverse object flux (ie, $10^{0.4*mag}$) is +fitted. The result of this fit is a measured sky bias (systematic +error in the sky measured by the fits), an effective +infinite-magnitude aperture correction (\code{ApResid}), and the +scatter of the aperture correction for the ensemble of PSF stars +(\code{dApResid}). The ultimate metric to intercompare multiple types +of PSF models is the value of the aperture correction scatter. + +The following functions are used to try out a single PSF model. +\begin{prototype} +pmPSFtry *pmPSFtryModel (psArray *sources, char *modelName, float RADIUS); +\end{prototype} +This function takes the input collection of sources and performs a +complete analysis to determine a PSF model of the given type +(specified by model name). The result is a \code{pmPSFtry} with the +results of the analysis. + +\begin{prototype} +bool pmPSFtryMetric (pmPSFtry *try, float RADIUS); +\end{prototype} +This function is used to measure the PSF model metric for the set of +results contained in the \code{pmPSFtry} structure. + +The following datatype defines the masks used by the \code{pmPSFtry} +analysis to identify sources which should or should not be included in +the analysis. +\begin{datatype} +enum { + PSFTRY_MASK_CLEAR = 0x00, + PSFTRY_MASK_OUTLIER = 0x01, // 1: outlier in psf polynomial fit (provided by psPolynomials) + PSFTRY_MASK_FLT_FAIL = 0x02, // 2: flt model failed to converge + PSFTRY_MASK_PSF_FAIL = 0x04, // 3: psf model failed to converge + PSFTRY_MASK_BAD_PHOT = 0x08, // 4: invalid source photometry + PSFTRY_MASK_ALL = 0x0f, +} pmPSFtryMaskValues; +\end{datatype} + +\begin{datatype} +typedef enum { + PM_CONTOUR_CRUDE; +} pmContourType; +\end{datatype} + +Allocators for the above structures are defined as follows: +\begin{prototype} +pmSource *pmSourceAlloc (); +pmPeak *pmPeakAlloc (int x, int y, float counts, psPeakType class); +pmMoments *pmMomentsAlloc (); +pmModel *pmModelAlloc (pmModelType type); +\end{prototype} + +\subsection{Basic Object Detection APIs} + +In this section, we specify a collection of basic functions which +operate on images and sources. We define them roughly in order in +which we expect to use them in a basic object detection process. + +\begin{prototype} +psVector *pmFindVectorPeaks(const psVector *vector, float threshold); +\end{prototype} + +Find all local peaks in the given vector above the given threshold. A +peak is defined as any element with a value greater than its two +neighbors and with a value above the threshold. Two types of special +cases must be addressed. Equal value elements: If an element has the +same value as the following element, it is not considered a peak. If +an element has the same value as the preceding element (but not the +following), then it is considered a peak. Note that this rule +(arbitrarily) identifies flat regions by their trailing edge. Edge +cases: At start of the vector, the element must be higher than its +neighbor. At the end of the vector, the element must be higher or +equal to its neighbor. These two rules again places the peak +associated with a flat region which touches the image edge at the +image edge. The result of this function is a vector containing the +coordinates (element number) of the detected peaks (type +\code{psU32}). + +\begin{prototype} +psArray *pmFindImagePeaks(const psImage *image, float threshold); +\end{prototype} + +Find all local peaks in the given image above the given threshold. +This function should find all row peaks using +\code{pmFindVectorPeaks}, then test each row peak and exclude peaks +which are not local peaks. A peak is a local peak if it has a higher +value than all 8 neighbors. If the peak has the same value as its +y +neighbor or +x neighbor, it is NOT a local peak. If any other +neighbors have an equal value, the peak is considered a valid peak. +Note two points: first, the +x neighbor condition is already enforced +by \code{pmFindVectorPeaks}. Second, these rules have the effect of +making flat-topped regions have single peaks at the (+x,+y) corner. +When selecting the peaks, their type must also be set. The result of +this function is an array of \code{pmPeak} entries. + +\begin{prototype} +psArray *pmPeaksSubset(psArray *peaks, float maxvalue, const psRegion valid); +\end{prototype} + +Create a new peaks array, removing certain types of peaks from the +input array of peaks based on the given criteria. Peaks should be +eliminated if they have a peak value above the given maximum value +limit or if the fall outside the valid region. The result of the +function is a new array with a reduced number of peaks. + +\begin{prototype} +bool pmSourceDefinePixels(pmSource *mySource, + pmReadout *readout, + psF32 x, + psF32 y, + psF32 Radius) +\end{prototype} + +Define \code{psImage} subarrays for the source located at coordinates +\code{x,y} on the image set defined by \code{readout}. The pixels +defined by this operation consist of a square window (of full width $2 +Radius + 1$) centered on the pixel which contains the given +coordinate, in the frame of the readout. The window is defined to +have limits which are valid within the boundary of the \code{readout} +image, thus if the radius would fall outside the image pixels, the +subimage is truncated to only consist of valid pixels. If +\code{readout->mask} or \code{readout->weight} are not \code{NULL}, +matching subimages are defined for those images as well. This +function fails if no valid pixels can be defined (x or y less than +Radius, for example). This function should be used to define a region +of interest around a source, including both source and sky pixels. + +\begin{prototype} +pmSource *pmSourceLocalSky(pmSource *source, + psStatsOptions statsOptions, + psF32 Radius) +\end{prototype} + +Measure the local sky in the vicinity of the given \code{source}. The +\code{Radius} defines the square aperture in which the moments will be +measured. This function assumes the source pixels have been defined, +and that the value of \code{Radius} here is smaller than the value of +\code{Radius} used to define the pixels. The annular region not +contained within the radius defined here is used to measure the local +background in the vicinity of the source. The local background +measurement uses the specified statistic passed in via the +\code{statsOptions} entry. This function allocates the +\code{pmMoments} structure. The resulting sky is used to set the +value of the \code{pmMoments.sky} element of the provided +\code{pmSource} structure. + +\begin{prototype} +bool pmSourceMoments(pmSource *source, float radius); +\end{prototype} + +Measure source moments for the given \code{source}, using the value of +\code{source.moments.sky} provided as the local background value and +the peak coordinates as the initial source location. The resulting +moment values are applied to the \code{source.moments} entry, and the +source is returned. The moments are measured within the given +circular radius of the \code{source.peak} coordinates. The return +value indicates the success (TRUE) of the operation. + +\begin{prototype} +pmPSFClump pmSourcePSFClump(psArray *sources, psMetadata *metadata); +\end{prototype} + +We use the source moments to make an initial, approximate source +classification, and as part of the information needed to build a PSF +model for the image. As long as the PSF shape does not vary +excessively across the image, the sources which are represented by a +PSF (the start) will have very similar second moments. The function +\code{pmSourcePSFClump} searches a collection of \code{sources} with +measured moments for a group with moments which are all very similar. +The function returns a \code{pmPSFClump} structure, representing the +centroid and size of the clump in the $\sigma_x$, $\sigma_y$ +second-moment plane. + +The goal is to identify and characterize the stellar clump within the +$\sigma_x, \sigma_y$ plane. To do this, an image is constructed to +represent this plane. The units of $\sigma_x$ and $\sigma_y$ are in +image pixels. A pixel in this analysis image represents 0.1 pixels in +the input image. The dimensions of the image need only be 10 pixels. +The peak pixel in this image (above a threshold of half of the image +maximum) is found. The coordinates of this peak pixel represent the +2D mode of the $\sigma_x, \sigma_y$ distribution. The sources with +$\sigma_x, \sigma_y$ within 0.2 pixels of this value are then used to +calculate the median and standard deviation of the $\sigma_x, +\sigma_y$ values. These resulting values are returned via the +\code{pmPSFClump} structure. + +The return value indicates the success (TRUE) of the operation. + +\tbd{limit the S/N of the candidate sources (part of Metadata)?} + +\tbd{save the clump parameters on the Metadata} + +\begin{prototype} +bool pmSourceRoughClass(psArray *sources, psMetadata *metadata, pmPSFClump clump) +\end{prototype} + +Based on the specified data values, make a guess at the source +classification. The sources are provides as a \code{psArray} of +\code{pmSource} entries. Definable parameters needed to make the +classification are provided to the routine with the \code{psMetadata} +structure. The rules below refer to values which can be extracted +from the metadata using the given keywords. Except as noted, the data +type for these parameters are \code{psF32}. + +The following rules are used to make the classification. Sources with +peak pixel above the value of \code{SATURATE} are identified as type +\code{PM_SOURCE_SATURATED}. For the remaining sources, the signal to +noise of the detection is approximated by: +\[ +SN = \frac{S \sqrt{g}}{\sqrt{S + A B + \frac{A R_e^2}{\sqrt{g}}}} +\] +where +\[ +A = \pi \sigma_x \sigma_y +\] +and $\sigma_x$ is \code{pmMoment.Sx}, $\sigma_y$ is +\code{pmMoment.Sy}, $B$ is \code{pmMoment.Sky} and $S$ is +\code{pmMoment.Sum}. The two parameters, $g$ and $R_e$ are the gain, +stored with the metadata keyword \code{GAIN} and the read-noise in +electrons, stored with the metadata keyword \code{READ_NOISE}. + +All sources with: +\[ |\sigma_x - \mbox{CLUMP}_x| < \mbox{CLUMP}_{dx}\] +and +\[ |\sigma_y - \mbox{CLUMP}_y| < \mbox{CLUMP}_{dy}\] +should be identified as type \code{PM_SOURCE_PSFSTAR}. Sources with +\[ \sigma_x < \mbox{CLUMP}_{x} - \mbox{CLUMP}_{dx}\] +and +\[ \sigma_y < \mbox{CLUMP}_{y} - \mbox{CLUMP}_{dy}\] +should be identified as type \code{PM_SOURCE_DEFECT}. Sources with +\[ \sigma_x > \mbox{CLUMP}_{x} + \mbox{CLUMP}_{dx}\] +and +\[ \sigma_y > \mbox{CLUMP}_{y} + \mbox{CLUMP}_{dy}\] +should be identified as type \code{PM_SOURCE_GALAXY}. All other +sources should be identified as type \code{PM_SOURCE_OTHER}. +\tbd{need to exclude stars outside valid data region}. + +Sources with $SN$ less than \code{FAINT_SN_LIM} are identified as type +\code{PM_SOURCE_FAINTSTAR}. Sources with $SN$ greater than +\code{PSF_SN_LIM} are then selected as possible PSF stars. These +sources are used to determine a guess at the shape of the PSF, based +on the collection of $\sigma_x$ and $\sigma_y$ values. + +\tbd{this discussion needs to be adjusted to match the implementation} + +\subsection{Object Fitting} + +We need a way to fit a particular functional model to an object. +PSLib includes the \code{psMinimizeLMChi2} and \code{psMinimizePowell} +functions, which form the core of this processes. However, additional +support functions and wrapping functions are necessary for the +specific case of source fitting. The operations can be broken down +into discrete steps: + +\begin{enumerate} +\item Identify the pixels of interest + +\item Make a guess at the model parameters. For some models, the +parameters may be guessed based on only the moments. For others, +additional measurements must be made. + +\item Construct the input vectors from the pixels of interest. + +\item Apply fitting function \code{psMinimizeLMChi2()} + +\item Construct model image. + +\item Subtract model from image. +\end{enumerate} + +\begin{prototype} +bool pmSourceModelGuess(pmSource *source, const psImage *image, pmModelType model); +\end{prototype} + +Convert available data to an initial guess for the given model. This +function allocates a \code{pmModel} entry for the \code{pmSource} +structure based on the provided model selection. The method of +defining the model parameter guesses are specified for each model +below. The guess values are placed in the model parameters. The +function returns \code{TRUE} on success or \code{FALSE} on failure. + +\begin{prototype} +psArray *pmSourceContour(const pmSource *source, const psImage *image, float level, pmContourType type); +\end{prototype} + +Find points in a contour for the given source at the given level. If +\code{type} is \code{PM_CONTOUR_CRUDE}, the contour is found by starting at +the source peak, running along each pixel row until the level is +crossed, then interpolating to the level coordinate for that row. +This is done for each row, with the starting point determined by the +midpoint of the previous row, until the starting point has a value +below the contour level. The returned contour consists of two vectors +giving the x and y coordinates of the contour levels. This function +may be used as part of the model guess inputs. + +\tbd{Other contour types may be specified in the future for more refined contours} + +\begin{prototype} +bool pmSourceFitModel(pmSource *source, psImage *image); +\end{prototype} + +Fit the requested model to the specified source. The starting guess +for the model is given by the input \code{source.model} parameter +values. The pixels of interest are specified by the +\code{source.pixels} and \code{source.mask} entries. This function +calls \code{psMinimizeLMChi2()} on the image data. The function +returns \code{TRUE} on success or \code{FALSE} on failure. + +\begin{prototype} +bool pmModelFitStatus (pmModel *model); +\end{prototype} + +This function wraps the call to the model-specific function returned +by \code{pmModelFitStatusFunc_GetFunction}. The model-specific +function examines the model parameters, parameter errors, Chisq, S/N, +and other parameters available from \code{model} to decide if the +particular fit was successful or not. + +\begin{prototype} +bool pmSourceAddModel(psImage *image, pmSource *source, bool center, bool sky); +bool pmSourceSubModel(psImage *image, pmSource *source, bool center, bool sky); +\end{prototype} + +Add or subtract the given source model flux to/from the provided +image. The boolean option \code{center} selects if the source is +re-centered to the image center or if it is placed at its centroid +location. The boolean option \code{sky} selects if the background sky +is applied (\code{TRUE}) or not. The pixel range in the target image +is at most the pixel range specified by the \code{source.pixels} +image. The success status is returned. + +\begin{prototype} +bool pmSourcePhotometry (float *fitMag, // integrated fit magnitude + float *obsMag, // aperture flux magnitude + pmModel *model, // model used for photometry + psImage *image, // image pixels to be used + psImage *mask // mask of pixels to ignore +); +\end{prototype} + +The function returns both the magnitude of the fit, defined as $-2.5 +\log{\rm flux}$, where the flux is integrated under the model, +theoretically from a radius of 0 to infinity. In practice, we +integrate the model beyond $50 \sigma$. The aperture magnitude is +defined as $-2.5 \log{\rm flux}$, where the flux is summed for all +pixels which are not excluded by the aperture mask. The model flux is +calculated by calling the model-specific function provided by +\code{pmModelFlux_GetFunction}. + +\begin{prototype} +int pmSourceDophotType (pmSource *source); +\end{prototype} +This function converts the source classification into the closest +available approximation to the Dophot classification scheme. The +following list gives the correspondence: +\begin{verbatim} +PM_SOURCE_DEFECT: 8 +PM_SOURCE_SATURATED: 8 +PM_SOURCE_SATSTAR: 10 +PM_SOURCE_PSFSTAR: 1 +PM_SOURCE_GOODSTAR: 1 +PM_SOURCE_POOR_FIT_PSF: 7 +PM_SOURCE_FAIL_FIT_PSF: 4 +PM_SOURCE_FAINTSTAR: 4 +PM_SOURCE_GALAXY: 2 +PM_SOURCE_FAINT_GALAXY: 2 +PM_SOURCE_DROP_GALAXY: 2 +PM_SOURCE_FAIL_FIT_GAL: 2 +PM_SOURCE_POOR_FIT_GAL: 2 +PM_SOURCE_OTHER: ? +\end{verbatim} + +\begin{prototype} +int pmSourceSextractType (pmSource *source); +\end{prototype} +This function converts the source classification into the closest +available approximation to the Sextractor classification scheme. +\tbd{the correspondence is not yet defined}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Image Combination} + +The image combination for \PS{} will employ an iterative approach, in +order to identify cosmic rays. The first pass involves transforming +and combining the input images, and noting pixels which are apparently +deviant. These pixels are examined in further detail, before a subset +of them are declared to be bad, whereupon these pixels are +re-transformed, and the images are combined properly. Here we +introduce two functions which will perform the combination and +examination steps. Prototype code exists for each of these functions. +\tbd{For further details, see the document about image combination for +\PS{}.} + +\subsection{Combining images} + +\begin{prototype} +psImage *pmCombineImages(psImage *combined, // Combined image + psArray **questionablePixels, // Array of rejection masks + const psArray *images, // Array of input images + const psArray *errors, // Array of input error images + const psArray *masks,// Array of input masks + unsigned int maskVal, // Mask value + const psPixels *pixels, // Pixels to combine + int numIter, // Number of rejection iterations + float sigmaClip, // Number of standard deviations at which to reject + const psStats *stats // Statistics to use in the combination + ); +\end{prototype} + +\code{pmCombineImages} shall combine the input \code{images}, +returning the \code{combined} image and a list of +\code{questionablePixels} in each input image. The array of error +images, \code{errors}, shall be used to calculate the value in the +combined image and the list of questionable pixels, if +non-\code{NULL}. Pixels whose corresponding value in the array of +mask images, \code{masks}, matches \code{maskVal} shall be masked from +the combination. The \code{images}, \code{errors} and \code{masks} +arrays, if non-\code{NULL}, shall all carry the same number of images; +otherwise the function shall generate an error and return \code{NULL}. +The sizes of all images in the \code{images}, \code{errors} and +\code{masks} arrays shall be identical; otherwise the function shall +generate an error and return \code{NULL}. + +If \code{pixels} is non-\code{NULL}, only those pixels specified shall +be combined. The combination consists of \code{numIter} iterations in +which a stack of pixels is combined using the specified \code{stats}. +In each iteration, questionable pixels are identified as lying more +than \code{sigmaClip} standard deviations from the combined value; +these pixels are excluded from the stack for the next iteration. The +value for the combined image is that produced by the \textit{first} +iteration (i.e., with no pixels excluded except those which have their +corresponding mask match the \code{maskVal}); this allows subsequent +calls to the function to only act on a small fraction of the pixels, +since questionable pixels identified in the first call of the function +will be properly rejected at a later point (see the example, below). + +In the event that \code{images} or \code{stats} are \code{NULL}, the +function shall generate an error and return \code{NULL}. + +\subsection{Rejecting pixels} + +\begin{prototype} +psArray *pmRejectPixels(const psArray *images, // Array of input images + const psArray *masks, // Array of masks for input images + const psArray *pixels, // These are the pixels which were rejected in the combination + const psArray *inToOut, // Transformations from input to output system + const psArray *outToIn, // Transformations from output to input system + float rejThreshold, // Rejection threshold + float gradLimit // Gradient limit + ); +\end{prototype} + +\tbd{This algorithm will change: an addition will be made to avoid +masking pixels in the wings of a star when combining images taken in +different seeing, and the gradient limit criteria will be changed.} + +\code{pmRejectPixels} inspects those questionable \code{pixels} +identified by \code{pmCombineImages} to determine if they are truly +discrepant. This inspection is performed in the coordinate frame of +the detector, where the pixels haven't been smeared by transformation. +Two tests are applied to each of the \code{images}: +\begin{enumerate} +\item The list of questionable pixels for an image is converted to an + image which is transformed back to the coordinate frame of the + detector. Those pixels in the detector frame which have a value + exceeding \code{rejThreshold} are suspected cosmic rays and + subjected to the next test. Depending on the value of the + \code{rejThreshold}, this test basically amounts to demanding that + questionable pixels neighbor each other in the transformed image. +\item The cores of point sources may mimic a cosmic ray, especially in + under-sampled images. To minimize flagging stars as cosmic rays, we + determine the gradient around the pixel of interest; if the gradient + is large, then the pixel is likely the core of a point source. In + order to reliably measure the gradient in the presence of a + suspected cosmic ray, we use the companion images --- the gradient + is the mean gradient at the corresponding position on the other + images. In order to calculate the corresponding positions, the + \code{inToOut} and \code{outToIn} transformations are required. If + the gradient is less than \code{gradLimit}, then the pixel is + identified as a cosmic ray. +\end{enumerate} + +The function shall return an array of \code{psPixels}, one for each of +the input \code{images}, containing pixels that have been identified +as cosmic rays according to the above criteria. + +If any of the input pointers are \code{NULL}, then the function shall +generate an error and return \code{NULL}. + +\subsection{Example} + +Here is an example of what the image combination routine looks like, +demonstrating how the various pieces fit together. The inputs are: +\begin{itemize} +\item \code{psArray *inputs}: Input detector images, each a + \code{psImage} of type \code{psF32} +\item \code{psArray *inputMask}: Input mask images, each a + \code{psImage} of type \code{psU8} +\item \code{psArray *inputsErr}: Input error images, each a + \code{psImage} of type \code{psF32} +\item \code{psPlaneTransform *skyToDetector}: Maps from sky + coordinates to detector coordinates, each a \code{psPlaneTransform} +\item \code{psRegion *combineRegion}: Sky coordinate pixels to combine +\item \code{int numIter}: Number of iterations in combination +\item \code{float rejThreshold}: Threshold for rejection +\item \code{float gradLimit}: Limit for gradient +\end{itemize} + +The output is the combined image. + +\begin{verbatim} + psArray *transformed = psArrayAlloc(nImages); // Array of transformed images + psArray *transformedErr = psArrayAlloc(nImages); // Array of transformed error images + psArray *transformedMask = psArrayAlloc(nImages); // Array of masks for transformed images + + for (int i = 0; i < nImages; i++) { + psPixels *blanks = NULL; // List of blank pixels + transformed->data[i] = psImageTransform(NULL, &blanks, inputs->data[i], + inputMask->data[i], inputMaskVal, NAN, skyToDetector, + combineRegion, NULL, PS_INTERPOLATE_BILINEAR); + transformedErr->data[i] = psImageTransform(NULL, NULL, inputsErr->data[i], inputMask->data[i], + inputMaskVal, NAN, skyToDetector, combineRegion, NULL, + PS_INTERPOLATE_BILINEAR_VARIANCE); + psImage *skyImage = transformed->data[i]; // Dereference the transformed image + psRegion *blankRegion = psRegionAlloc(0, 0, skyImage->numCols, skyImage->numRows); // Size of + // transformed + // image + transformedMask->data[i] = psPixelsToMask(NULL, blanks, *blankRegion, PS_MASK_BLANK); + psFree(blankRegion); + psFree(blanks); + } + + psArray *rejected = NULL; // Array of rejected pixel lists + psStats *combineStats = psStatsAlloc(PS_STAT_SAMPLE_MEAN); // Statistic to use in doing the combination + psImage *combined = pmCombineImages(NULL, &rejected, transformed, transformedErr, transformedMask, 0, + NULL, numIter, sigmaClip, combineStats); // Combined image + psArray *bad = pmRejectPixels(inputs, rejected, NULL, skyToDetector, rejThreshold, gradLimit); // Bad pix + psPixels *combinePixels = NULL; // Pixels to combine + for (int i = 0; i < nImages; i++) { + psPixels *badSource = psPixelsTransform(NULL, bad->data[i], skyToDetector); // Bad pixels on the input + psImage *badMask = psPixelsToMask(NULL, badSource, PS_MASK_COSMICRAY); // Mask image for the input + (void)psBinaryOp(inputMask->data[i], inputMask->data[i], "|", badMask); // Put CRs into original mask + psFree(badSource); + psFree(badMask); + + combinePixels = psPixelsConcatenate(redo, bad->data[i]); + + // Update transformed image + psPixels *blanks = NULL; // List of blank pixels + transformed->data[i] = psImageTransform(transformed->data[i], &blanks, inputs->data[i], + inputMask->data[i], inputMaskVal | PS_MASK_COSMICRAY, NAN, + skyToDetector, combineRegion, bad->data[i], + PS_INTERPOLATE_BILINEAR); + transformedErr->data[i] = psImageTransform(transformedErr->data[i], NULL, inputsErr->data[i], + inputMask->data[i], inputMaskVal | PS_MASK_COSMICRAY, + NAN, skyToDetector, combineRegion, bad->data[i], + PS_INTERPOLATE_BILINEAR_VARIANCE); + psImage *skyImage = transformed->data[i]; // Dereference the transformed image + psRegion *blankRegion = psRegionAlloc(0, 0, skyImage->numCols, skyImage->numRows); // Size of + // transformed + // image + transformedMask->data[i] = psPixelsToMask(transformedMask->data[i], blanks, *blankRegion, + PS_MASK_BLANK); + psFree(blankRegion); + psFree(blanks); + } + psFree(bad); + + // Combine with no rejection + combined = pmCombineImages(combined, NULL, transformed, transformedErr, transformedMask, + PS_MASK_BLANK, combinePixels, 0, 0.0, combineStats); + psFree(combineStats); + psFree(combinePixels); + psFree(transformed); + psFree(transformedErr); + psFree(transformedMask); +\end{verbatim} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Image Subtraction} + +Image subtraction is arguably the best method of identifying faint +variable sources in images with different point-spread functions. It +relies on fitting for a convolution kernel that minimizes the +residuals in subtracting small regions of the image. The use of a +convolution kernel consisting of a linear combination of basis +functions allows the problem to be solved with only modest computing +power. + +\subsection{The kernels} + +We will allow for the use of two convolution kernels. The first is +that employed by the popular image subtraction program, +\href{http://www2.iap.fr/users/alard/package.html}{ISIS}, consisting +of Gaussians modified by polynomials: +\begin{equation} +B_{ijk}(u,v) = e^{-(u^2 + v^2)/2\sigma_i^2} u^j v^k +\end{equation} +The second simply consists of delta functions, which we refer to as +POIS (Pan-STARRS Optimal Image Subtraction): +\begin{equation} +B_{ij}(u,v) = \delta(u - i)\ \delta(v - j) +\end{equation} +\tbd{For further details, see the document about image subtraction for +\PS{}.} The former is widely used, while the second appears to be +equally useful and faster, though not as tried and proven. + +\begin{datatype} +typedef enum { + PM_SUBTRACTION_KERNEL_POIS, // POIS kernel --- delta functions + PM_SUBTRACTION_KERNEL_ISIS // ISIS kernel --- gaussians modified by polynomials +} pmSubtractionKernelsType; +\end{datatype} + +In order to simplify the book-keeping for the kernels, we will define +a \code{pmSubtractionKernels}, which keeps track of the details of the +each of the kernel basis functions: + +\begin{datatype} +typedef struct { + pmSubtractionKernelType type; // Type of kernels --- allowing the use of multiple kernels + int size; // Size of kernel in x and y + int spatialOrder; // Maximum order of spatial variations + psVector *u, *v; // Offset (for POIS) or polynomial order (for ISIS) + psVector *sigma; // Width of Gaussian (for ISIS) + psVector *xOrder, *yOrder; // Spatial polynomial order (for all) + int subIndex; // Index of kernel to be subtracted to maintain flux conservation + psArray *preCalc; // Array of images containing pre-calculated kernel (to + // accelerate ISIS; don't use for POIS) +} pmSubtractionKernels; +\end{datatype} + +This structure caters for both choices of kernel type. For a POIS +kernel, the \code{u} and \code{v} vectors shall be set to the +coordinates for the delta functions for the corresponding kernel. For +an ISIS kernel, the \code{sigma} vector shall be set to the Gaussian +widths and the \code{u} and \code{v} vectors shall be set to the +orders of the modifying polynomials for the corresponding kernel. For +both choices of kernel, the \code{xOrder} and \code{yOrder} vectors +specify the order of the spatial variation. + +In order to maintain flux conservation when the kernel is spatially +variable, we need to treat one kernel in the set differently. The +convolutions for this kernel, identified by the \code{subIndex}, are +calculated in the usual way, while all others have the \code{subIndex} +kernel subtracted from them. For details, see the +\href{http://www.edpsciences.org/journal/index.cfm?v_url=aas/full/2000/11/ds8706/ds8706.html}{paper +by Alard (2000, A\&AS, 144, 363)}. + +Since the ISIS kernels are continuous functions, it is worth +pre-calculating them instead of calculating them each time they are +required. The \code{preCalc} array, consisting of \code{psImage}s is +provided for this purpose. + +The \code{pmSubtractionKernels} are generated by the following functions: + +\begin{prototype} +pmSubtractionKernels *pmSubtractionKernelsAllocPOIS(int size, int spatialOrder); +pmSubtractionKernels *pmSubtractionKernelsAllocISIS(const psVector *sigmas, const psVector *orders, + int size, int spatialOrder); +\end{prototype} + +\code{pmSubtractionKernelsAllocPOIS} shall generate the +\code{pmSubtractionKernels} suitable for the POIS kernel basis set. +This involves setting the \code{u}, \code{v}, \code{xOrder} and +\code{yOrder} to the appropriate values. \code{size} is the half-size +of the kernel, and \code{spatialOrder} is the maximum spatial order +(the spatial variation is $x^i y^j$ with $i+j <$ \code{spatialOrder}). +The \code{subIndex} is set to the kernel which has \code{u = 0}, +\code{v = 0}, \code{xOrder = 0} and \code{yOrder = 0}. There should +be \code{(2 * size + 1) * (2 * size + 1) * (spatialOrder + 1) * +(spatialOrder + 2) / 2} kernels. + +\code{pmSubtractionKernelsAllocISIS} shall generate the +\code{pmSubtractionKernels} suitable for the ISIS kernel basis set. +This involves setting the \code{sigma}, \code{u}, \code{v}, +\code{xOrder} and \code{yOrder} to the appropriate values, as well as +generating the \code{preCalc} images. Note that the \code{sigma} +vector contained within the \code{pmSubtractionKernels} is not the +same as the input \code{sigmas} vector, but contains repeated entries. +\code{size} is the half-size of the kernel, which specifies the size +of the \code{preCalc} images. The \code{spatialOrder} is the maximum +spatial order (the spatial variation is $x^i y^j$ with $i+j <$ +\code{spatialOrder}). The \code{subIndex} is set to the kernel which +has \code{u = 0}, \code{v = 0}, \code{xOrder = 0} and \code{yOrder = +0}, for the first of the Gaussian widths in the \code{sigmas} vector. + +\subsection{Stamps} + +Sub-regions on an image which are used to derive the best-fit +convolution kernel are referred to as ``stamps''. + +\begin{datatype} +typedef struct { + int x, y; // Position + psImage *matrix; // Associated matrix + psVector *vector; // Associated vector + pmStampStatus status; // Status of stamp +} pmStamp; +\end{datatype} + +A stamp is the region around a central pixel, \code{x,y}. The +\code{matrix} and \code{vector} are generated in the process of +solving for the best-fit convolution kernel; each of these will likely +be of type \code{psF64} in order to maintain the best possible +precision (we will be summing squares). In order to allow us to throw +out stamps without having to laboriously recompute the total +least-squares matrix and vector, we use a separate matrix and vector +for each stamp. + +To allow iteration on the choice of stamps, a stamp contains a +\code{status}, an enumerated type: + +\begin{datatype} +typedef enum { + PM_STAMP_USED, // Use this stamp + PM_STAMP_REJECTED, // This stamp has been rejected + PM_STAMP_RECALC, // Having been reset, this stamp needs to be recalculated + PM_STAMP_NONE // No stamp in this region +} pmStampStatus; +\end{datatype} + +\begin{prototype} +psArray *pmSubtractionFindStamps(psArray *stamps, // Output stamps, or NULL + const psImage *image, // Image for which to find stamps + const psImage *mask, // Mask + unsigned int maskVal, // Value for mask + float threshold, // Threshold for stamps in the image + int xNum, int yNum, // Number of stamps in x and y + int border // Border around image to ignore (should be size of kernel) + ); +\end{prototype} + +\code{pmSubtractionFindStamps} returns an array of stamps on the +\code{image} suitable for use in calculating the best-fit convolution +kernel. Except for a \code{border} all the way around, the +\code{image} is broken into \code{xNum} $\times$ \code{yNum} +rectangles; there will be a stamp within each rectangle. If +\code{stamps} is non-\code{NULL}, then the function shall only attempt +to identify a new stamp in a particular rectangle if the corresponding +stamp \code{status} is \code{PM_STAMP_REJECTED}. + +A stamp shall be recognized as the pixel with the greatest value that +does not have the corresponding pixel in the \code{mask} matching +\code{maskVal}. If the value of the this pixel does not exceed +\code{threshold}, then the stamp \code{status} shall be marked as +\code{PM_STAMP_NONE}, which means that the stamp will be ignored in +future iterations. If a legitimate stamp is found within the region, +then its status shall be changed to \code{PM_STAMP_RECALC}. + + +\subsection{Solving for the kernel} + +Calculating the best-fit convolution kernel requires solving a matrix +equation, the elements of which are obtained by applying the kernel +basis functions to the stamps. The final matrix and vector are the +sum of the matrices and vectors obtained for each of the individual +stamps. + +\begin{prototype} +bool pmSubtractionCalculateEquation(psArray *stamps, // The stamps for which to calculate the equation + const psImage *reference, // Reference image + const psImage *input, // Input image + const psSubtractionKernels *kernels, // The kernel basis functions + int footprint // Half-size of region over which to calculate equation + ); +\end{prototype} + +\code{pmSubtractionCalculateEquation} shall calculate the +\code{matrix} and \code{vector} for each of the \code{stamps} which +have \code{status} set to \code{PM_STAMP_RECALC}. The calculation is +made over a region with a half size of \code{footprint} on the +\code{reference} and \code{input} images, using each of the +\code{kernels}. In the event that any of the input pointers are +\code{NULL}, the function shall generate an error and return +\code{false}; otherwise, the function shall return \code{true}. + +The vector is: +\begin{equation} +v_i = \sum_{x,y} I(x,y) [ R(x,y) \otimes B_i(u,v) ] / \sigma(x,y)^2 +\end{equation} +and the matrix is: +\begin{equation} +M_{ij} = \sum_{x,y} \left[ R(x,y) \otimes B_i(u,v) \right] \ \left[ R(x,y) \otimes B_j(u,v) \right] / \sigma(x,y)^2 +\end{equation} +where $I(x,y)$ is the input image, $R(x,y)$ is the reference image, +$B_i(u,v)$ is the $i$-th kernel basis function, $\otimes$ denotes +convolution, $\sigma(x,y) = R(x,y)^{1/2}$ is an estimate of the error, +and the sum over $x,y$ indicates summing over the stamp regions. + +In addition to the each of the \code{kernels}, an additional parameter +for which we must solve is the difference in the background level +between the \code{reference} and \code{input} images. The appropriate +term shall be added to the \code{matrix} and \code{vector}. + +In order to maintain flux conservation when the kernel is spatially +variable, for each of the kernel basis functions apart from the first, +the kernel actually employed shall be the first kernel function +subtracted from the original kernel function. + +Having calculated the matrix equation for a stamp, its \code{status} +is set to \code{PM_STAMP_USED}. + +Since this step is one of the major rate-limiting factors in image +subtraction, care should be taken with optimization. + +\begin{prototype} +psVector *pmSubtractionSolveEquation(psVector *solution, // Solution vector, or NULL + const psArray *stamps // Array of stamps + ); +\end{prototype} + +\code{pmSubtractionSolveEquation} shall solve the matrix equation +provided by each of the \code{stamps}, returning the \code{solution} +vector. This involves summing the \code{matrix} and \code{vector} of +each of the stamps which have \code{status} set to +\code{PM_STAMP_USED}, and multiplying the inverse of the matrix by the +\code{vector}. If the \code{solution} is \code{NULL}, then the +function shall allocate and return a new vector; otherwise, the +\code{solution} vector shall be modified in-place. If \code{stamps} +is \code{NULL}, then the function shall generate an error and return +\code{NULL}. The type of the \code{solution} vector should be +\code{psF64}, since the matrix equation involves summing squares. + + +\subsection{Rejection of stamps} + +\begin{prototype} +bool pmSubtractionRejectStamps(psArray *stamps, // Array of stamps to check for rejection + psImage *mask, // Mask image + unsigned int badStampMaskVal, // Value to use in mask for bad stamp + int footprint, // Region to mask if stamp is bad + float sigmaRej, // Number of RMS deviations above zero at which to reject + const psImage *refImage, // Reference image + const psImage *inImage, // Input image + const psVector *solution, // Solution vector + const pmSubtractionKernels *kernels // Array of kernel parameters + ); +\end{prototype} + +\code{pmSubtractionRejectStamps} shall apply the \code{solution} to +the \code{stamps}, rejecting stamps for which the mean square +residuals exceed \code{sigmaRej} RMS deviations from zero. +\code{stamps} which are rejected have their \code{status} set to +\code{PM_STAMP_REJECTED}, and have pixels within \code{footprint} of +the corresponding position in the \code{mask} set to +\code{badStampMaskVal} so they will not be used again. + +The deviations are calculated through extracting the stamps from the +\code{refImage} and \code{inImage}, convolving the reference stamp by +the best-fit kernel (derived from the \code{solutions} vector and the +\code{kernels}), subtracting and then dividing by the stamp from the +input image, and then squaring to obtain the mean square residual. + +\subsection{Visualization of kernel} + +Having solved for the best-fit kernel, it is often useful to visualize +it. + +\begin{prototype} +psImage *pmSubtractionKernelImage(psImage *out, const psVector *solution, + const pmSubtractionKernels *kernels, float x, float y); +\end{prototype} + +\code{pmSubtractionKernelImage} shall create an image of the kernel +from the \code{solution} vector and the \code{kernels}. The relative +position (between -1 and +1) on the image at which to evaluate the +kernel (important if the kernel is spatially variable) is specified by +\code{x} and \code{y}. If \code{out} is \code{NULL}, then the +function shall allocate a new image of sufficient size (matching the +\code{precalc} images), and return the result; otherwise, \code{out} +shall be modified in-place. + + +\subsection{Example} + +Here is an example of what the image subtraction routine looks like, +demonstrating how the various pieces fit together. The inputs are: +\begin{itemize} +\item \code{psImage *reference}: Reference image +\item \code{psImage *refMask}: Mask for reference image +\item \code{psImage *input}: Input image +\item \code{psImage *inMask}: Mask for input image +\item \code{unsigned int maskVal}: Value to be masked +\item \code{pmSubtractionKernelType kernelType}: Type of kernel to use +\item \code{int kernelHalfSize}: Half the kernel size (full size is \code{2*kernelHalfSize + 1}) +\item \code{psVector *sigmas}: Widths for the ISIS Gaussians +\item \code{psVector *polyOrders}: Polynomial orders for ISIS Gaussians +\item \code{int spatialOrder}: Maximum spatial order for spatially variable kernel +\item \code{float stampThreshold}: Threshold for finding stamps +\item \code{int nStampsX, nStampsY}: Number of stamps in x and y +\item \code{int stampSize}: Half size of stamp footprint +\item \code{int numIter}: Number of iterations on the stamps +\item \code{float sigmaRej}: Rejection threshold for stamps +\end{itemize} + +The output is the subtracted image and the corresponding mask. + +\begin{verbatim} + // Mask around bad pixels in the reference image. There are two cases to worry about: + // 1. Bad pixels within the kernel, which will affect the subtracted image + // 2. Bad pixels within the stamp, which affects the calculation of the kernel + psImage *subMask = psImageGrowMask(NULL, refMask, maskVal, kernelHalfSize, PS_MASK_NEAR_BAD); + (void)psImageGrowMask(subMask, refMask, maskVal, stampSize, PS_MASK_BAD_STAMP); + // Add in the mask for the input image. Don't need to grow this, since it isn't convolved. + (void)psBinaryOp(subMask, subMask, "|", inMask); + + // Generate kernel basis functions + psArray *kernels = NULL; // Array of kernel basis functions + switch (kernelType) { + case PM_SUBTRACTION_KERNEL_POIS: + // Create the kernel basis functions + kernels = pmSubtractionKernelsGeneratePOIS(kernelHalfSize, spatialOrder); + break; + case PM_SUBTRACTION_KERNEL_ISIS: + kernels = pmSubtractionKernelsGenerateISIS(sigmas, polyOrders, kernelHalfSize, spatialOrder); + break; + default: + barf(); + } + + psArray *stamps = NULL; // Array of stamps + psVector *kernelCoeffs = NULL; // Coefficients for the kernels + bool rejected = true; // Did we reject a stamp in the last iteration? + + // Iterate for a solution + for (int iter = 0; iter < numIter && rejected; iter++) { + + // Find stamps + stamps = pmSubtractionFindStamps(stamps, reference, subMask, maskVal | PS_MASK_BAD_STAMP, + stampThreshold, nStampsX, nStampsY, stampSize, kernelHalfSize); + + // Generate and solve matrix equations + (void)pmSubtractionCalculateEquation(stamps, reference, input, kernels, stampSize); + kernelCoeffs = pmSubtractionSolveEquation(kernelCoeffs, stamps); + + // Reject bad stamps + rejected = pmSubtractionRejectStamps(stamps, subMask, PS_MASK_BAD_STAMP, stampSize, sigmaRej, + reference, input, kernelCoeffs, kernels); + } + + // Convolve the reference image + psImage *referenceConvolved = pmSubtractionConvolveImage(NULL, reference, subMask, kernelCoeffs, kernels); + // Subtract + psImage *subtracted = (psImage*)psBinaryOp(NULL, input, "-", referenceConvolved); + + // What does the kernel look like? + psImage *kernelImage = pmSubtractionKernelImage(NULL, kernelCoeffs, kernels, 0.0, 0.0); + // Check/save kernel image, print statistics.... + + psFree(referenceConvolved); + psFree(stamps); + psFree(kernels); + psFree(kernelCoeffs); +\end{verbatim} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\appendix + +\section{Basic Object Models} +\label{ObjectModels} + +We specify a variety of basic object models which are required. +Details of the model functional forms, parameters, and the derivatives +are specified in the ADD. + +\subsubsection{Real 2D Gaussian} + +\begin{prototype} +float pmMinLM_Gauss2D(psVector *deriv, psVector *params, psVector *x); +\end{prototype} + +This function is a two-dimensional Gaussian with an elliptical +cross-section and a constant local background. + +The initial guess for the Gaussian parameters may be taken from the +moments, peak value, and local sky. + +\subsubsection{Pseudo-Gaussian} + +\begin{prototype} +float pmMinLM_PseudoGauss2D(psVector *deriv, psVector *params, psVector *x); +\end{prototype} + +This function is a polynomial approximation of a 2D Gaussian otherwise +very similar to the real Gaussian. It is used in place of a real +Gaussian for speed. + +The initial guess for the Gaussian parameters may be taken from the +moments, peak value, and local sky. + +\subsubsection{Waussian} + +\begin{prototype} +float pmMinLM_Wauss2D(psVector *deriv, psVector *params, psVector *x); +\end{prototype} + +The Waussian is a modified polynomial approximation of a 2D Gaussian, +with non-linear polynomial terms having variable coefficients, rather +than the Taylor series values of 1/2 and 1/6. + +\subsubsection{Twisted Gaussian} + +\begin{prototype} +float pmMinLM_TwistGauss2D(psVector *deriv, psVector *params, psVector *x); +\end{prototype} + +This function describes an object with power-law wings and a flattened +core, where the core has a different contour from the wings. + +The initial guess for the Gaussian parameters may be taken from the +moments, peak value, and local sky. + +\tbd{future galaxy models to be implemented} + +\subsubsection{Sersic Galaxy Model} + +\begin{prototype} +float pmMinLM_Sersic(psVector *deriv, psVector *params, psVector *x); +\end{prototype} + +\subsubsection{Sersic with Core Galaxy Model} + +\begin{prototype} +float pmMinLM_SersicCore(psVector *deriv, psVector *params, psVector *x); +\end{prototype} + +\subsubsection{Pseudo Sersic Galaxy Model} + +\begin{prototype} +float pmMinLM_PseudoSersic(psVector *deriv, psVector *params, psVector *x); +\end{prototype} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Example Camera Configuration Files} + +\tbd{Some of these don't exactly match the specifications of this +document yet, because they have been changed from the prototype, but +it is hoped that they will be useful. Questions are welcome.} + +\subsection{MegaCam Raw} + +\begin{verbatim} +# The raw MegaCam data comes off the telescope with each of the chips stored in extensions of a MEF file. + +# How to identify this type +RULE METADATA + TELESCOP STR CFHT 3.6m + DETECTOR STR MegaCam + EXTEND BOOL T + NEXTEND S32 72 +END + +# How to read this data +PHU STR FPA # The FITS file represents an entire FPA +EXTENSIONS STR CELL # The extensions represent cells + +# What's in the FITS file? +CONTENTS METADATA + # Extension name, chip name:type + amp00 STR ccd00:left + amp01 STR ccd00:right + amp02 STR ccd01:left + amp03 STR ccd01:right + amp04 STR ccd02:left + amp05 STR ccd02:right + amp06 STR ccd03:left + amp07 STR ccd03:right + amp08 STR ccd04:left + amp09 STR ccd04:right + amp10 STR ccd05:left + amp11 STR ccd05:right + amp12 STR ccd06:left + amp13 STR ccd06:right + amp14 STR ccd07:left + amp15 STR ccd07:right + amp16 STR ccd08:left + amp17 STR ccd08:right + amp18 STR ccd09:left + amp19 STR ccd09:right + amp20 STR ccd10:left + amp21 STR ccd10:right + amp22 STR ccd11:left + amp23 STR ccd11:right + amp24 STR ccd12:left + amp25 STR ccd12:right + amp26 STR ccd13:left + amp27 STR ccd13:right + amp28 STR ccd14:left + amp29 STR ccd14:right + amp30 STR ccd15:left + amp31 STR ccd15:right + amp32 STR ccd16:left + amp33 STR ccd16:right + amp34 STR ccd17:left + amp35 STR ccd17:right + amp36 STR ccd18:left + amp37 STR ccd18:right + amp38 STR ccd19:left + amp39 STR ccd19:right + amp40 STR ccd20:left + amp41 STR ccd20:right + amp42 STR ccd21:left + amp43 STR ccd21:right + amp44 STR ccd22:left + amp45 STR ccd22:right + amp46 STR ccd23:left + amp47 STR ccd23:right + amp48 STR ccd24:left + amp49 STR ccd24:right + amp50 STR ccd25:left + amp51 STR ccd25:right + amp52 STR ccd26:left + amp53 STR ccd26:right + amp54 STR ccd27:left + amp55 STR ccd27:right + amp56 STR ccd28:left + amp57 STR ccd28:right + amp58 STR ccd29:left + amp59 STR ccd29:right + amp60 STR ccd30:left + amp61 STR ccd30:right + amp62 STR ccd31:left + amp63 STR ccd31:right + amp64 STR ccd32:left + amp65 STR ccd32:right + amp66 STR ccd33:left + amp67 STR ccd33:right + amp68 STR ccd34:left + amp69 STR ccd34:right + amp70 STR ccd35:left + amp71 STR ccd35:right +END + +# Specify the cell data +CELLS METADATA + left METADATA # Left amplifier + CELL.BIASSEC STR HEADER:BIASSEC + CELL.TRIMSEC STR HEADER:DATASEC + CELL.XPARITY S32 1 # We could have specified this as a DEFAULT, but this works + END + right METADATA # Right amplifier + CELL.BIASSEC STR HEADER:BIASSEC + CELL.TRIMSEC STR HEADER:DATASEC + CELL.XPARITY S32 -1 # This cell is read out in the opposite direction + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.NAME STR EXPNUM + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR ROTANGLE + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + FPA.MJD STR MJD-OBS + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.XBIN STR CCDBIN1 + CELL.YBIN STR CCDBIN2 + CELL.GAIN STR GAIN + CELL.READNOISE STR RDNOISE + CELL.SATURATION STR SATURATE +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + CELL.BAD S32 0 + CELL.YPARITY_DEPEND STR CHIP.NAME + CELL.YPARITY METADATA + ccd00 S32 -1 + ccd01 S32 -1 + ccd02 S32 -1 + ccd03 S32 -1 + ccd04 S32 -1 + ccd05 S32 -1 + ccd06 S32 -1 + ccd07 S32 -1 + ccd08 S32 -1 + ccd09 S32 -1 + ccd10 S32 -1 + ccd11 S32 -1 + ccd12 S32 -1 + ccd13 S32 -1 + ccd14 S32 -1 + ccd15 S32 -1 + ccd16 S32 -1 + ccd17 S32 -1 + ccd18 S32 1 + ccd19 S32 1 + ccd20 S32 1 + ccd21 S32 1 + ccd22 S32 1 + ccd23 S32 1 + ccd24 S32 1 + ccd25 S32 1 + ccd26 S32 1 + ccd27 S32 1 + ccd28 S32 1 + ccd29 S32 1 + ccd30 S32 1 + ccd31 S32 1 + ccd32 S32 1 + ccd33 S32 1 + ccd34 S32 1 + ccd35 S32 1 + END +END + +# How to translate PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS +# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME +# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME + +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). + +END +\end{verbatim} + +\subsection{MegaCam Splice} + +\begin{verbatim} +# The spliced MecaCam data is stored in single extensions for each chip + +# How to recognise this type +RULE METADATA + TELESCOP STR CFHT 3.6m + DETECTOR STR MegaCam + EXTEND BOOL T + NEXTEND S32 36 +END + +# How to read this data +PHU STR FPA # The FITS file represents an entire FPA +EXTENSIONS STR CHIP # The extensions represent chips + +# What's in the FITS file? +CONTENTS METADATA + # Extension name, components + ccd00 STR left right + ccd01 STR left right + ccd02 STR left right + ccd03 STR left right + ccd04 STR left right + ccd05 STR left right + ccd06 STR left right + ccd07 STR left right + ccd08 STR left right + ccd09 STR left right + ccd10 STR left right + ccd11 STR left right + ccd12 STR left right + ccd13 STR left right + ccd14 STR left right + ccd15 STR left right + ccd16 STR left right + ccd17 STR left right + ccd18 STR left right + ccd19 STR left right + ccd20 STR left right + ccd21 STR left right + ccd22 STR left right + ccd23 STR left right + ccd24 STR left right + ccd25 STR left right + ccd26 STR left right + ccd27 STR left right + ccd28 STR left right + ccd29 STR left right + ccd30 STR left right + ccd31 STR left right + ccd32 STR left right + ccd33 STR left right + ccd34 STR left right + ccd35 STR left right +END + +# Specify the cells +CELLS METADATA + left METADATA + CELL.BIASSEC STR HEADER:BSECA + CELL.TRIMSEC STR HEADER:TSECA + END + + right METADATA + CELL.BIASSEC STR HEADER:BSECB + CELL.TRIMSEC STR HEADER:TSECB + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.NAME STR EXPNUM + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR ROTANGLE + FPA.RA STR RA + FPA.DEC STR DEC + FPA.RADECSYS STR RADECSYS + FPA.MJD STR MJD-OBS + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.XBIN STR CCDBIN1 + CELL.YBIN STR CCDBIN2 + CELL.GAIN STR GAIN + CELL.READNOISE STR RDNOISE + CELL.SATURATION STR SATURATE +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + CELL.BAD S32 0 + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 +END + + +# How to translate PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS +# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME +# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME + +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). + +END +\end{verbatim} + +\subsection{LRIS Blue} + +\begin{verbatim} +# The Low Resolution Imager and Spectrograph (LRIS) blue side + +# We have no choice but to hard-code the various regions, because Keck +# only stores them as: +# WINDOW = '1,0,0,2048,4096' +# PREPIX = 51 +# POSTPIX = 80 +# BINNING = '1,1 ' +# AMPPSIZE= '[1:1024,1:4096]' + +# I don't know how we would get the IPP to react to changes in the +# windowing on the fly --- we have no mechanism for setting the region +# sizes on the basis of the above keywords. Therefore, we hard-code +# the regions and assert on our assumptions in the RULE. + + +# How to identify this type +RULE METADATA + TELESCOP STR Keck I + INSTRUME STR LRISBLUE + AMPLIST STR 1,4,0,0 + WINDOW STR 1,0,0,2048,4096 + PREPIX S32 51 + POSTPIX S32 80 + BINNING STR 1,1 + AMPPSIZE STR [1:1024,1:4096] + NAXIS1 S32 4620 + NAXIS2 S32 4096 +END + +# How to read this data +PHU STR FPA # The FITS file represents an entire FPA +EXTENSIONS STR NONE # There are no extensions + +# What's in the FITS file? +CONTENTS METADATA + LeftChip STR amp1 amp2 + RightChip STR amp3 amp4 +END + +# Specify the cell data +CELLS METADATA + amp1 METADATA + CELL.BIASSEC STR VALUE:[1:51,1:4096];[4301:4380,1:4096] + CELL.TRIMSEC STR VALUE:[205:1228,1:4096] + CELL.GAIN STR VALUE:1.2 + CELL.READNOISE STR VALUE:5.6 + END + + amp2 METADATA + CELL.BIASSEC STR VALUE:[52:102,1:4096];[4381:4460,1:4096] + CELL.TRIMSEC STR VALUE:[1229:2252,1:4096] + CELL.GAIN STR VALUE:1.3 + CELL.READNOISE STR VALUE:6.7 + END + + amp3 METADATA + CELL.BIASSEC STR VALUE:[103:153,1:4096];[4461:4540,1:4096] + CELL.TRIMSEC STR VALUE:[2253:3276,1:4096] + CELL.GAIN STR VALUE:1.4 + CELL.READNOISE STR VALUE:7.8 + END + + amp4 METADATA + CELL.BIASSEC STR VALUE:[154:204,1:4096];[4541:4620,1:4096] + CELL.TRIMSEC STR VALUE:[3277:4300,1:4096] + CELL.GAIN STR VALUE:1.5 + CELL.READNOISE STR VALUE:8.9 + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR BLUFILT + FPA.POSANGLE STR ROTPOSN + FPA.RA STR RA + FPA.DEC STR DEC + CELL.EXPOSURE STR EXPOSURE + CELL.DARKTIME STR EXPOSURE // No special darktime header; use exposure time + CELL.DATE STR DATE // NOTE: There are TWO keywords called "DATE" (creation, exp)! + CELL.TIME STR UT +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.RADECSYS STR ICRS +END +\end{verbatim} + +\subsection{LRIS Red} + +\begin{verbatim} +# The Low Resolution Imager and Spectrograph (LRIS) red side + +# We have no choice but to hard-code the various regions, because Keck +# only stores them as: +# WINDOW = '0,0,0,2048,2048' +# PREPIX = 20 +# POSTPIX = 80 +# BINNING = '1,1 ' +# AMPPSIZE= '[1:1024,1:4096]' + +# I don't know how we would get the IPP to react to changes in the +# windowing on the fly --- we have no mechanism for setting the region +# sizes on the basis of the above keywords. Therefore, we hard-code +# the regions and assert on our assumptions in the RULE. + + +# How to identify this type +RULE METADATA + TELESCOP STR Keck I + INSTRUME STR LRIS + AMPLIST STR 2,1,0,0 + WINDOW STR 0,0,0,2048,2048 + PREPIX S32 20 + POSTPIX S32 80 + BINNING STR 1, 1 + CCDPSIZE STR [1:2048,1:2048] + NAXIS1 S32 2248 + NAXIS2 S32 2048 + IMTYPE STR TWOAMPTOP +END + +# How to read this data +PHU STR CHIP # The FITS file represents a single chip +EXTENSIONS STR NONE # There are no extensions + +# What's in the FITS file? +CONTENTS STR LeftSide RightSide + +# Specify the cell data +CELLS METADATA + LeftSide METADATA + CELL.BIASSEC STR VALUE:[1:20,1:2048];[2089:2168,1:2048] + CELL.TRIMSEC STR VALUE:[41:1064,1:2048] + CELL.GAIN STR VALUE:1.2 + CELL.READNOISE STR VALUE:5.6 + END + + RightSide METADATA + CELL.BIASSEC STR VALUE:[21:40,1:2048];[2169:2248,1:2048] + CELL.TRIMSEC STR VALUE:[1065:2088,1:2048] + CELL.GAIN STR VALUE:1.3 + CELL.READNOISE STR VALUE:6.5 + END +END + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + FPA.AIRMASS STR AIRMASS + FPA.FILTER STR FILTER + FPA.POSANGLE STR POSANG + FPA.RA STR OBJ-RA + FPA.DEC STR OBJ-DEC + CELL.EXPOSURE STR EXPTIME + CELL.DARKTIME STR DARKTIME + CELL.DATE STR DATE-OBS + CELL.TIME STR TIME-OBS +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.RADECSYS STR ICRS +END +\end{verbatim} + +\subsection{GPC OTA} + +\begin{verbatim} +# The raw GPC data comes off the telescope with each of the chips stored in separate files + +# How to identify this type +RULE METADATA +# TELESCOP STR PS1 +# DETECTOR STR GPC1 + EXTEND BOOL T + NEXTEND S32 64 + NAMPS S32 64 +END + +# How to read this data +PHU STR CHIP # The FITS file represents a single chip +EXTENSIONS STR CELL # The extensions represent cells + +# What's in the FITS file? +CONTENTS METADATA + # Extension name, type + xy00 STR pitch10u + xy01 STR pitch10u + xy02 STR pitch10u + xy03 STR pitch10u + xy04 STR pitch10u + xy05 STR pitch10u + xy06 STR pitch10u + xy07 STR pitch10u + xy10 STR pitch10u + xy11 STR pitch10u + xy12 STR pitch10u + xy13 STR pitch10u + xy14 STR pitch10u + xy15 STR pitch10u + xy16 STR pitch10u + xy17 STR pitch10u + xy20 STR pitch10u + xy21 STR pitch10u + xy22 STR pitch10u + xy23 STR pitch10u + xy24 STR pitch10u + xy25 STR pitch10u + xy26 STR pitch10u + xy27 STR pitch10u + xy30 STR pitch10u + xy31 STR pitch10u + xy32 STR pitch10u + xy33 STR pitch10u + xy34 STR pitch10u + xy35 STR pitch10u + xy36 STR pitch10u + xy37 STR pitch10u + xy40 STR pitch10u + xy41 STR pitch10u + xy42 STR pitch10u + xy43 STR pitch10u + xy44 STR pitch10u + xy45 STR pitch10u + xy46 STR pitch10u + xy47 STR pitch10u + xy50 STR pitch10u + xy51 STR pitch10u + xy52 STR pitch10u + xy53 STR pitch10u + xy54 STR pitch10u + xy55 STR pitch10u + xy56 STR pitch10u + xy57 STR pitch10u + xy60 STR pitch10u + xy61 STR pitch10u + xy62 STR pitch10u + xy63 STR pitch10u + xy64 STR pitch10u + xy65 STR pitch10u + xy66 STR pitch10u + xy67 STR pitch10u + xy70 STR pitch10u + xy71 STR pitch10u + xy72 STR pitch10u + xy73 STR pitch10u + xy74 STR pitch10u + xy75 STR pitch10u + xy76 STR pitch10u + xy77 STR pitch10u +END + +# Specify the cell data +CELLS METADATA + pitch10u METADATA + CELL.BIASSEC STR VALUE:[575:606,1:594] + CELL.TRIMSEC STR VALUE:[1:574,1:594] + # CELL.BIASSEC STR HEADER:BIASSEC + # CELL.TRIMSEC STR HEADER:DATASEC + END + + # This is just in here for fun + pitch12u METADATA + CELL.BIASSEC STR VALUE:[1:10,1:512];[523:574,1:512] + CELL.TRIMSEC STR VALUE:[11:522,1:512] + # CELL.BIASSEC STR HEADER:BIASSEC + # CELL.TRIMSEC STR HEADER:TRIMSEC + END +END + + +# How to translate PS concepts into FITS headers +TRANSLATION METADATA + CELL.BIN STR CCDSUM + CELL.SATURATION STR SATURATE +END + +# Default PS concepts that may be specified by value +DEFAULTS METADATA + FPA.AIRMASS F32 0.0 + FPA.FILTER STR NONE + FPA.POSANGLE F32 0.0 + FPA.RA STR 0:0:0 + FPA.DEC STR 0:0:0 + FPA.RADECSYS STR ICRS + FPA.NAME S32 0 + FPA.MJD F32 12345.6789 + CELL.EXPOSURE F32 0.0 + CELL.DARKTIME F32 0.0 + CELL.GAIN F32 1.0 + CELL.READNOISE F32 0.0 + CELL.BAD S32 0 + CELL.BIN S32 1 + CELL.XPARITY S32 1 + CELL.YPARITY S32 1 +END + +# How to translate PS concepts into database lookups +DATABASE METADATA + TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS + CELL.GAIN dbEntry Camera gain chipId,cellId CHIP,CELL + CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP,CELL + +# A database entry refers to a particular column (COLUMN) in a +# particular table (TABLE), given certain PS concepts (GIVENPS) that +# match certain database columns (GIVENDBCOL). + +END +\end{verbatim} + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Revision Change Log} +\input{ChangeLogSDRS.tex} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\bibliographystyle{plain} +\bibliography{panstarrs} + +\end{document} + +%%% All this is here for storage only --- it's not part of the document. + + +\subsection{Cosmic rays} + +Given an input image, a choice of a particular algorithm with +corresponding parameters, \code{psPhase2MaskCRs} shall mask cosmic +rays on the input image on the basis of their morphology. The API shall be the following: +\begin{prototype} +/** Masks Cosmic Rays on the input image on the basis of morphology. */ +psReadout *psPhase2MaskCRs(psReadout *in, ///< Input image to be masked, and output + int algorithm, ///< Algorithm number to use + const void *params ///< Parameters for algorithm + ); +\end{prototype} + +This is one case in which the best choice of algorithm is not known, +and may change in the future. For this reason, we specify the inputs +as an integer to specify the choice of algorithm in addition to a +pointer to some parameters which will be decoded on the basis of the +choice of algorithm. + +Note that the input image is modified in-place. + +\subsection{psPhase2MaskOpticalDefects} + +Given an input image, a list of nearby stars, and a growing radius, +\code{psPhase2MaskOpticalDefects} shall mask optical defects on the +image. The API shall be the following: +\begin{prototype} +/** Masks optical defects in the input image */ +psChip *psPhase2MaskOpticalDefects(psChip *in, ///< Image to be masked (with astrometry), and output + const psDlist *catalog, ///< Catalog stars nearby: a list of psObjects + int grow ///< Number of pixels to grow + ); +\end{prototype} + +\tbd{It's not clear to me how this is accomplished apart from an +optical model of the camera. Put this one on the backburner?} + + +\section{Objects} + +To identify and measure objects, we must measure the PSF, and then +convolve the image by this PSF and apply a threshold. + +\subsection{Measure the PSF} + +Given an input image, a choice of algorithm with corresponding +parameters, \code{psPhase2MeasurePSF} shall return the PSF for the +image. The API shall be the following: +\begin{prototype} +/** Measures the PSF on the input image. Returns the PSF */ +psImage *psPhase2MeasurePSF(const psReadout *in, ///< Input image for which to measure the PSF + int algorithm, ///< Algorithm number to use + const void *params ///< Parameters for algorithm + ); +\end{prototype} + +This is another case where the algorithm is not currently clear, and +may change in the future. For this reason, we specify the inputs to +be a choice of algorithm, and a pointer to some parameters, which are +interpreted on the basis of the algorithm choice. + +\subsection{Find and measure objects} + +Given an input image, the PSF of that image, and a list of flux levels +at which to threshold, \code{psPhase2GetObjects} shall return a +readout with the \code{objects} member set to a list of objects. The +API shall be the following: +\begin{prototype} +/** Find and measure objects on the input image. Fills in the "objects" member of the psReadout. */ +/** THIS NEEDS WORK. */ +psReadout *psPhase2FindObjects(psReadout *in, ///< Input image on which to find objects, and output + const psImage *psf, ///< PSF to use to find objects + const psVector *levels ///< Threshold levels (std dev.s) + ); +\end{prototype} + +Note that the input image shall be modified in-place, only insofar as +the \code{objects} member shall be set to the list of objects on that +image. + +\tbd{This needs a lot more work.} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Astrometry} + +Given a chip, the elements of which have objects found and measured on +them, a list of catalog stars which lie on (or near) the chip, and +clipping parameters, \code{psPhase2Astrometry} shall fit an +astrometric solution. The API shall be the following: +\begin{prototype} +/** Corrects astrometry on the input chip */ +psChip *psPhase2Astrometry(psChip *in, ///< Input chip for which to do astrometry, and output + const psDlist *catalog, ///< Catalog stars on the chip: a list of psObjects + int nClips, ///< Number of clipping iterations + float clipLevel ///< Level at which to clip + ); +\end{prototype} + +Note that the input chip shall be modified in-place, only insofar as +the appropriate astrometry members shall be updated to correspond to +the fit solution (specifically, \code{in->cells[i]->cellToSky}, for +each \code{i}). + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Postage stamps} + +Postage stamps consist of subimages of specific objects of interest, +which are saved for more careful analysis of the pixels. + +\subsection{Specification} + +The postage stamps are specified by a center and size, in celestial +coordinates. We define a \code{psPostageStampSpec} to specify the +parameters for a postage stamp: +\begin{datatype} +/** Specification of a postage stamp: location on the sky */ +typedef struct { + psSphereCoord *center; ///< Centre of postage stamp + psSphereCoord *size; ///< Size of postage stamp +} psPostageStampSpec; +\end{datatype} + +The \code{center} shall be specified in ICRS coordinates, which is the +\PS{} standard system. The \code{size} shall be specified in +arcseconds on the sky. + +\subsection{Extracting postage stamps} + +Given an input chip, and a linked-list of regions on or near the input +chip, \code{psPhase2PostageStamps} shall output an array of subimages, +containing each of the regions. The API shall be the following: +\begin{prototype} +/** Return postage stamps of a set of regions */ +psImageArray *psPhase2PostageStamps(const psChip *in, ///< Chip from which to form postage stamps + const psDlist *regions ///< Regions to postage-stampise: a list of + ///< psPostageStampSpec + ); +\end{prototype} + +\code{regions} shall be a linked list of \code{psPostageStampSpec}s, +not all of which may correspond to legal positions on the input chip, +\code{in}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Fixed Pattern} + +The fixed pattern is a correction to the general astrometric solution +formed by summing the residuals from many observations. The intent is +to correct for higher-order distortions in the camera system on a +coarse grid (larger than individual pixels, but smaller than a single +cell). Hence, in addition to the offsets, we need to specify the size +and scale of the grid in $x$ and $y$, as well as the origin of the +grid. + +\begin{datatype} +typedef struct { + int nX; ///< Number of elements in x + int nY; ///< Number of elements in y + double x0; ///< Position of 0,0 corner on focal plane + double y0; ///< Position of 0,0 corner on focal plane + double xScale; ///< Scale of the grid + double yScale; ///< Scale of the grid + double **x; ///< The grid of offsets in x + double **y; ///< The grid of offsets in y +} psFixedPattern; +\end{datatype} + +The constructor for \code{psFixedPattern} shall be: +\begin{prototype} +psFixedPattern *psFixedPatternAlloc(double x0, double y0, + double xScale, double yScale, + const psImage *x, const psImage *y); +\end{prototype} +Here, \code{x0}, \code{y0}, \code{xScale} and \code{yScale} have the +same meaning as in the \code{psFixedPattern} structure. Note that the +values of the fixed pattern offsets are specified as images, the +values from which need to be copied into the \code{double **x} and +\code{double **y} of \code{psFixedPattern}, and that the number of +elements may be derived from the size of the images. + +\tbd{Usage of this type is not clear, and awaits prototyping --- do not +worry about coding this in detail yet.} + Index: /tags/ipp-docs/MSDR-08/doc/modules/configProposal.tex =================================================================== --- /tags/ipp-docs/MSDR-08/doc/modules/configProposal.tex (revision 22201) +++ /tags/ipp-docs/MSDR-08/doc/modules/configProposal.tex (revision 22201) @@ -0,0 +1,466 @@ +%%% $Id: configProposal.tex,v 1.1 2004-06-30 09:43:35 price Exp $ +\documentclass[panstarrs]{panstarrs} + +% basic document variables +\title{Pan-STARRS IPP Image Hierachy} +\subtitle{Configuration File Proposal} +\author{Paul Price} +\shorttitle{Image Hierachy Configurations} +\group{Pan-STARRS Algorithm Group} +\project{Pan-STARRS Image Processing Pipeline} +\organization{Institute for Astronomy} +\version{DR} +\docnumber{PSDC-430-???} +% note the use of the docnumber & version number: +% the complete PSDC document number is given by +% \thedocnumber-\theversion + +\setlength{\topsep}{-2pt} + +\begin{document} +\maketitle +\sloppy + +% -- Revision History -- +% provide explicit values for the old versions +% use '\theversion' for the current version (set above) +% use \hline between each table row +\RevisionsStart +% version Date Description +DR & 2004 Jun 29 & Draft \\ \hline +\RevisionsEnd + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\DocumentsInternal +PSCD-130-001 & PS-1 Design Reference Mission \\ \hline +PSCD-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline +PSCD-430-005 & Pan-STARRS IPP SRS \\ \hline +PSCD-430-006 & Pan-STARRS IPP ADD \\ \hline +PSCD-430-008 & Pan-STARRS IPP Architecture SDR \\ +\DocumentsExternal +Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ +\DocumentsEnd + +\tableofcontents +\pagebreak +\pagenumbering{arabic} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Introduction} + +Although the \PS{} Image Processing Pipeline (IPP) is required +primarily to support \PS{} operations, it is important to test it +thoroughly beforehand on data from other sources. The great +heterogeneity of the format of FITS files used in the astronomical +community means that we require a configuration file to specify how +images, both from other sources and from \PS{}, are converted into the +internal representation of the IPP. It is the object of this document +to propose a format for this configuration file. + +This document, it should be noted, is a preliminary proposal, and +hence the format has not yet been specified in the excruciating detail +required for a file format. If this proposal is perceived to be of +some use, it shall be expanded upon in the near future. + +\section{Requirements} + +It is desirable to associate a single name with an entire focal plane +image. Some observatories write images from the focal plane to +multiple FITS files. Others write to multiple extensions within the +same FITS file. + +The naming of particular information within FITS headers is extremely +heterogeneous. For example, the exposure time is variably stored as +\code{EXPTIME} and \code{EXPOSURE} throughout different observatories. +Furthermore, not all observatories store particular information, for +example, the hour angle of the observation, so in some cases, these +will have to be calculated on the basis of other information. Even +further, some values may not be able to be derived from information in +the FITS headers at all, but need external information, e.g., the +orientation of the focal plane elements on the sky. + +In some cases, the extension names/numbers may not be known \emph{a +priori}, as in the case of \PS{}, where a file may contain readouts +from multiple cells. Nevertheless, it should be possible to +reconstruct the focal plane on the basis of information recorded in +the headers. + +\section{Proposal} + +I propose the following format, which I believe satisfies the above +requirements. + +\subsection{File} + +The file shall be human-readable, plain text. + +\subsection{Comments} + +For the ease of parsing, we specify that only lines beginning with a +hash mark (\#) are to be ignored as comment lines. A comment may not +start in the course of a line. + +\subsection{Key/Value} + +Options are specified as key/value pairs. In some cases (e.g., the +specification of bias sections), there may be several ``values'' to +specify a single value, listed one after the other, delimited by +whitespace. + +\subsection{Hierarchy} + +The configuration file may be split into each of the hierarchical +levels: FPA, Chip, Cell and Readout. Options may be specified at any +of the levels. Options specified at one level flow down to all lower +levels, unless explicitly overridden. Some options are additive as +they flow down (for example, the offset of an element is cumulative as +one descends down the hierarchy, since the chip is offset so far from +the centre of the focal plane, while a cell is offset with respect to +the chip). + +The hierarchical levels may each be specified in turn, from the +highest level (FPA) to the lowest level (Readout), followed by a +commma-delimited list of allowed values for the name of an element of +the hierarchical level (or an asterisk if any value is permitted). In +options that follow, the name is denoted by the hierarchical level +name in angled brackets: \code{}, \code{}, \code{}, +\code{}. The name shall only be available at a hierarchical +level equal or lower than the corresponding level. + +For example: + +\begin{verbatim} +FPA * +Chip chip00,chip01,chip10,chip11 +Cell 1,2 +Readout * +\end{verbatim} + +In this example, the FPA may have any name, the chip name must be one +of \code{chip00}, \code{chip01}, \code{chip10} or \code{chip11}, the +cell name must be one of ``1'' or ``2'', and the readout may also have +any name. In this case, this likely represents a focal plane +consisting of four pieces of silicon, each with two amplifiers. + +Of course, the above example does not specify any options, which we +shall discuss next. + +\subsection{Options} + +By options, we mean the association of a particular value or FITS +header with a concept that is used in the IPP. It is the options that +provide the translation between the FITS headers and the concepts used +by the IPP, such as airmass, right ascension, exposure time, etc. + +An option is specified at the appropriate hierarchical level, indented +by a single tab (for clarity in reading by humans), and consists of a +keyword, followed by a value or FITS header from which the value may +be read. + +Below we note the options required for the IPP, along with the option +keyword: +\begin{itemize} +\item Right Ascension of the observation or of the hierarchical level; + \code{ra}. +\item Declination of the observation or of the hierarchical level; + \code{dec}. +\item Equinox for the coordinates; \code{equinox}. +\item Rotation of the observation or of the hierarchical level; + \code{rotation}. +\item Handedness of hierarchical level; \code{flip} --- either $+1$ or + $-1$. +\item Pixel scale of hierarchical level; \code{scale}. +\item Filename to retrieve the hierarchical level from; + \code{filename}. +\item FITS extension name from which to retrieve the hierarchical + level; \code{extname}. +\item FITS extension number from which to retrieve the hierarchical + level; \code{extnum}. +\item Bias regions; \code{bias} --- there may be more than one, + comma-delimited, and each specified in the format + \code{[x1:x2,y1:y2]}. +\item Axis to use for overscan subtraction; \code{axis} --- may be one + of \code{x}, \code{y}, \code{rows} or \code{columns}. +\item Detector regions to crop after overscan subtraction; + \code{detector}. +\item RA offset for the hierarchical level from the level(s) above; + \code{deltara}. +\item Dec offset for the hierarchical level from the level(s) above; + \code{deltadec}. +\item Rotation offset for the hierarchical level from the level(s) above; + \code{deltarot}. +\item Difference in handedness for the hierarchical level from the + level(s) above; \code{deltaflip}. +\item Multiplication of scale for the hierarchical level from the + level(s) above; \code{deltascale}. +\item Exposure time; \code{exptime}. +\item Airmass; \code{airmass}. +\item Hour angle; \code{ha}. +\item Sidereal time; \code{st}. +\item Latitude of observatory; \code{obslat}. +\item Longitude of observatory; \code{obslong}. +\item Altitude of observatory; \code{obsalt}. +\item Date of exposure; \code{date}. +\item Time of exposure; \code{time}. +\item Time reference; \code{timeref} --- may be \code{start}, + \code{stop} or \code{mid}, to specify when the observation time is + referenced to. +\item Format of date, to specify how to convert; \code{dateformat}. +\item Format of time, to specify how to convert; \code{timeformat}. +\item Extension containing OT shifts; \code{shiftsext}. +\item File containing OT shifts; \code{shiftsfile}. +\item Filter; \code{filter} --- multiple FITS headers may be + specified, comma-delimited, in order to account for instruments + which have multiple filter wheels. +\item Object/field name; \code{name}. +\item Observation type; \code{obstype}. +\item CCD dark time; \code{darktime}. +\item CCD gain; \code{gain}. +\item CCD read noise; \code{readnoise}. +\item Binning in x; \code{xbin}. +\item Binning in y; \code{ybin}. +\item Telescope altitude; \code{altitude}. +\item Telescope azimuth; \code{azimuth}. +\end{itemize} + +Note that the \code{deltara}, \code{deltadec}, \code{deltarot} and +\code{deltaflip} options should be propagated down the hierarchical +levels, while for the other options, the option defined last applies. + +Some unavailable options (e.g., not all observatories record the +sidereal time) may be calculated from other available options after +parsing. However, once this has been attempted, if any required +option is missing, and if the option cannot default to a reasonable +value, then the read should fail. + +A value specified for an option may include the name of an +hierarchical level in angled brackets (i.e., \code{}, +\code{}, \code{}, \code{}), which shall be +replaced iteratively by each of the allowed names, in turn, of the +hierarchical level. + +\subsubsection{Extensions not known a priori} + +In some cases, the extension name or number may not be known \emph{a +priori}. In particular, OTA data or NIR data from some instruments +may include multiple reads of a cell simply included in the same MEF +file along with other cells. + +In this case, there is no choice for the program but to create a table +of contents of the FITS file specified by scanning the extensions, and +associating different extensions with the appropriate hierarchical +level on the basis of a particular header. We specify the special +option, \code{id}, the corresponding value for which indicates which +header may be examined to indicate the name of the instance of a +hierarchical level. See the PS-GPC example for a demonstration. + +The \code{id} option may only be specified when the file name has +already been specified unambiguously at higher hierarchical level(s). + +Once it is known which extension corresponds to which +FPA/chip/cell/readout combination, then the focal plane may easily be +reconstructed. + +\subsection{Dependent options} + +In cases where the option is not available in the FITS header, the +value must be specified manually in the configuration file. However, +often the value is dependent upon the particular instance of an +hierarchical level. For such cases, we specify a syntax similar to +a \code{switch} statement in C. + +When an option is followed by a colon, then this means that it is +dependent upon the particular hierarchical level that is specified in +angled brackets following the colon. For example: +\begin{verbatim} + gain: +\end{verbatim} +specifies that the \code{gain} option is dependent upon the chip. + +A dependent option is followed on subsequent lines by the possible +values for the name of the hierarchical level, indented by two tabs +(for readability by humans) and the corresponding value for the +option. For example: +\begin{verbatim} +chip 00,01,10,11 + gain: + 00 1.0 + 01 2.1 + 10 3.2 + 11 4.3 + readnoise: + 00 5 + 01 6 + 10 7 + 11 8 + exptime EXPOSURE +\end{verbatim} + +This specifies that chip 00 has gain 1.0 and read noise 5, chip 01 has +gain 2.1 and read noise 6, chip 10 has gain 3.2 and read noise 7, and +chip 11 has gain 4.3 and read noise 8, and all chips have an exposure +time specified by the \code{EXPOSURE} FITS header. + +\section{Examples} + +I here offer some practical examples to help decipher these (no doubt, +cryptic) instructions. Not all options are specified, but hopefully +sufficient to give an idea of the format. + +\subsection{CFHT 12k MEF format} + +Some CFHT 12k images are written in MEF FITS format, with each of the +chips as separate extensions in a single FITS file for each exposure. + +\begin{verbatim} +FPA * + file .fits + ra RA + dec DEC + exptime OPENTIME +Chip chip1,chip2,chip3,chip4,chip5,chip6,chip7,chip8,chip9,chip10,chip11,chip12 + extnum + dra: + chip1 -110 + chip2 -90 + chip3 -70 + chip4 -50 + chip5 -30 + chip6 -10 + chip7 10 + chip8 30 + chip9 50 + chip10 70 + chip11 90 + chip12 110 + ddec: + chip1 -20 + chip2 -20 + chip3 -20 + chip4 -20 + chip5 -20 + chip6 -20 + chip7 20 + chip8 20 + chip9 20 + chip10 20 + chip11 20 + chip12 20 +Cell * +Readout * + bias [1:100,1:4096],[2150:2248,1:4096] + axis x + detector [101:2148,1:4096] +\end{verbatim} + +This specifies, firstly, that any name may be used to specify an FPA +image (I suspect that this is usually the case). Filenames are +specified on the FPA level (since all the chips are in the same FITS +file). If the FPA name given to the IPP is \code{img12345}, then the +corresponding filename that the IPP searches for is +\code{img12345.fits}. + +The Right Ascension, Declination and exposure time are specified for +the FPA, and read from the \code{RA}, \code{DEC} and \code{OPENTIME} +headers, respectively. These headers are to be found in the Primary +Header Unit (PHU), since an extension has not been specified. + +The allowed chip names are \code{chip1}, \code{chip2}, \code{chip3}, +\code{chip4}, \code{chip5}, \code{chip6}, \code{chip7}, \code{chip8}, +\code{chip9}, \code{chip10}, \code{chip11} and \code{chip12}. A chip +is to be found in the extension of the same name (as specified by the +\code{}. + +The RA and Dec offsets are specified, such that \code{chip1} is the +lower-left hand chip and \code{chip12} is the upper-right hand chip on +the focal plane. + +All Cell and Readout names are valid. Two bias regions are specified +for each chip, namely a prescan region, from column 1 to 100, and an +overscan region, from column 2150 to 2248. The bias region is to be +collapsed along the x axis, and following overscan subtraction, only +columns 101 to 2148 are of any interest. + + +\subsection{PS GPC} + +We expect that a \PS{} GigaPixel Camera exposure will be written as +several MEF files, one OTA to each file, with an extension within that +file for each cell and each readout (since there may be multiple +readouts per cell to incorporate fast guiding). To specify all 64 +OTAs and 64 cells in each would make for a long example, so we only +show 4 OTAs and 4 cells in each. + +This is a case in which the extensions are not known \emph{a priori}, +since there may exist multiple readouts for one cell in the same MEF +file as other cells. Note the use of the \code{id} option to specify +this. + +\begin{verbatim} +FPA * +Chip ota00,ota01,ota10,ota11 + file ..fits + ra RA + dec DEC + exptime EXPOSURE + dra: + ota00 -20 + ota01 20 + ota10 -20 + ota11 20 + ddec: + ota00 -20 + ota01 -20 + ota10 20 + ota11 20 +Cell cell00,cell01,cell10,cell11 + id CELLNUM + dra: + cell00 -10 + cell01 10 + cell10 -10 + cell11 10 + ddec: + cell00 -10 + cell01 -10 + cell10 10 + cell11 10 + shiftsext ot +Readout * + id READNUM + bias BIASSEC + axis x + detector DETSEC +\end{verbatim} + +Note first of all that all FPA names are valid (as is usually the +case). Valid chip names are \code{ota00}, \code{ota01}, \code{ota10} +and \code{ota11}. If the FPA name is \code{img12345}, then the +filename for \code{ota01} is \code{img12345.ota01.fits}. The RA, Dec +and exposure time are derived from the \code{RA}, \code{DEC} and +\code{EXPOSURE} headers, respectively. The chips are laid out in a +$2\times 2$ mosaic. + +Now we come to the cells. The allowed cell names are \code{cell00}, +\code{cell01}, \code{cell10} and \code{cell11}. However, the +extension these cells reside in is not known \emph{a priori}, and so +the \code{id} option specifies that the program should create a table +of contents of the extensions in the file, and determine the chip name +from the \code{CELLNUM} header in each extension. The cells are laid +out in a $2\times 2$ pattern on the chip. The OT shifts made for the +cell may be found in the extension of name \code{ot}. + +Finally, the readouts. Again, the particular readout is not known +\emph{a priori}, but the particular readout may be deduced from the +\code{READNUM} header. The bias region is specified by the +\code{BIASSEC} header, the bias region is to be collapsed along the x +axis, and the region to be cropped following the overscan subtraction +is specified by the \code{DETSEC} header. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\bibliographystyle{plain} +\bibliography{panstarrs} + +\end{document} Index: /tags/ipp-docs/MSDR-08/doc/modules/modules-delta =================================================================== --- /tags/ipp-docs/MSDR-08/doc/modules/modules-delta (revision 22201) +++ /tags/ipp-docs/MSDR-08/doc/modules/modules-delta (revision 22201) @@ -0,0 +1,80 @@ +#!/bin/sh + +# Copyright (C) 2005 Joshua Hoblitt +# +# $Id: modules-delta,v 1.1 2005-06-28 22:01:42 jhoblitt Exp $ + +MODULE=psmodule +DOC=ModulesSDRS.tex +UTIL_MODULE=utils +ROOTDIR=$HOME/tmp/$0 +WORKDIR=$ROOTDIR/work +UTILDIR=$ROOTDIR/$UTIL_MODULE +TEXBLOCK=$UTILDIR/texblock.pl +APIDELTA=$UTILDIR/apidelta.pl +CVS=/usr/bin/cvs +SSH=/usr/bin/ssh + +CVS_RSH=$SSH +CVSROOT=cvs.pan-starrs.ifa.hawaii.edu:/cvsroot/pan-starrs +export CVS_RSH CVSROOT + +PERL5LIB=$UTILDIR +export PERL5LIB + +STARTDIR=`pwd` + +# checkout pslib sources + +CVS_ERROR=0 + +if [ ! -d "$ROOTDIR/$MODULE" ] ; then + mkdir -p $ROOTDIR + cd $ROOTDIR + $CVS co $MODULE > /dev/null 2>&1 || CVS_ERROR=1 +else + cd "$ROOTDIR/$MODULE" + $CVS up -dP > /dev/null 2>&1 || CVS_ERROR=1 +fi + +if [ $CVS_ERROR -eq 1 ] ; then + echo "CVS error" + exit 1; +fi + +# checkout utils + +CVS_ERROR=0 + +if [ ! -d "$ROOTDIR/$UTIL_MODULE" ] ; then + mkdir -p $ROOTDIR + cd $ROOTDIR + $CVS co $UTIL_MODULE > /dev/null 2>&1 || CVS_ERROR=1 +else + cd "$ROOTDIR/$UTIL_MODULE" + $CVS up -dP > /dev/null 2>&1 || CVS_ERROR=1 +fi + +if [ $CVS_ERROR -eq 1 ] ; then + echo "CVS error from utils" + exit 1; +fi + +# compare sources + +if [ ! -d "$WORKDIR" ] ; then + mkdir -p $WORKDIR +fi + +cd $STARTDIR + +find $ROOTDIR/$MODULE/src -name "*.h" -exec cat {} \; > $WORKDIR/code.txt +$TEXBLOCK -b prototype -i $DOC > $WORKDIR/spec.txt +$TEXBLOCK -b datatype -i $DOC >> $WORKDIR/spec.txt +$APIDELTA --color=always --spec $WORKDIR/spec.txt --code $WORKDIR/code.txt + +# cleanup + +if [ -d "$WORKDIR" ] ; then + rm -rf $WORKDIR +fi Index: /tags/ipp-docs/MSDR-08/doc/modules/notes.config =================================================================== --- /tags/ipp-docs/MSDR-08/doc/modules/notes.config (revision 22201) +++ /tags/ipp-docs/MSDR-08/doc/modules/notes.config (revision 22201) @@ -0,0 +1,207 @@ +CFH12kMEF config: + +FPA * + file .fits + ra RA + dec DEC + exptime OPENTIME +Chip chip1,chip2,chip3,chip4,chip5,chip6,chip7,chip8,chip9,chip10,chip11,chip12 + extnum + dra: + chip1 -110 + chip2 -90 + chip3 -70 + chip4 -50 + chip5 -30 + chip6 -10 + chip7 10 + chip8 30 + chip9 50 + chip10 70 + chip11 90 + chip12 110 + ddec: + chip1 -20 + chip2 -20 + chip3 -20 + chip4 -20 + chip5 -20 + chip6 -20 + chip7 20 + chip8 20 + chip9 20 + chip10 20 + chip11 20 + chip12 20 +Cell * +Readout * + bias [2050:2148,1:4096] + axis x + use [1:2048,1:4096] + + +CFH12kSplit config: + +FPA: * +Chip: 1,2,3,4,5,6,7,8,9,10,11,12 + file ..fits + orientation: + 1 dRA dDec Rot Sign + 2 dRA dDec Rot Sign + 3 dRA dDec Rot Sign + 4 dRA dDec Rot Sign + 5 dRA dDec Rot Sign + 6 dRA dDec Rot Sign + 7 dRA dDec Rot Sign + 8 dRA dDec Rot Sign + 9 dRA dDec Rot Sign + 10 dRA dDec Rot Sign + 11 dRA dDec Rot Sign + 12 dRA dDec Rot Sign +Cell: * +Readout: * + bias [x1:x2,y1:y2] [x3:x4,y3:y4] [x5:x6,*] BIASHEADER1 BIASHEADER2 BIASHEADER3 + axis x + use [x1:x2,y1:y2] + + +MEGACAMraw config: + +FPA: * + file .fits +Chip: * + id CHIPHEADER + orientation: + 1 dRA dDec Rot Sign + 2 dRA dDec Rot Sign + ..... + 39 dRA dDec Rot Sign + 40 dRA dDec Rot Sign +Cell: * + id CELLHEADER + offset: + 1 dRA dDec + 2 dRA dDec +Readout: * + bias [x1:x2,y1:y2] [x3:x4,y3:y4] [x5:x6,*] BIASHEADER1 BIASHEADER2 BIASHEADER3 + axis x + use [x1:x2,y1:y2] + + +MEGACAMspliced config: + +FPA: * + file .fits +Chip: * + id CHIPHEADER + orientation: + 1 dRA dDec Rot Sign + 2 dRA dDec Rot Sign + ..... + 39 dRA dDec Rot Sign + 40 dRA dDec Rot Sign +Cell: * + id CELLHEADER +Readout: 2 + region: + 1 [x1:x2,y1:y2] [x3:x4,y3:y4] + 2 [x5:x6,y5:y6] [x7:x8,y7:y8] + bias: + 1 [x3:x4,y3:y4] + 2 [x7:x8,y7:y8] + axis x + use: + 1 [x1:x2,y1:y2] + 2 [x5:x6,y5:y6] + + +PS-GPC-raw config: + +FPA * +Chip 00,01,02,03,04,05,06,07,10,11,12,13,14,15,16,17,20,21,22,23,24,25,26,27,30,31,32,33,34,35,36,37,40,41,42,43,44,45,46,47,50,51,52,53,54,55,56,57,60,61,62,63,64,65,66,67,70,71,72,73,74,75,76,77 + file ..fits + orientation: + 00 dRA dDec Rot Sign + 01 dRA dDec Rot Sign + ..... + 76 dRA dDec Rot Sign + 77 dRA dDec Rot Sign +Cell: 00,01,02,03,04,05,06,07,10,11,12,13,14,15,16,17,20,21,22,23,24,25,26,27,30,31,32,33,34,35,36,37,40,41,42,43,44,45,46,47,50,51,52,53,54,55,56,57,60,61,62,63,64,65,66,67,70,71,72,73,74,75,76,77 + id CELLHEADER + offset: + 00 dRA dDec + 01 dRA dDec + ..... + 76 dRA dDec + 77 dRA dDec + shifts SHIFTS_EXT_NAME +Readout: * + id READOUT + bias BIASHEADER + axis x + use USEHEADER + + +PS-GPC-spliced config (corrected by Camera group on the summit): + +FPA: * + file .fits +Chip: 00,01,02,03,04,05,06,07,10,11,12,13,14,15,16,17,20,21,22,23,24,25,26,27,30,31,32,33,34,35,36,37,40,41,42,43,44,45,46,47,50,51,52,53,54,55,56,57,60,61,62,63,64,65,66,67,70,71,72,73,74,75,76,77 + id CHIPHEADER + orientation: + 00 dRA dDec Rot Sign + 01 dRA dDec Rot Sign + ..... + 76 dRA dDec Rot Sign + 77 dRA dDec Rot Sign +Cell: 00,01,02,03,04,05,06,07,10,11,12,13,14,15,16,17,20,21,22,23,24,25,26,27,30,31,32,33,34,35,36,37,40,41,42,43,44,45,46,47,50,51,52,53,54,55,56,57,60,61,62,63,64,65,66,67,70,71,72,73,74,75,76,77 + id CELLHEADER + offset: + 00 dRA dDec + 01 dRA dDec + ..... + 76 dRA dDec + 77 dRA dDec + shifts SHIFTS_EXT_NAME +Readout: 0 + use ALL + + + +Example for "small" GPC: + +FPA * + ra RA + dec DEC + exptime EXPOSURE +Chip ota00,ota01,ota10,ota11 + file ..fits + dra: + ota00 -20 + ota01 20 + ota10 -20 + ota11 20 + ddec: + ota00 -20 + ota01 -20 + ota10 20 + ota11 20 +Cell cell00,cell01,cell10,cell11 + id CELLNUM + dra: + cell00 -5 + cell01 5 + cell10 -5 + cell11 5 + ddec: + cell00 -5 + cell01 -5 + cell10 5 + cell11 5 + shiftsext ot +Readout * + id READNUM + bias BIASSEC + axis x + use DETSEC +