Index: /trunk/doc/modules/.cvsignore =================================================================== --- /trunk/doc/modules/.cvsignore (revision 970) +++ /trunk/doc/modules/.cvsignore (revision 970) @@ -0,0 +1,2 @@ +*.log *.dvi *.aux *.toc *.log *.out +phase2ADD.pdf phase2SDRS.pdf Index: /trunk/doc/modules/Makefile =================================================================== --- /trunk/doc/modules/Makefile (revision 970) +++ /trunk/doc/modules/Makefile (revision 970) @@ -0,0 +1,23 @@ +# $Id: Makefile,v 1.1 2004-06-10 01:48:28 price Exp $ + +PDFLATEX = pdflatex + +all : phase2SDRS.pdf phase2ADD.pdf + +%.pdf : %.tex + @find . -name $*.aux -cnewer $*.tex -exec rm {} \; + @if [ ! -f $*.aux ]; then \ + echo $(PDFLATEX) $*.tex; \ + env TEXINPUTS=.:LaTeX:$(TEXINPUTS): $(PDFLATEX) $*.tex; \ + fi && env TEXINPUTS=.:LaTeX:$(TEXINPUTS): $(PDFLATEX) $*.tex || $(RM) $*.pdf + +clean : + $(RM) *.log *.dvi *.aux *.toc *.lof *.out *~ core + +empty : clean + @for f in `find . -maxdepth 1 -name \*.pdf`; do \ + if [ -f `echo $$f | sed 's/pdf/tex/'` ]; then \ + echo $(RM) $$f; \ + $(RM) $$f; \ + fi; \ + done Index: /trunk/doc/modules/ModulesSDRS.tex =================================================================== --- /trunk/doc/modules/ModulesSDRS.tex (revision 970) +++ /trunk/doc/modules/ModulesSDRS.tex (revision 970) @@ -0,0 +1,568 @@ +%%% $Id: ModulesSDRS.tex,v 1.1 2004-06-10 01:48:28 price Exp $ +\documentclass[panstarrs]{panstarrs} + +% basic document variables +\title{Pan-STARRS IPP Phase 2} +\subtitle{Supplementary Design Requirements} +\author{Paul Price} +\shorttitle{Phase 2 SDRS} +\group{Pan-STARRS Algorithm Group} +\project{Pan-STARRS Image Processing Pipeline} +\organization{Institute for Astronomy} +\version{DR} +\docnumber{PSDC-430-???} +% note the use of the docnumber & version number: +% the complete PSDC document number is given by +% \thedocnumber-\theversion + +\setlength{\topsep}{-2pt} + +\begin{document} +\maketitle +\sloppy + +% -- Revision History -- +% provide explicit values for the old versions +% use '\theversion' for the current version (set above) +% use \hline between each table row +\RevisionsStart +% version Date Description +DR & 2004 Jun 7 & Draft \\ \hline +\RevisionsEnd + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\DocumentsInternal +PSCD-130-001 & PS-1 Design Reference Mission \\ \hline +PSCD-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline +PSCD-430-005 & Pan-STARRS IPP SRS \\ \hline +PSCD-430-006 & Pan-STARRS IPP ADD \\ \hline +PSCD-430-008 & Pan-STARRS IPP Architecture SDR \\ +\DocumentsExternal +Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ +\DocumentsEnd + +\tableofcontents +\pagebreak +\pagenumbering{arabic} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Introduction} + +This document describes the Pan-STARRS Image Processing Pipeline (IPP) +Phase 2 processing modules. Phase 2 is the processing stage wherein +the instrumental signatures are removed from the detector images, in +preparation for the combination of multiple images in Phase 4. + +The Phase 2 processing \textit{stage} is composed of several +\textit{modules} which perform the individual steps in the processing, +each of which will act upon a data element of the appropriate size. +For example, the bias subtraction module will act upon a detector +``readout'' (\code{psReadout}). It will be the responsibility of the +caller of the Phase 2 processing modules (i.e., the Phase 2 processing +stage, which will likely be script written in a high-level language +such as \tbd{Python}) to iterate over the components with which it is +charged. + +The Phase 2 processing modules are: +\begin{itemize} +\item Calculate the convolution kernel; +\item Convolve an image with the kernel; +\item Subtract bias; +\item Trim an image; +\item Correct for non-linearity; +\item Flat-field; +\item Subtract the background; +\item Mask bad pixels; +\item Mask cosmic rays; +\item Mask optical defects; +\item Measure the PSF; +\item Find and measure objects; +\item Correct astrometry; and +\item Get postage stamps. +\end{itemize} + +Each of these shall be discussed in turn, below. + +These modules are built on top of, and are dependent upon, the \PS{} +Library, PSLib. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Convolution} + +If images on the sky are obtained in Orthogonal Transfer (OT) mode, +then the calibration frames used to correct them must be convolved by +a kernel derived from the list of OT shifts made during the exposure. + +\subsection{Calculate the convolution kernel} + +Given a list of pixel shifts made in the course of OT guiding, +\code{psPhase2GetKernel} shall return an image that represents the +kernel. The API shall be the following: +\begin{verbatim} +/** Returns the kernel of OT shifts made during the course of the exposure. */ +psImage *psPhase2GetKernel(psVector *xShifts, ///< List of OT shifts in x; integers + psVector *yShifts ///< List of OT shifts in y; integers; xshifts->n == yShifts->n + ); +\end{verbatim} + +The elements of the shift vectors should be of an integer type, +otherwise the values shall be truncated to integers. The output +kernel shall be normalised such that the sum over the kernel is unity. + +The format of the shifts vectors will be a list of the net shift +position relative to the starting point for each timestep (there will +likely be about 10~Hz $\times$ 30~sec = 300 timesteps). + +If the vectors are not of the same size, then the function shall +generate an error. + +\subsection{Convolve an image with the kernel} + +Given an input image and the convolution kernel, +\code{psPhase2ConvolveWithKernel} shall convolve the input image with +the kernel and return the convolved image. The API shall be the +following: +\begin{verbatim} +/** Returns an image that is the result of convolving the input image with the specified kernel. */ +psImage *psPhase2ConvolveWithKernel(psImage *out, ///< Output image, or NULL + const psImage *in, ///< Input image to be convolved + const psImage *kernel ///< Kernel by which to convolve + ); +\end{verbatim} + +Two methods shall be used to do the convolution: for small kernels +(smaller than 200 total pixels), the convolution shall be performed in +real space; for large kernels (larger than 200 total pixels), the +convolution shall be performed using Fast Fourier Transforms (FFTs). + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + + +\section{Bias subtraction and trim} + +The following modules correct detector images for the electronic +pedestal introduced by the readout electronics. + +\subsection{Image region specification} + +In order to specify regions on the detector image which may be used to +determine the bias levels (i.e., a prescan or overscan region), we +define the following structure: +\begin{verbatim} +/** Image regions */ +typedef struct { + const int x0, y0; ///< Offset to region + const int nX, nY; ///< Size of region +} psImageRegion; +\end{verbatim} + +A \code{psImageRegion} specifies the offset from $(0,0)$ on the +detector\footnote{Note that an image may be smaller than the entire +area of the detector if the detector is windowed.} and the size of the +region. + +We also require a corresponding constructor and destructor: +\begin{verbatim} +/** Constructor */ +psImageRegion *psImageRegionAlloc(int x0, ///< x offset to region + int y0, ///< y offset to region + int nX, ///< Size of region in x + int nY ///< Size of region in y + ); +/** Destructor */ +void psImageRegionFree(psImageRegion *reg ///< Region to destroy + ); +\end{verbatim} + +Multiple image regions may be specified using a linked list +(\code{psList}) of \code{psImageRegion}s. + +\subsection{Bias subtraction} + +Given an input image and various other parameters, +\code{psPhase2SubtractBias} shall subtract the bias from the image. +The API shall be the following: +\begin{verbatim} +/** Subtracts an overscan and bias from the input image. */ +psReadout *psPhase2SubtractBias(psReadout *in, ///< Input image to be de-biased, and output + psPolynomial1D *polySpec, ///< Polynomial specification to use for fit, + ///< outputed + const psList *regions, ///< Linked list of psImageRegion types. + psOverscanAxis overscanAxis, ///< Overscan axis + const psStatsOptions *stat, ///< Statistic to use + const psImage *bias ///< Bias (or dark) image to subtract + ); +\end{verbatim} + +The input image shall have the bias subtracted in-place. + +The polynomial specification, \code{polySpec}, specifies the order of +a polynomial fit which may be made to the overscan if the +\code{overscanAxis} is \code{PS_OVERSCAN_ROWS_FIT} or +\code{PS_OVERSCAN_COLUMNS_FIT}. \code{polySpec} shall, upon return, +contain the coefficients of the overscan fit, if performed. + +The prescan and/or overscan regions to be used are specified in +\code{regions}. + +The \code{overscanAxis} specifies how the prescan/overscan subtraction +is to be performed. It is an enumerated type: +\begin{verbatim} +/** Overscan axis */ +typedef enum { + PS_OVERSCAN_NONE = 0, ///< No overscan subtraction + PS_OVERSCAN_ROWS = 1, ///< Subtract rows + PS_OVERSCAN_COLUMNS = 2, ///< Subtract columns + PS_OVERSCAN_ROWS_FIT = -1, ///< Subtract a fit to rows + PS_OVERSCAN_COLUMNS_FIT = -2, ///< Subtract a fit to columns + PS_OVERSCAN_ALL = 3 ///< Subtract the statistic of all pixels in overscan region +} psOverscanAxis; +\end{verbatim} +No prescan/overscan subtraction shall be performed if the +\code{overscanAxis} is \code{PS_OVERSCAN_NONE} or if \code{regions} is +\code{NULL}. The subtraction shall be on the basis of rows or columns +if the \code{overscanAxis} is \code{PS_OVERSCAN_ROWS} or +\code{PS_OVERSCAN_COLUMNS}, respectively. If the \code{overscanAxis} +is \code{PS_OVERSCAN_ROWS_FIT} or \code{PS_OVERSCAN_COLUMNS_FIT}, a +fit to the prescan/overscan region shall be subtracted from the rows +or columns, respectively. Finally, if the \code{overscanAxis} is +\code{PS_OVERSCAN_ALL}, then the appropriate statistic (specified by +\code{stat}) of all of the pixels in the prescan/overscan region(s) +shall be subtracted from the image. + +The statistic to use in combining multiple pixels in the +prescan/overscan regions is specified by \code{stat}. Legal values +shall be \code{PS_STAT_SAMPLE_MEAN}, \code{PS_STAT_SAMPLE_MEDIAN}, +\code{PS_STAT_ROBUST_MEAN}, \code{PS_STAT_ROBUST_MEDIAN}, +\code{PS_STAT_ROBUST_MODE}. \tbd{Note in particular that +%\code{PS_STAT_CLIPPED_MEAN} +a clipped mean is not supported, because that means we have to supply +clipping parameters which isn't very neat.} + +A full-frame bias (or dark) image shall be subtracted if \code{bias} +is non-NULL. Note that the input image, \code{in}, and the +\code{bias} image need not be the same size, but the function shall +use the offsets in the image (\code{in->x0} and \code{in->y0}) to +determine the appropriate offsets to obtain the correct pixel on the +\code{bias}. In the event that the \code{bias} image is too small +(i.e., pixels on the input image refer to pixels outside the range of +the \code{bias} image), the function shall generate an error. + + +\subsection{Trim} + +Given an input image and region to keep, \code{psPhase2Trim} shall +trim the input image, leaving only the specified region. The API +shall be the following: +\begin{verbatim} +/** Trims the input image to remove the edges corrupted by OT shifting, and the overscan. */ +psReadout *psPhase2Trim(psReadout *in, ///< Input image to be trimmed, and output + const psImageRegion *region ///< Region to keep + ); +\end{verbatim} + +Note that the trim shall be performed on the input image in-place. + +The offset of the input image (\code{in->x0} and \code{in->y0}) shall +be updated to reflect the trim. + +The function shall generate an error if the \code{region} is outside +the bounds of the input image. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Preparation for photometry} + +The following modules take an image and process it so that counts are +proportional to the flux from an astronomical source. + +\subsection{Non-linearity} + +Given an input image, and the coefficients of a polynomial, +\code{psPhase2CorrectNonLinearity} shall correct the input image for +non-linearity by applying the polynomial to the flux of each pixel. +The API shall be the following: +\begin{verbatim} +/** Applies the correction for detector non-linearity. */ +psReadout *psPhase2CorrectNonLinearity(psReadout *in, ///< Input image to be corrected, and output + psPolynomial1D *coeff ///< Polynomial with which to correct + ); +\end{verbatim} + +The polynomial coefficients, \code{coeff}, will be supplied by the +caller, likely from the image metadata. + +\subsection{Flat-fielding} + +Given an input image and a flat-field image, \code{psPhase2FlatField} +shall divide the input image by the flat-field image. The API shall be the following: +\begin{verbatim} +/** Flat-fields the image. */ +psReadout *psPhase2FlatField(psReadout *in, ///< Input image to be flat-fielded, and output + const psImage *flat ///< Flat-field image + ); +\end{verbatim} + +Note that the input image, \code{in}, and the flat-field image, +\code{flat}, need not be the same size, since the input image may +already have been trimmed (following overscan subtraction), but the +function shall use the offsets in the image (\code{in->x0} and +\code{in->y0}) to determine the appropriate offsets to obtain the +correct pixel on the flat-field. In the event that the \code{flat} +image is too small (i.e., pixels on the input image refer to pixels +outside the range of the \code{flat} image), the function shall +generate an error. + +Pixels which are negative or zero in the \code{flat} shall be masked +in the input image. + +The function shall not normalise the \code{flat}; this responsibility +is left to the caller. + +\subsection{Subtract sky} + +Given an input image, a polynomial specifying the order of a desired +fit, a list of template background images, a list of catalog stars on +or near the chip, and parameters specifying how to grow the radius of +the objects, and the binning factor, \code{psPhase2SubtractSky} shall +fit and subtract the background of the image. The API shall be the +following: +\begin{verbatim} +/** Subtracts the sky background. Needs work. */ +psReadout *psPhase2SubtractSky(psReadout *in, ///< Input image to be sky-subtracted, and output + psPolynomial2D *poly, ///< Polynomial specification, returns coeffcients + psVector *outNorms, ///< Normalisations of the template sky images: output + const psList *skyImages, ///< Template sky images: a list of type psImage, or + ///< NULL + const psList *catalog, ///< Sources nearby to mask when measuring the + ///< background: a list of type psObject, or NULL + float radiusMultiply, ///< Multiplying factor of object radius for mask + float radiusAdd, ///< Additive factor to grow the object masks + int binFactor ///< Binning factor to use on image before solving for polynomial + ); +\end{verbatim} + +Note that the input image shall be subtracted in-place. The function +shall return the subtracted image, and also update the polynomial, +\code{poly}, to hold the polynomial coefficients used in the +subtraction, and update the vector \code{outNorms} to hold the +normalisations of each of the template background images used in the +subtraction. The \code{outNorms} vector shall be resized if necessary +to match the number of template sky images in the list +\code{skyImages}. + +Note that the input image, \code{in}, and the template sky images, +\code{skyImages}, need not be the same size, since the input image may +already have been trimmed (following overscan subtraction), but the +function shall use the offsets in the image (\code{in->x0} and +\code{in->y0}) to determine the appropriate offsets to obtain the +correct pixels. In the event that one (or more) of the +\code{skyImages} is too small (i.e., pixels on the input image refer +to pixels outside the range of the \code{skyImages} image), the +function shall generate an error. + +If the size of a source in the \code{catalog} is \code{size}, then the +radius to mask around the source before fitting the background shall +be \code{radiusMultiply} $\times$ \code{size} + \code{radiusAdd}. + +\tbd{Are the catalog stars in celestial or pixel coordinates? I lean +to pixel coordinates.} + +When fitting a polynomial, the function shall bin the input image by +\code{binFactor} in order to reduce the required processing time. + +\code{poly} may be \code{NULL} in order to specify that only the +template sky images should be used in the fitting, and no polynomial +terms. \code{outNorms} may be \code{NULL} if the caller is not +interested in the normalisations. \code{skyImages} may be \code{NULL} +in order to specify that there are no input sky template images. +\code{catalog} may be \code{NULL} in order to specify that there are +no objects to mask. If both \code{poly} and \code{skyImages} are +\code{NULL}, then the function shall simply return the input image +with no processing. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Masking} + +The following modules mask various defects on an image. + +\subsection{Bad pixels} + +Given an input image, a bad pixel mask, a corresponding value in the +bad pixel mask to mask in the input image, a saturation level, and a +growing radius, \code{psPhase2MaskBadPixels} shall mask in the input +image those pixels in the bad pixel mask that match the value to mask. +The API shall be the following: +\begin{verbatim} +/** Returns an image that has the bad pixels masked. Also masks saturated pixels */ +psReadout *psPhase2MaskBadPixels(psReadout *in ///< Image to be masked, and output + const psImage *mask, ///< Bad pixel mask to apply + int maskVal, ///< Mask the pixels for which mask & maskVal > 0. + float sat, ///< Saturation level: pixels brighter than this level are masked + int grow ///< Radius to grow the bad pixels + ); +\end{verbatim} + +Note that the input image, \code{in}, is modified in-place. All +pixels in the \code{mask} which satisfy the \code{maskVal} or have +corresponding flux greater than \code{sat}, shall have their +corresponding pixels in the input image, \code{in}, along with all +pixels within the \code{grow} radius masked. + +\subsection{Cosmic rays} + +Given an input image, a choice of a particular algorithm with +corresponding parameters, \code{psPhase2MaskCRs} shall mask cosmic +rays on the input image on the basis of their morphology. The API shall be the following: +\begin{verbatim} +/** Masks Cosmic Rays on the input image on the basis of morphology. */ +psReadout *psPhase2MaskCRs(psReadout *in, ///< Input image to be masked, and output + int algorithm, ///< Algorithm number to use + const void *params ///< Parameters for algorithm + ); +\end{verbatim} + +This is one case in which the best choice of algorithm is not known, +and may change in the future. For this reason, we specify the inputs +as an integer to specify the choice of algorithm in addition to a +pointer to some parameters which will be decoded on the basis of the +choice of algorithm. + +Note that the input image is modified in-place. + +\subsection{psPhase2MaskOpticalDefects} + +Given an input image, a list of nearby stars, and a growing radius, +\code{psPhase2MaskOpticalDefects} shall mask optical defects on the +image. The API shall be the following: +\begin{verbatim} +/** Masks optical defects in the input image */ +psChip *psPhase2MaskOpticalDefects(psChip *in, ///< Image to be masked (with astrometry), and output + const psDlist *catalog, ///< Catalog stars nearby: a list of psObjects + int grow ///< Number of pixels to grow + ); +\end{verbatim} + +\tbd{It's not clear to me how this is accomplished apart from an +optical model of the camera. Put this one on the backburner?} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Objects} + +To identify and measure objects, we must measure the PSF, and then +convolve the image by this PSF and apply a threshold. + +\subsection{Measure the PSF} + +Given an input image, a choice of algorithm with corresponding +parameters, \code{psPhase2MeasurePSF} shall return the PSF for the +image. The API shall be the following: +\begin{verbatim} +/** Measures the PSF on the input image. Returns the PSF */ +psImage *psPhase2MeasurePSF(const psReadout *in, ///< Input image for which to measure the PSF + int algorithm, ///< Algorithm number to use + const void *params ///< Parameters for algorithm + ); +\end{verbatim} + +This is another case where the algorithm is not currently clear, and +may change in the future. For this reason, we specify the inputs to +be a choice of algorithm, and a pointer to some parameters, which are +interpreted on the basis of the algorithm choice. + +\subsection{Find and measure objects} + +Given an input image, the PSF of that image, and a list of flux levels +at which to threshold, \code{psPhase2GetObjects} shall return a +readout with the \code{objects} member set to a list of objects. The +API shall be the following: +\begin{verbatim} +/** Find and measure objects on the input image. Fills in the "objects" member of the psReadout */ +/** THIS NEEDS WORK. */ +psReadout *psPhase2GetObjects(psReadout *in, ///< Input image on which to find objects, and output + const psImage *psf, ///< PSF to use to find objects + const psVector *levels ///< Threshold levels (std dev.s) + ); +\end{verbatim} + +Note that the input image shall be modified in-place, only insofar as +the \code{objects} member shall be set to the list of objects on that +image. + +\tbd{This needs a lot more work.} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Astrometry} + +Given a chip, the elements of which have objects found and measured on +them, a list of catalog stars which lie on (or near) the chip, and +clipping parameters, \code{psPhase2Astrometry} shall fit an +astrometric solution. The API shall be the following: +\begin{verbatim} +/** Corrects astrometry on the input chip */ +psChip *psPhase2Astrometry(psChip *in, ///< Input chip for which to do astrometry, and output + const psDlist *catalog, ///< Catalog stars on the chip: a list of psObjects + int nClips, ///< Number of clipping iterations + float clipLevel ///< Level at which to clip + ); +\end{verbatim} + +Note that the input chip shall be modified in-place, only insofar as +the appropriate astrometry members shall be updated to correspond to +the fit solution (specifically, \code{in->cells[i]->cellToSky}, for +each \code{i}). + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Postage stamps} + +Postage stamps consist of subimages of specific objects of interest, +which are saved for more careful analysis of the pixels. + +\subsection{Specification} + +The postage stamps are specified by a center and size, in celestial +coordinates. We define a \code{psPostageStampSpec} to specify the +parameters for a postage stamp: +\begin{verbatim} +/** Specification of a postage stamp: location on the sky */ +typedef struct { + psSphereCoord *center; ///< Centre of postage stamp + psSphereCoord *size; ///< Size of postage stamp +} psPostageStampSpec; +\end{verbatim} + +The \code{center} shall be specified in ICRS coordinates, which is the +\PS{} standard system. The \code{size} shall be specified in +arcseconds on the sky. + +\subsection{Extracting postage stamps} + +Given an input chip, and a linked-list of regions on or near the input +chip, \code{psPhase2PostageStamps} shall output an array of subimages, +containing each of the regions. The API shall be the following: +\begin{verbatim} +/** Return postage stamps of a set of regions */ +psImageArray *psPhase2PostageStamps(const psChip *in, ///< Chip from which to form postage stamps + const psDlist *regions ///< Regions to postage-stampise: a list of + ///< psPostageStampSpec + ); +\end{verbatim} + +\code{regions} shall be a linked list of \code{psPostageStampSpec}s, +not all of which may correspond to legal positions on the input chip, +\code{in}. + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\bibliographystyle{plain} +\bibliography{panstarrs} + +\end{document} +