Index: trunk/doc/modules/ModulesSDRS.tex =================================================================== --- trunk/doc/modules/ModulesSDRS.tex (revision 9713) +++ trunk/doc/modules/ModulesSDRS.tex (revision 9744) @@ -1,3 +1,3 @@ -%%% $Id: ModulesSDRS.tex,v 1.79 2006-10-21 04:27:38 price Exp $ +%%% $Id: ModulesSDRS.tex,v 1.80 2006-10-25 01:48:17 price Exp $ \documentclass[panstarrs,spec]{panstarrs} @@ -627,8 +627,10 @@ image detrending steps are: \begin{itemize} -\item Subtract bias; +\item Subtract overscan, bias and dark; \item Correct for non-linearity; +\item Shutter correction; \item Flat-field; \item Mask bad pixels; +\item Fringe subtraction; \item Subtract the background; \item Mask cosmic rays; @@ -636,4 +638,9 @@ \end{itemize} +We also include here some of the (more complicated) functions for +generating the image detrend calibrations (fringes, flat-field +normalisation), and determining the suitable calibrations from the +detrend database. + \subsection{Bias subtraction} \label{sec:bias} @@ -641,48 +648,21 @@ The bias subtraction module provides a facility to correct detector images for the electronic pedestal introduced by the readout -electronics. - -Given an input image and various other parameters, -\code{pmSubtractBias} shall subtract the bias from the image: - -\begin{prototype} -pmReadout *pmSubtractBias(pmReadout *in, pmOverscanOptions *overscanOpts, - psRegion imageRegion, psList *overscanRegions, - const pmReadout *bias, const pmReadout *dark); -\end{prototype} - -Three types of bias correction may optionally be performed on the -input image, \code{in}. The first is the subtraction of an overscan. -Multiple overscan regions may be specified and fit as a function of -row (or column). The second is the subtraction of a full-frame bias -image. The third is the subtraction of a suitably scaled full-frame -dark image. - -The input image, \code{in}, shall have the bias subtracted in-place. -The input image may be of type U16, S32, or F32. The region of the -input image that shall have the overscan or full-frame subtractions -applied is specified by \code{imageRegion}. - -Overscan subtraction is performed if \code{overscanOpts} is -non-\code{NULL} (see \S\ref{sec:overscan}). \code{overscanRegions} -shall be a list of \code{psRegion}s that specify the regions that -comprise the overscans. - -A \code{bias} frame shall be subtracted pixel-by-pixel from the input -image if \code{bias} is non-NULL. If \code{dark} is non-\code{NULL}, -then the dark image, scaled by the ratio of dark times (from -\code{CELL.DARKTIME}) shall be subtracted pixel-by-pixel from the -input image. The full-frame subtractions (both bias and dark) should -only be performed on the image region specified by -\code{CELL.TRIMSEC}. Note that the input image, \code{in}, and the -\code{bias} and \code{dark} frames need not be the same size, but the -function shall use the offsets in the image (\code{in->x0} and -\code{in->y0}) to determine the appropriate offsets to obtain the -correct pixel on the \code{bias}. In the event that the \code{bias} -image is too small (i.e., pixels on the input image refer to pixels -outside the range of the \code{bias} image), the function shall -generate an error. Any pixels masked in the \code{bias} or -\code{dark} shall also be masked in the output. The bias and dark -images may be copied to the same type as the input image if required. +electronics. The pedestal is measured from one or more of the +overscan, bias frame and dark frames. + +The following function subtracts the overscan, bias and/or dark from +the input readout. The overscan is taken from the \code{bias} member +of the input \code{pmReadout}, and can be turned off by supplying a +\code{NULL} value for \code{overscanOpts}. A bias image is subtracted +if \code{bias} is non-\code{NULL}, and a dark frame is scaled (by +\code{CELL.DARKTIME}) and subtracted if \code{dark} is +non-\code{NULL}. +\begin{prototype} +bool pmSubtractBias(pmReadout *in, ///< Input readout, to be overscan/bias/dark corrected + pmOverscanOptions *overscanOpts, ///< Options for overscan subtraction, or NULL + const pmReadout *bias, ///< Bias image to subtract, or NULL + const pmReadout *dark ///< Dark image to scale and subtract, or NULL + ); +\end{prototype} @@ -696,41 +676,25 @@ typedef struct { // Inputs - bool single; // Reduce all overscan regions to a single value? - bool scanRows; // Scan direction was rows? (otherwise columns) - pmFit fitType; // Type of fit to overscan - unsigned int order; // Order of polynomial, or number of spline pieces - psStats *stat; // Statistic to use when reducing the minor direction + bool single; ///< Reduce all overscan regions to a single value? + pmFit fitType; ///< Type of fit to overscan + unsigned int order; ///< Order of polynomial, or number of spline pieces + psStats *stat; ///< Statistic to use when reducing the minor direction // Outputs - psPolynomial1D *poly; // Result of polynomial fit - psSpline1D *spline; // Result of spline fit + psPolynomial1D *poly; ///< Result of polynomial fit + psSpline1D *spline; ///< Result of spline fit } pmOverscanOptions; \end{datatype} -The mode in which the overscan is subtracted is specified by the -\code{single} boolean. If \code{single} is \code{true}, then the -entire overscan region is reduced to a single value using the -\code{stat}. If \code{single} is \code{false}, the overscan shall be -reduced along the dimension specified by \code{scanRows} (rows if -\code{scanRows} is true; otherwise columns). - -If the overscan is not defined for each row/column, -\code{pmSubtractBias} shall generate an error if \code{fitType} is -\code{PM_FIT_NONE}; otherwise, the function shall shall generate a -warning and the undefined values shall be interpolated using the -provided functional form. - -The statistic to use in combining multiple pixels in the -prescan/overscan regions is specified by \code{stat}. \code{stat} is -of type \code{psStats} instead of simply \code{psStatsOptions} so that -clipping levels may be specified, if desired. In the event that -multiple options are specified by \code{stats}, a warning shall be -generated, and the option with the highest priority shall be used, -according to the following priority order: \code{PS_STAT_SAMPLE_MEAN}, -\code{PS_STAT_SAMPLE_MEDIAN}, \code{PS_STAT_CLIPPED_MEAN}, -\code{PS_STAT_ROBUST_MEAN}, \code{PS_STAT_ROBUST_MEDIAN}, -\code{PS_STAT_ROBUST_MODE}. - -\code{fitType} is an enumerated type which specifies the type of fit -to employed on the overscan vector: +The overscan subtraction may be performed by reducing all overscan +regions to a single value (e.g., if there is no structure); or the +overscan may be fit perpendicular to the read direction (usually the +columns) with a particular functional form; or a single value may be +subtracted for each read/scan without fitting (if the structure defies +characterisation). In any case, statistics are required to reduce +multiple values to a single value (either for the scan, or for the +entire overscan regions). + +The \code{fitType} is an enumerated type which specifies the type of +fit to employed on the overscan vector: \begin{datatype} typedef enum { @@ -748,9 +712,11 @@ allocated and updated with the results of the fit. -The allocator for a \code{pmOverscanOptions} shall be: -\begin{prototype} -pmOverscanOptions *pmOverscanOptionsAlloc(bool single, bool scanRows, - pmFit fitType, unsigned int order, - psStats *stat); +The allocator for a \code{pmOverscanOptions} is: +\begin{prototype} +pmOverscanOptions *pmOverscanOptionsAlloc(bool single, ///< Reduce all overscan regions to a single value? + pmFit fitType, ///< Type of fit to overscan + unsigned int order, ///< Order of polynomial, or number of splines + psStats *stat ///< Statistic to use + ); \end{prototype} @@ -762,83 +728,185 @@ since either (or both) might be used to specify the correction. -These operations act only on the region of the readout specified by -\code{CELL.TRIMSEC}. - -The first, \code{pmNonLinearityPolynomial} shall correct the input -image for non-linearity by replacing the flux in each pixel of the -input image, \code{in}, with the result of the specified polynomial, -\code{coeff}, acting on the flux. The API shall be the following: - -\begin{prototype} -pmReadout *pmNonLinearityPolynomial(pmReadout *in, const psPolynomial1D *coeff); -\end{prototype} - -The polynomial coefficients, \code{coeff}, will be supplied by the -caller, likely from the image metadata. - -The second function, \code{pmNonLinearityLookup} shall correct -the input image for non-linearity by using a lookup table. The API -shall be the following: - -\begin{prototype} -pmReadout *pmNonLinearityLookup(pmReadout *in, const char *filename); -\end{prototype} - -For each pixel in the input image, the function shall replace the flux -with the corresponding value from the supplied lookup table, specified -by the \code{filename}. The lookup table file shall consist of two -columns of data, the first being the original flux value and the -second being the replaced flux value. The file shall be in a format -suitable for reading by \code{psLookupTableRead}. - -Both \code{pmNonLinearityPolynomial} and \code{pmNonLinearityLookup} -shall modify the input image in-place. The input image may be of -type U16, S32, or F32. +The first, \code{pmNonLinearityPolynomial}, applies a polynomial to +the flux of each pixel in the input image to determine the corrected +flux. The second function, \code{pmNonLinearityLookup} performs +linear interpolation on the table (specified by two vectors) to +determine the corrected flux. + +\begin{prototype} +pmReadout *pmNonLinearityPolynomial(pmReadout *in, ///< Input image, to correct + const psPolynomial1D *coeff ///< Polynomial for non-linearity correction + ); +pmReadout *pmNonLinearityLookup(pmReadout *in, ///< Input image, to correct + const psVector *inFlux, ///< Table column with input fluxes + const psVector *outFlux ///< Table column with output fluxes + ); +\end{prototype} + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\subsection{Shutter correction} + +A mechanical shutter may not yield uniform exposure times as a +function of position on the detector. The typical error consists of a +constant exposure-time offset relative to the requested value, +i.e. the exposure time is $T_o + \delta T(x,y)$. The exposure error, +$\delta T$, may be measured with the following scheme. Obtain a set +of exposures with different exposures times taken of the same +flat-field source; the source must be spatially stable between the +exposures, but need not have a stable amplitude. For an illuminating +flux of intensity $F(x,y) = F_o f(x,y)$, the signal recorded by any +pixel in the detector is given by: $S(t,x,y) = F_o(t) f(x,y) (T_o + +\delta T(x,y))$ where $F_o$ is the $F_o(t)$ is the (variable) overall +intensity of the illuminating source and $f(x,y)$ is the spatial +illumination patter times the flat-field response. Choose a reference +location in the image (e.g., the detector center) and divide by the +value of that region (i.e., mean or median): + +\begin{eqnarray} +s(t,x,y) & = & S(t,x,y) / S(t,0,0) \\ +s(t,x,y) & = & F_o(t) f(x,y) (T_o + \delta T(x,y)) / F_o(t) f(0,0) (T_o + \delta T(0,0)) \\ +s(t,x,y) & = & f(x,y) (T_o + \delta T(x,y)) / f(0,0) (T_o + \delta T(0,0)) \\ +\end{eqnarray} + +We can absorb the term $f(0,0)$ into $f(x,y)$, as we have no +motivation for the scale of $f(x,y)$. For any single pixel, over the +set of exposures, we thus need to solve for $\delta T(x,y)$, $\delta +T(0,0)$, and $f'(x,y)$ in the equation: + +\begin{equation} +s(t,x,y) = f'(x,y) (T_o + dT(x,y)) / (T_o + \delta T(0,0)) +\end{equation} + +We avoid directly fitting these values, as the process would be a +non-linear least-squares problem for every pixel in the image, and +thus very time consuming. There are linear options which may be used +instead. First, as $T_o$ goes to a large value, $s$ approaches the +value of $f'(x,y)$. Next, as $T_o$ goes to a very small value, $s$ +approaches the value of $f'(x,y) \delta T(x,y)/ \delta T(0,0)$. +Finally, when s has the value of $f'(x,y)*(1 + \delta T(x,y)/ \delta +T(0,0))/2$, then $T_o$ has the value of $\delta T(0,0)$. With data +points covering a reasonable dynamic range, we can solve for these +three values by interpolation and/or extrapolation. + +To take the strategy one step further, we could use the above recipe +to obtain a guess for the three parameters and then apply non-linear +fitting to solve more accurately for the parameters. If we limit this +operation to a handful of positions in the image (user defined, but +the obvious choice would be positions near the center, edges, and +corners), then we may determine a good value for $\delta T(0,0)$. +Since there is only one $\delta T(0,0)$ for the image, we can apply +the resulting measurement to the rest of the pixels in the image. If +$\delta T(0,0)$ is not a free parameter, then the fitting process is +linear in terms of $\delta T(x,y)$ and $f'(x,y)$. + +\subsubsection{Components to creating a shutter correction} + +We define a shutter correction, applicable for a single pixel (or region): +\begin{datatype} +typedef struct { + double scale; ///< The normalisation for an exposure, A(k) + double offset; ///< The time offset, dTk + double offref; ///< The reference time offset, dTo +} pmShutterCorrection; +\end{datatype} + +The allocator is: +\begin{prototype} +pmShutterCorrection *pmShutterCorrectionAlloc(); +\end{prototype} + + +The following function provides a guess for the shutter correction, +based on plot of counts vs exposure time. It should be used before +doing the full non-linear fit, to get parameters close to the true. +The function assumes the \code{exptime} vector is sorted (ascending +order; longest is last) prior to input. +\begin{prototype} +pmShutterCorrection *pmShutterCorrectionGuess(const psVector *exptime, ///< Exposure times for each exposure + const psVector *counts ///< Counts for each exposure + ); +\end{prototype} + +A full non-linear fit may then be performed to the counts as a +function of exposure time. The main purpose is to solve for the +reference time offset, so that future fits may be performed using +linear fitting with the reference time offset fixed. +\begin{prototype} +pmShutterCorrection *pmShutterCorrectionFullFit(const psVector *exptime, ///< Exposure times for each exposure + const psVector *counts, ///< Counts for each exposure + const psVector *cntError, ///< Error in the counts + const pmShutterCorrection *guess ///< Initial guess + ); +\end{prototype} + +With the reference time offset fixed to a known value, a (much faster) +linear fit can be performed at each pixel to generate the shutter +correction. This function performs the fit with iterative clipping, +if \code{nIter} $> 1$. +\begin{prototype} +pmShutterCorrection *pmShutterCorrectionLinFit(const psVector *exptime, ///< Exposure times for each exposure + const psVector *counts, ///< Counts for each exposure + const psVector *cntError, ///< Error in the counts + const psVector *mask, ///< Mask for each exposure + float offref, ///< Reference time offset + int nIter, ///< Number of iterations + float rej, ///< Rejection threshold (sigma) + psMaskType maskVal ///< Mask value + ); +\end{prototype} + +\subsubsection{Creating a shutter correction} + +The above steps are all performed using the following function. Given +an array of images with known exposure times, this function measures +the shutter correction (our principal concern is for the time offset, +rather than the normalisation) by measuring the reference time offset +using the full non-linear fit for a small number of representative +regions (middle and corners), and then using that to perform a linear +fit to each pixel. +\begin{prototype} +psImage *pmShutterCorrectionMeasure(const psVector *exptimes, ///< Exposure times + const psArray *images, ///< Input images + const psArray *weights, ///< Weight images + const psArray *masks, ///< Mask images + unsigned int size, ///< Size of samples for statistics for non-linear fit + psStatsOptions meanStat, ///< Statistic to use for mean + psStatsOptions stdevStat, ///< Statistic to use for stdev + int nIter, ///< Number of iterations + float rej, ///< Rejection threshold (sigma) + psMaskType maskVal ///< Mask value + ); +\end{prototype} + +\subsubsection{Applying a shutter correction} + +Given a shutter correction (a measurement of $\delta T$ for each +pixel), the following function applies this correction to an input +image. +\begin{prototype} +bool pmShutterCorrectionApply(pmReadout *readout, ///< Readout to which to apply shutter correction + const pmReadout *shutter ///< Shutter correction readout, with dT for each pixel + ); +\end{prototype} + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + \subsection{Flat-fielding} -Given an input image and a flat-field image, \code{pmFlatField} shall -divide the input image by the flat-field image and return it in place, -updating the mask contained within the input image as appropriate. -The API shall be the following: -\begin{prototype} -bool pmFlatField(pmReadout *in, const pmReadout *flat); -\end{prototype} - -Note that the input image, \code{in}, and the flat-field image, -\code{flat}, need not be the same size, since the input image may -already have been trimmed (following overscan subtraction), but the -function shall use the offsets of the readout (\code{in->col0, -in->row0}) and the image subarray (\code{in->image->x0, -in->image->y0}) to determine the appropriate offsets to obtain the -correct detector pixels in the flat-field image. Note that the image -offset is relative to its parent, so this offset must be followed to -the top level image which is not a child of another image and the -offsets summed. The detector pixel coordinates of pixel \code{x,y} in -a top-level image are thus \code{x + in->image->x0 + in->col0, y + -in->image->y0 + in->row0}. In the event that the \code{flat} image is -too small (i.e., pixels on the input image refer to pixels outside the -range of the \code{flat} image), the function shall generate an error. - -Pixels which are negative or zero in the \code{flat} shall be masked -in the input image with the value \code{PM_MASK_FLAT} (see -\S\ref{sec:maskValues}). Negative pixels in the \code{flat} may be -set to zero so that they are treated identically to zeroes. Any -pixels masked in the \code{flat} shall be masked with corresponding -values in the \code{output}. - -The function shall not normalize the \code{flat}; this responsibility -is left to the caller. This function is basically equivalent to a -divide (with \code{psImageOp}), but with care for the region that is -divided, checking for zero and negative pixels, and copying of the -mask from the \code{flat} to the output. - -The images in the input and flat-field readouts must both be of type -F32. - -This operation acts only on the region of the readout specified by -\code{CELL.TRIMSEC}. + +The following function applies the flat-field calibration to the input +readout. The relative offsets between the input and flat images is +determined from the readout row0,col0 and the CELL.X0 and CELL.Y0 +concepts. Normalisation of the flat is left as the responsibility of +the caller. Non-positive pixels in the flat are masked, if there is a +mask present in the input readout. +\begin{prototype} +bool pmFlatField(pmReadout *in, ///< Readout with input image + const pmReadout *flat ///< Readout with flat image + ); +\end{prototype} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -846,65 +914,301 @@ \subsection{Masking} -\subsubsection{Mask values} -\label{sec:maskValues} - -We define several mask values for use in the detrend processing: +The following function applies a mask readout to an input (science) readout. + +Pixels marked as bad (according to the supplied mask value) within the +mask are marked as bad within the input image's mask. If +\code{maskVal} is non-zero, all pixels in the mask have any of the +same bits sets as \code{maskVal} shall have the corresponding bits +raised. If \code{maskVal} is zero, any zero pixels in the mask are +OR-ed with \code{PM_MASK_BAD}. Position offsets (such as due to +trimming) between the input and mask are applied so that the same +pixels are referred to. The science readout must already have a +supplied mask element (use e.g. \code{pmReadoutSetMask}). +\begin{prototype} +bool pmMaskBadPixels(pmReadout *input, ///< Input science image + const pmReadout *mask, ///< Mask image to apply + psMaskType maskVal ///< Mask value to apply + ); +\end{prototype} + +\tbd{Masks for the charge traps need to be grown by the extent of the +OT convolution kernel. For other pixel types, orthogonal transfer of +the flux in this pixel will not (necessarily) affect the flux in +neighbouring pixels.} + + +\subsection{Fringes} + +Some images contain a signal caused by thin-film interference in the +device due to strong emission lines. The resulting instrumental +effect consists of a pattern (the ``fringe pattern'') of bright and +dark bands corresponding to the constructive and destructive +interference of the emission lines. In the case that a single +emission line causes the line structure, the resulting pattern can be +described by two independent parameters: First, the amplitude of the +emission line determines the overall amplitude of the pattern. +Second, the three-dimensional surface structure of the device +determines the shape of the pattern. In a typical situation, the +device is illuminated by multiple emission lines, as well as a +continuum spectral source, which contributes to the overall light +detected by the device without following the fringe pattern. The +relative intensities of the continuum background and the fringe +pattern depend on the device structure (thickness) and on the ratio of +the continuum and line emission fluxes. + +A simple approach to the fringe pattern is to subtract a master fringe +frame scaled by the amplitude of the fringe pattern. The amplitude of +the fringe pattern is used both in the process of constructing the +master image and in scaling the master image when it is applied to +science image. This is the method currently in use at CFHT and it +usually performs well. However, the fringe signal can vary as the +emission lines in the atmosphere change, and the above method breaks +down unless different fringe images corresponding to different +atmospheric conditions are constructed. + +An alternative technique is to use multiple master fringe images at +the same time, each representing different atmospheric conditions. +The observed fringe frame can be considered as a linear combination of +different fringe patterns, depending on the relative strengths of the +lines active in creating each of the fringe masters. It is not +critical that the fringe master images represent completely orthogonal +fringe patterns, they need only sample sufficiently different +conditions to provide a handle on the underlying fringe signals. + +We define a method of measuring the fringe pattern which is robust in +the presence of stars and which is fast. We implement a varient on +the method used at CFHT in which the fringe pattern is mapped by a +series of points distributed across the image. At CFHT, manual effort +is used to carefully define point pairs which correspond to peaks and +valleys of the fringe pattern. We implement a different approach in +which the fringe points are randomly chosen across the image. At each +point in the image, the median flux is measured in a box of specified +size. A low-frequency spatial filter is then applied to these +measurements. The resulting array of points and fluxes then +represents the strength of the fringe pattern on that image. The +comparison between any two fringe images is then just a linear fit +between these fringe statistics vectors, as follows: +\[ +S_i = C_0 + C_1 F_i +\] +where $S_i$ is the fringe statistic on the science image and $F_i$ is +the fringe statistic on the reference fringe image. Extending this +logic to any number of reference fringe images results in the +following relationship: +\[ +S_i = C_0 + \sum_j C_j F_j +\] + +In order to correct a single science image, the collection of fringe +statistics ($S_i$) are used to measure the coefficients $C_0$, $C_j$. +The linear combination of the reference fringe images is then used to +build a master image which is subtracted from the science image. The +following structures and functions implement the above concepts. + + +\subsubsection{Fringe regions} + +Fringes are measured within a box of size \code{dX,dY}. A large scale +smoothing is performed by subtracting the background within large +divisions of the image. The coordinates of the fringe points and the +mask are originally set to \code{NULL}, which means that they will be +generated when required. + \begin{datatype} -/** Mask values */ -typedef enum { - PM_MASK_TRAP = 0x0001, ///< The pixel is a charge trap - PM_MASK_BADCOL = 0x0002, ///< The pixel is a bad column - PM_MASK_SAT = 0x0004, ///< The pixel is saturated - PM_MASK_FLAT = 0x0008 ///< The pixel is non-positive in the flat-field -} pmMaskValue; +typedef struct { + psU32 nRequested; // Number of fringe points selected + psU32 nAccepted; // Number of fringe points not masked + psU32 dX; // Median box half-width + psU32 dY; // Median box half-height + psU32 nX; // Number of large-scale smoothing divisions in x (col) + psU32 nY; // Number of large-scale smoothing divisions in y (row) + psVector *x; // Fringe point coordinates (col), or NULL + psVector *y; // Fringe point coordinates (row), or NULL + psVector *mask; // Fringe point on/off mask, or NULL +} pmFringeRegions; \end{datatype} -Of these, masks for the charge traps need to be grown by the extent of -the OT convolution kernel. For other pixel types, orthogonal transfer -of the flux in this pixel will not (necessarily) affect the flux in -neighbouring pixels. - -\subsubsection{Bad pixels} - -Given an input image, \code{in}, a bad pixel \code{mask}, a -corresponding value in the bad pixel mask to mask in the input image, -\code{maskVal}, a saturation level, and a growing radius, -\code{pmMaskBadPixels} shall mask in the input image those -pixels in the bad pixel mask that match the value to mask. The API -shall be the following: -\begin{prototype} -pmReadout *pmMaskBadPixels(pmReadout *in, const pmReadout *mask, unsigned int maskVal, - float sat, unsigned int growVal, int grow); -\end{prototype} - -Note that the input image, \code{in}, is modified in-place. All -pixels in the \code{mask} which satisfy the \code{maskVal} shall have -their corresponding pixels masked in the input image, \code{in}. All -pixels which satisfy the \code{growVal} shall have their corresponding -pixels, along with all pixels within the \code{grow} radius masked. -Pixels which have flux greater than \code{sat} shall also be masked, -and grown by a single pixel (in addition to the \code{grow} done on -the \code{growVal}). - -\tbd{In the future, may change {\tt grow} to a convolution kernel}. - -Note that the input image, \code{in}, and the \code{mask} need not be -the same size, since the input image may already have been trimmed -(following overscan subtraction), but the function shall use the -offsets in the image (\code{in->x0} and \code{in->y0}) to determine -the appropriate offsets to obtain the correct pixel on the mask. In -the event that the \code{mask} image is too small (i.e., pixels on the -input image correspond to pixels outside the range of the \code{mask} -image), the function shall generate an error. - -The input image may be of type U16, S32 or F32. The mask image -must be of type U8. - -This operation acts only on the region of the readout specified by -\code{CELL.TRIMSEC}. +The allocator is: +\begin{prototype} +pmFringeRegions *pmFringeRegionsAlloc (int nPts, ///< Number of fringe points to create + int dX, ///< Half-width of fringe boxes + int dY, ///< Half-height of fringe boxes + int nX, ///< Smoothing scale in x + int nY ///< Smoothing scale in y + ); +\end{prototype} + +The following function generates fringe points randomly over an image. +No effort is made to avoid masked regions (indeed, the function knows +nothing about masks). If the random number generator is \code{NULL}, +then a new one will be used. +\begin{prototype} +bool pmFringeRegionsCreatePoints(pmFringeRegions *fringe, ///< Fringe regions to generate + const psImage *image, ///< Image for the regions (defines the size) + psRandom *random ///< Random number generator, or NULL + ); +\end{prototype} + +Fringe regions may be written to and read from FITS files with the following functions. + +\code{pmFringeRegionsWriteFits} writes fringe regions to a FITS file, +with the given extension name. The header is supplemented with scalar +values \code{dX}, \code{dY}, \code{nX} and \code{nY} (as +\code{PSFRNGDX}, \code{PSFRNGDY}, \code{PSFRNGNX}, \code{PSFRNGNY}) +from the fringe regions, while the fringe coordinates and mask are +written as a FITS table (as \code{x}, \code{y}, \code{mask}). + +\code{pmFringeRegionsReadFits} reads fringe regions from a FITS file, +at the given extension name. The scalars are retrieved from the +header, while the table provides the fringe coordinates and mask. + +\begin{prototype} +bool pmFringeRegionsWriteFits(psFits *fits, ///< Output FITS file + psMetadata *header, ///< Additional headers to write, or NULL + const pmFringeRegions *regions, ///< Regions to write + const char *extname ///< Extension name, or NULL + ); +pmFringeRegions *pmFringeRegionsReadFits(psMetadata *header, ///< Header to read, or NULL + const psFits *fits, ///< Input FITS file + const char *extname ///< Extension name, or NULL + ); +\end{prototype} + +\subsubsection{Fringe Statistics} + +Fringe measurements for a particular image are contained in the +following structure, which contains the median and standard deviation +for each of the fringe regions. +\begin{datatype} +typedef struct { + pmFringeRegions *regions; ///< Fringe regions + psVector *f; ///< Fringe point median + psVector *df; ///< Fringe point stdev +} pmFringeStats; +\end{datatype} + +The allocator is: +\begin{prototype} +pmFringeStats *pmFringeStatsAlloc(pmFringeRegions *regions // The fringe regions which will be measured + ); +\end{prototype} + +The following function measures the fringe statistics (median and +standard deviation) at each of the fringe regions for an image +(supplied within a readout). If the fringe regions are undefined, +they are generated. +\begin{prototype} +pmFringeStats *pmFringeStatsMeasure(pmFringeRegions *fringe, ///< Fringe regions at which to measure + const pmReadout *readout, ///< Readout for which to measure + psMaskType maskVal ///< Mask value for image + ); +\end{prototype} + +Like the fringe regions, the fringe statistics may be written to and +read from a FITS file, allowing rapid re-generation of the fringe +measurements for the calibration files. + +\code{pmFringeStatsWriteFits} writes the fringe statistics for an +image to a FITS table, with the given extension name. The median and +standard deviation measurements are written as a FITS table (as +\code{f} and \code{df}). + +\code{pmFringeStatsReadFits} reads the fringe stats from a FITS table, +at the given extension name. It is assumed that the fringe +measurements correspond to the regions provided. + +\begin{prototype} +bool pmFringeStatsWriteFits(psFits *fits, ///< FITS file to which to write + psMetadata *header, ///< Additional headers to write, or NULL + const pmFringeStats *fringe, ///< Fringe statistics to be written + const char *extname ///< Extension name for table + ); +pmFringeStats *pmFringeStatsReadFits(psMetadata *header, ///< Header to read, or NULL + const psFits *fits, ///< FITS file from which to read + const char *extname, ///< Extension name to read + pmFringeRegions *regions ///< Corresponding regions + ); +\end{prototype} + +Since each readout of each chip must be measured separately (so as to +avoid any gaps between the cells, as in the case for GPC), while the +fit must be performed with all the readouts belonging to a chip (in +order to get a secure measurement of the fringe amplitudes), we +provide the following function, which concatenates the fringe stats +for several readouts into a single fringe stats. The function +generates a new \code{pmFringeStats} from concatenating those in the +array. The corresponding \code{pmFringeRegions} is also generated. +\begin{prototype} +pmFringeStats *pmFringeStatsConcatenate(const psArray *fringes, ///< Array of pmFringeStats for the readouts + const psVector *x0, ///< Offset in x for the readout + const psVector *y0 ///< Offset in y for the readout + ); +\end{prototype} + + +\subsubsection{Fringe scales} + +The following structure contains the fringe solution --- the +coefficients of the linear system of library fringe images that best +fit the science image. + +\begin{datatype} +typedef struct { + int nFringeFrames; ///< Number of fringe frames + psVector *coeff; ///< Fringe coefficients; size = nFringeFrames + psVector *coeffErr; ///< Error in fringe coefficients; size = nFringeFrames +} pmFringeScale; +\end{datatype} + +The corresponding allocator is: +\begin{prototype} +pmFringeScale *pmFringeScaleAlloc(int nFringeFrames ///< Number of fringe frames + ); +\end{prototype} + +The following function measures the fringe scales for the library +fringe images, solving the linear system that best fits the fringe +measurements made on the science image. Given a fringe measurement +for a science image, and an array of template fringe measurements, the +function measures the contribution of each of the templates to the +input. Rejection is performed on the fringe regions, to weed out +stars etc. +\begin{prototype} +pmFringeScale *pmFringeScaleMeasure(pmFringeStats *science, ///< Fringe measurements from science image + psArray *fringes, ///< Array of fringe measurements from templates + float rej, ///< Rejection threshold (in standard deviations) + unsigned int nIter, ///< Maximum number of iterations + float keepFrac ///< Minimum fraction of regions to keep + ); +\end{prototype} + + +\subsubsection{Fringe correction} + +The following function solves for and applies the fringe correction. +It is a wrapper around each of the fringe correction components to +measure the fringe points, solve for the fringe correction, and apply +the fringe correction. The input fringe images are modified (scaled +by the solution coefficients in order to correct the science image). +The function returns the summed fringe image, and corrects the input +science image. + +\begin{prototype} +psImage *pmFringeCorrect(pmReadout *in, ///< Input science image + pmFringeRegions *fringes, ///< The fringe regions used + psArray *fringeImages, ///< Fringe template images to use in correction + psArray *fringeStats, ///< Fringe stats (for templates) to use in correction + psMaskType maskVal, ///< Value to mask for science image + float rej, ///< Rejection threshold, for pmFringeScaleMeasure + unsigned int nIter, ///< Maximum number of iterations, for pmFringeScaleMeasure + float keepFrac ///< Minimum fraction of regions to keep, for pmFringeScaleMeasure + ); +\end{prototype} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Subtract sky} -\tbd{This may be deferred.} +\tbd{This has been deferred.} Given an input image, a polynomial or spline specifying the order of a @@ -962,4 +1266,6 @@ \subsection{Paper Trail} + +\tbd{This is not implemented yet.} The elements of the focal plane hierarchy each contain an @@ -1098,67 +1404,13 @@ will have the same noise characteristics. -\begin{datatype} -typedef struct { - psStats *stats; // Statistics to use in combining pixels - unsigned int maskVal, // Mask pixels where mask & maskVal == 1 - float fracHigh; // Fraction of high pixels to throw - float fracLow; // Fraction of low pixels to throw - int nKeep; // Number of pixels to be sure to keep -} pmCombineParams; -\end{datatype} - -\begin{prototype} -psImage * -pmReadoutCombine(psImage *output, // Output image, or NULL - const psList *inputs, // List of input readouts - pmCombineParams *params, // Combination parameters - const psVector *zero, // Offsets to apply for each image - const psVector *scale, // Scales to apply for each image - bool applyZeroScale, // Are zero and scale for application, or only noise properties? - float gain, // Gain in e/ADU - float readnoise // Read noise in e - ); -\end{prototype} - -\code{pmReadoutCombine} combines input images pixel by pixel --- for -each pixel of the output image, a stack of contributing input pixels -is formed and combined. Several of its input parameters are lists or -vectors, and if these are not all of the same length (or \code{NULL}), -the module shall generate an error and return \code{NULL}. - -If the provided \code{output} is \code{NULL}, then the module shall -allocate a new image of sufficient size for the input images. If the -\code{output} image is non-\code{NULL} and is not of sufficient size -for the combined image, the module shall generate an error and return -\code{NULL}. - -If the \code{inputs} is \code{NULL}, the module shall generate an -error and return \code{NULL}. Otherwise, the \code{inputs} shall be a -list of \code{pmReadout}s. The images contained within the -\code{pmReadout}s need not all be of the same size, but the module -shall take into account the offsets (\code{col0,row0}) from the corner -of the detector when comparing pixels, so that it is the same -\textit{physical} pixels that are combined. - -The parameters used in the combination, including how the pixels are -to be combined, and how the rejection is performed is contained within -the \code{params}, which may not be \code{NULL} (otherwise the module -shall generate an error and return \code{NULL}). We choose to use -this structure instead of supplying the values separately in order to -keep down the number of parameters to \code{pmReadoutCombine}; the -\code{pmCombineParams} may be recycled for subsequent calls to -\code{pmReadoutCombine} since the values are not dependent upon the -choice of inputs, but merely specify how the combination is to be -performed. - -The particular statistic specified by \code{stats} shall be used to -combine each stack of pixels from the input images. Only one of the -statistics choices may be specified, otherwise the module shall -generate an error and return \code{NULL}. - -If the \code{maskVal} is non-zero, then pixels in the \code{mask} of -each \code{pmReadout} in the \code{inputs} which satisfy the -\code{maskVal} shall not have the corresponding pixels placed in the -stack for combination. +The following structure specifies how the combination of a stack of +pixels is performed. The particular statistic specified by +\code{combine} is used to combine each stack of pixels from the input +images. If the \code{combine} specifies clipped fitting, the +\code{iter} and \code{rej} members provide the number of iterations +and the rejection threshold for clipping. + +The \code{maskVal} specifies mask values for pixels to be +masked out of the combination. After masking, but before performing the combination, the highest @@ -1168,191 +1420,52 @@ rejection is performed. -If the \code{zero} vector is non-\code{NULL} and \code{applyZeroScale} -is \code{true}, then the appropriate values shall be added to the -\code{inputs} before rejection is performed. If \code{zero} is -non-\code{NULL} and \code{applyZeroScale} is false, then the values -shall only be used in calculating the Poisson variances. - -If the \code{scale} vector is non-\code{NULL} and -\code{applyZeroScale} is \code{true}, then the appropriate values -shall multiply the \code{inputs} before rejection is performed. If -\code{scale} is non-\code{NULL} and \code{applyZeroScale} is false, -then the values shall only be used in calculating the Poisson -variances. - -The purpose of \code{applyZeroScale} is to allow combination of fringe -frames, where the frames have been deliberately sky-subtracted and -rescaled (to get the fringes amplitudes running from -1 to 1), which -actions should not be undone when combining, but yet it is desirable -to provide the \code{zero} and \code{scale} values so that the correct -noise properties are used in the combination. - -If the \code{gain} and \code{readnoise} are positive and non-negative -(respectively), then these shall be used to provide weights for the -combination using Poisson statistics ($\sigma_i$ below). - -In summary, pixels corresponding to the same physical pixel are -combined, having values $x_i \pm \sigma_i$. In the case that -\code{applyZeroScale} is \code{true}, then: -\begin{eqnarray} -x_i & = & s_i f_i + z_i \\ -\sigma_i & = & [g x_i + r^2]^{1/2} / g -\end{eqnarray} -Where $f_i$ is the value of the pixel in image $i$, $s_i$ is the scale -applied to image $i$, $z_i$ is the zero offset applied to image $i$, -$g$ is the gain, and $r$ is the read noise. If scales are not -provided, they are set to unity; if zero offsets are not provided, -they are set to zero. - -If \code{applyZeroScale} is \code{false}, then the values are: -\begin{eqnarray} -x_i & = & f_i \\ -\sigma_i & = & [g (s_i f_i + z_i) + r^2]^{1/2} / g -\end{eqnarray} -where the same symbols are used as above. - -The \code{inputs, zero} and \code{scale} may be of U16, S32 and F32 -types, and must all be of the same type. The \code{output} shall be -of the same type. - -\subsection{Fringe Amplitude} - -Some images contain a signal caused by thin-film interference in the -device due to strong emission lines. The resulting instrumental -effect consists of a pattern (the ``fringe pattern'') of bright and -dark bands corresponding to the constructive and destructive -interference of the emission lines. In the case that a single -emission line causes the line structure, the resulting pattern can be -described by two independent parameters: First, the amplitude of the -emission line determines the overall amplitude of the pattern. -Second, the three-dimensional surface structure of the device -determines the shape of the pattern. In a typical situation, the -device is illuminated by multiple emission lines, as well as a -continuum spectral source, which contributes to the overall light -detected by the device without following the fringe pattern. The -relative intensities of the continuum background and the fringe -pattern depend on the device structure (thickness) and on the ratio of -the continuum and line emission fluxes. - -A simple approach to the fringe pattern is to subtract a master fringe -frame scaled by the amplitude of the fringe pattern. The amplitude of -the fringe pattern is used both in the process of constructing the -master image and in scaling the master image when it is applied to -science image. This is the method currently in use at CFHT and it -usually performs well. However, the fringe signal can vary as the -emission lines in the atmosphere change, and the above method breaks -down unless different fringe images corresponding to different -atmospheric conditions are constructed. - -An alternative technique is to use multiple master fringe images at -the same time, each representing different atmospheric conditions. -The observed fringe frame can be considered as a linear combination of -different fringe patterns, depending on the relative strengths of the -lines active in creating each of the fringe masters. It is not -critical that the fringe master images represent completely orthogonal -fringe patterns, they need only sample sufficiently different -conditions to provide a handle on the underlying fringe signals. - -We define a method of measuring the fringe pattern which is robust in -the presence of stars and which is fast. We implement a varient on -the method used at CFHT in which the fringe pattern is mapped by a -series of points distributed across the image. At CFHT, manual effort -is used to carefully define point pairs which correspond to peaks and -valleys of the fringe pattern. We implement a different approach in -which the fringe points are randomly chosen across the image. At each -point in the image, the median flux is measured in a box of specified -size. A low-frequency spatial filter is then applied to these -measurements. The resulting array of points and fluxes then -represents the strength of the fringe pattern on that image. The -comparison between any two fringe images is then just a linear fit -between these fringe statistics vectors, as follows: -\[ -S_i = C_0 + C_1 F_i -\] -where $S_i$ is the fringe statistic on the science image and $F_i$ is -the fringe statistic on the reference fringe image. Extending this -logic to any number of reference fringe images results in the -following relationship: -\[ -S_i = C_0 + \sum_j C_j F_j -\] - -In order to correct a single science image, the collection of fringe -statistics ($S_i$) are used to measure the coefficients $C_0$, $C_j$. -The linear combination of the reference fringe images is then used to -build a master image which is subtracted from the science image. The -following structures and functions implement the above concepts. - -The \code{pmFringeStats} structure represents the fringe statistics -for a given image. \begin{datatype} typedef struct { - psU32 nRequested; // number of fringe points selected - psU32 nAccepted; // number of fringe points not masked - psU32 dX; // median box half-width - psU32 dY; // median box half-height - psU32 nX; // large-scale smoothing in x (col) - psU32 nY; // large-scale smoothing in y (row) - psVector x; // fringe point coordinates (col) - psVector y; // fringe point coordinates (row) - psVector f; // fringe point median - psVector df; // fringe point stdev - psVector mask; // fringe point on/off mask -} pmFringeStats; + psStatsOptions combine; ///< Statistic to use when performing the combination + psMaskType maskVal; ///< Mask value + int nKeep; ///< Mimimum number of pixels to keep + float fracHigh; ///< Fraction of high pixels to immediately throw + float fracLow; ///< Fraction of low pixels to immediately throw + int iter; ///< Number of iterations for clipping (for CLIPPED_MEAN only) + float rej; ///< Rejection threshould for clipping (for CLIPPED_MEAN only) +} pmCombineParams; \end{datatype} -The \code{pmFringeStats} structure is allocated with the following -function: -\begin{prototype} -pmFringeStats *pmFringeStatsAlloc ( - int nPts, // number of points to create - int dX, // half-width of fringe boxes - int dY, // half-height of fringe boxes - int nX, // smoothing scale in x - int nY // smoothing scale in y -); -\end{prototype} - -A set of fringe points appropriate to the dimensions of a specific -image are created with the following function: -\begin{prototype} -bool pmFringeStatsCreatePoints (pmFringeStats *fringe, psImage *image); -\end{prototype} - -In general, \code{pmFringeStatsCreatePoints} should only be needed -when a new chip and filter are first use for analysis. Multiple -fringe images with the same chip and filter need to be examined with -the same fringe points in order for the statistical comparison to be -meaningful. The constructed fringe points should be saved and loaded -as a FITS table using the following function: -\begin{prototype} -bool pmFringeStatsWriteFits (psFits *fits, pmFringeStats *fringe); -bool pmFringeStatsReadFits (psFits *fits, pmFringeStats *fringe); -\end{prototype} - -In order to measure the fringe statistics for a given image, the -following function is defined: -\begin{prototype} -bool pmFringeStatsMeasure(pmFringeStats *fringe, pmReadout *readout) -\end{prototype} -This function measures the robust median at each of the fringe points -and saves the median values in \code{fringe->f} and the scatter in -\code{fringe->df}. - -Given the fringe statistics for a science image, and the fringe -statistics for a set of reference fringe images, the following -function can be used to measure the scaling coefficients of the -reference fringe frames which best fit the science image fringe -pattern: -\begin{prototype} -pmFringeScale *pmFringeScaleMeasure (pmFringeStats *science, psArray *fringes) -\end{prototype} - -Given a science image, a set of master fringe images, and a the set of -fringe statistics for the reference fringe images, the following -function can be used to correct the science image for the fringe pattern: -\begin{prototype} -psImage *pmFringeCorrect(psImage *out, psMetadata *info, psImage *science, psArray *fringeImage, psArray *fringeStats); -\end{prototype} +The allocator is: +\begin{prototype} +pmCombineParams *pmCombineParamsAlloc(psStatsOptions statsOptions ///< Statistic to use for combination + ); +\end{prototype} + +The combination is performed by the following function: +\begin{prototype} +bool pmReadoutCombine(pmReadout *output,///< Output readout; altered and returned + const psArray *inputs, ///< Array of input readouts (F32 image and weight, U8 mask) + const psVector *zero, ///< Zero corrections to subtract from input, or NULL + const psVector *scale, ///< Scale corrections to divide into input, or NULL + const pmCombineParams *params ///< Combination parameters + ); +\end{prototype} + +\code{pmReadoutCombine} combines input images pixel by pixel --- for +each pixel of the output image, a stack of contributing input pixels +is formed and combined, using the mask and weights if supplied in the +input readouts. + +The \code{output} readout is extended in size, if necessary, to +contain the input pixels. + +The \code{inputs} array contains the \code{pmReadout}s to be combined. +The images contained within the \code{pmReadout}s need not all be of +the same size, but the module will take into account the offsets from +the corner of the detector when comparing pixels, so that it is the +same \textit{physical} pixels that are combined. + +If the \code{zero} vector is non-\code{NULL} then the appropriate +values are subtracted from the \code{inputs} before rejection is +performed. If the \code{scale} vector is non-\code{NULL}, then the +appropriate values multiply the \code{inputs} before rejection is +performed. + \subsection{Flat-field Re-Normalization} @@ -1425,29 +1538,24 @@ to modify the flux levels on the master flat-field images. -We define the following function to perform the analysis discussed -above: -\begin{prototype} -bool pmFlatNormalization (psVector *sourceFlux, psVector *chipGains, psArray *fluxLevels); -\end{prototype} -The input array \code{fluxLevels} consists of $N_i$ vectors, one per -mosaic image. Each vector consists of $N_j$ elements, each a -measurement of the input flat-field image flux levels. All of these -vectors must be constructed with the same number of elements, or the -function will return an error. If a chip is missing from a particular -image, that element should be set to \code{NaN}. The vector -\code{chipGains} supplies initial guesses for the chip gains. If the -vector contains the values 0.0 or \code{NaN} for any of the elements, -the gain is set to the mean of the valid values. If the vector length -does not match the number of chips, an warning is raised, all chip -gain guesses will be set to 1.0, and the vector length modified to -match the number of chips defined by the supplied \code{fluxLevels}. -The \code{sourceFlux} input vector must be allocated (not -\code{NULL}), but the routine will set the vector length to the number -of source images regardless of the initial state of the vector. All -vectors used by this function must be of type \code{PS_DATA_F64}. +The following function solves this system. The matrix of background +measurements ($M_{i,j}$) contains the background for the flat fields +used in the combination, as a function of exposure (rows) and chip +(columns). The exposure fluxes and chip gains are modified upon +return with the solved values. The function returns true if the +solution converged. An initial guess for the chip gains might be +helpful, but is not necessary. +\begin{prototype} +bool pmFlatNormalize(psVector **expFluxesPtr, ///< Flux in each exposure, or NULL; modified + psVector **chipGainsPtr, ///< Initial guess of the chip gains or NULL; modified + const psImage *bgMatrix ///< Background measurements: rows are exposures, cols are chips + ); +\end{prototype} + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Objects on Images} + +\tbd{EAM to update, if necessary.} \subsection{Overview} @@ -2465,4 +2573,6 @@ \section{Image Combination} +\tbd{PAP to code and update.} + The image combination for \PS{} will employ an iterative approach, in order to identify cosmic rays. The first pass involves transforming @@ -2665,4 +2775,6 @@ \section{Image Subtraction} +\tbd{PAP to code and update.} + Image subtraction is arguably the best method of identifying faint variable sources in images with different point-spread functions. It @@ -3120,623 +3232,4 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\section{Example Camera Configuration Files} - -\tbd{Some of these don't exactly match the specifications of this -document yet, because they have been changed from the prototype, but -it is hoped that they will be useful. Questions are welcome.} - -\subsection{MegaCam Raw} - -\begin{verbatim} -# The raw MegaCam data comes off the telescope with each of the chips stored in extensions of a MEF file. - -# How to identify this type -RULE METADATA - TELESCOP STR CFHT 3.6m - DETECTOR STR MegaCam - EXTEND BOOL T - NEXTEND S32 72 -END - -# How to read this data -PHU STR FPA # The FITS file represents an entire FPA -EXTENSIONS STR CELL # The extensions represent cells - -# What's in the FITS file? -CONTENTS METADATA - # Extension name, chip name:type - amp00 STR ccd00:left - amp01 STR ccd00:right - amp02 STR ccd01:left - amp03 STR ccd01:right - amp04 STR ccd02:left - amp05 STR ccd02:right - amp06 STR ccd03:left - amp07 STR ccd03:right - amp08 STR ccd04:left - amp09 STR ccd04:right - amp10 STR ccd05:left - amp11 STR ccd05:right - amp12 STR ccd06:left - amp13 STR ccd06:right - amp14 STR ccd07:left - amp15 STR ccd07:right - amp16 STR ccd08:left - amp17 STR ccd08:right - amp18 STR ccd09:left - amp19 STR ccd09:right - amp20 STR ccd10:left - amp21 STR ccd10:right - amp22 STR ccd11:left - amp23 STR ccd11:right - amp24 STR ccd12:left - amp25 STR ccd12:right - amp26 STR ccd13:left - amp27 STR ccd13:right - amp28 STR ccd14:left - amp29 STR ccd14:right - amp30 STR ccd15:left - amp31 STR ccd15:right - amp32 STR ccd16:left - amp33 STR ccd16:right - amp34 STR ccd17:left - amp35 STR ccd17:right - amp36 STR ccd18:left - amp37 STR ccd18:right - amp38 STR ccd19:left - amp39 STR ccd19:right - amp40 STR ccd20:left - amp41 STR ccd20:right - amp42 STR ccd21:left - amp43 STR ccd21:right - amp44 STR ccd22:left - amp45 STR ccd22:right - amp46 STR ccd23:left - amp47 STR ccd23:right - amp48 STR ccd24:left - amp49 STR ccd24:right - amp50 STR ccd25:left - amp51 STR ccd25:right - amp52 STR ccd26:left - amp53 STR ccd26:right - amp54 STR ccd27:left - amp55 STR ccd27:right - amp56 STR ccd28:left - amp57 STR ccd28:right - amp58 STR ccd29:left - amp59 STR ccd29:right - amp60 STR ccd30:left - amp61 STR ccd30:right - amp62 STR ccd31:left - amp63 STR ccd31:right - amp64 STR ccd32:left - amp65 STR ccd32:right - amp66 STR ccd33:left - amp67 STR ccd33:right - amp68 STR ccd34:left - amp69 STR ccd34:right - amp70 STR ccd35:left - amp71 STR ccd35:right -END - -# Specify the cell data -CELLS METADATA - left METADATA # Left amplifier - CELL.BIASSEC STR HEADER:BIASSEC - CELL.TRIMSEC STR HEADER:DATASEC - CELL.XPARITY S32 1 # We could have specified this as a DEFAULT, but this works - END - right METADATA # Right amplifier - CELL.BIASSEC STR HEADER:BIASSEC - CELL.TRIMSEC STR HEADER:DATASEC - CELL.XPARITY S32 -1 # This cell is read out in the opposite direction - END -END - -# How to translate PS concepts into FITS headers -TRANSLATION METADATA - FPA.NAME STR EXPNUM - FPA.AIRMASS STR AIRMASS - FPA.FILTER STR FILTER - FPA.POSANGLE STR ROTANGLE - FPA.RA STR RA - FPA.DEC STR DEC - FPA.RADECSYS STR RADECSYS - FPA.MJD STR MJD-OBS - CELL.EXPOSURE STR EXPTIME - CELL.DARKTIME STR DARKTIME - CELL.XBIN STR CCDBIN1 - CELL.YBIN STR CCDBIN2 - CELL.GAIN STR GAIN - CELL.READNOISE STR RDNOISE - CELL.SATURATION STR SATURATE -END - -# Default PS concepts that may be specified by value -DEFAULTS METADATA - CELL.BAD S32 0 - CELL.YPARITY_DEPEND STR CHIP.NAME - CELL.YPARITY METADATA - ccd00 S32 -1 - ccd01 S32 -1 - ccd02 S32 -1 - ccd03 S32 -1 - ccd04 S32 -1 - ccd05 S32 -1 - ccd06 S32 -1 - ccd07 S32 -1 - ccd08 S32 -1 - ccd09 S32 -1 - ccd10 S32 -1 - ccd11 S32 -1 - ccd12 S32 -1 - ccd13 S32 -1 - ccd14 S32 -1 - ccd15 S32 -1 - ccd16 S32 -1 - ccd17 S32 -1 - ccd18 S32 1 - ccd19 S32 1 - ccd20 S32 1 - ccd21 S32 1 - ccd22 S32 1 - ccd23 S32 1 - ccd24 S32 1 - ccd25 S32 1 - ccd26 S32 1 - ccd27 S32 1 - ccd28 S32 1 - ccd29 S32 1 - ccd30 S32 1 - ccd31 S32 1 - ccd32 S32 1 - ccd33 S32 1 - ccd34 S32 1 - ccd35 S32 1 - END -END - -# How to translate PS concepts into database lookups -DATABASE METADATA - TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS -# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME -# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME - -# A database entry refers to a particular column (COLUMN) in a -# particular table (TABLE), given certain PS concepts (GIVENPS) that -# match certain database columns (GIVENDBCOL). - -END -\end{verbatim} - -\subsection{MegaCam Splice} - -\begin{verbatim} -# The spliced MecaCam data is stored in single extensions for each chip - -# How to recognise this type -RULE METADATA - TELESCOP STR CFHT 3.6m - DETECTOR STR MegaCam - EXTEND BOOL T - NEXTEND S32 36 -END - -# How to read this data -PHU STR FPA # The FITS file represents an entire FPA -EXTENSIONS STR CHIP # The extensions represent chips - -# What's in the FITS file? -CONTENTS METADATA - # Extension name, components - ccd00 STR left right - ccd01 STR left right - ccd02 STR left right - ccd03 STR left right - ccd04 STR left right - ccd05 STR left right - ccd06 STR left right - ccd07 STR left right - ccd08 STR left right - ccd09 STR left right - ccd10 STR left right - ccd11 STR left right - ccd12 STR left right - ccd13 STR left right - ccd14 STR left right - ccd15 STR left right - ccd16 STR left right - ccd17 STR left right - ccd18 STR left right - ccd19 STR left right - ccd20 STR left right - ccd21 STR left right - ccd22 STR left right - ccd23 STR left right - ccd24 STR left right - ccd25 STR left right - ccd26 STR left right - ccd27 STR left right - ccd28 STR left right - ccd29 STR left right - ccd30 STR left right - ccd31 STR left right - ccd32 STR left right - ccd33 STR left right - ccd34 STR left right - ccd35 STR left right -END - -# Specify the cells -CELLS METADATA - left METADATA - CELL.BIASSEC STR HEADER:BSECA - CELL.TRIMSEC STR HEADER:TSECA - END - - right METADATA - CELL.BIASSEC STR HEADER:BSECB - CELL.TRIMSEC STR HEADER:TSECB - END -END - -# How to translate PS concepts into FITS headers -TRANSLATION METADATA - FPA.NAME STR EXPNUM - FPA.AIRMASS STR AIRMASS - FPA.FILTER STR FILTER - FPA.POSANGLE STR ROTANGLE - FPA.RA STR RA - FPA.DEC STR DEC - FPA.RADECSYS STR RADECSYS - FPA.MJD STR MJD-OBS - CELL.EXPOSURE STR EXPTIME - CELL.DARKTIME STR DARKTIME - CELL.XBIN STR CCDBIN1 - CELL.YBIN STR CCDBIN2 - CELL.GAIN STR GAIN - CELL.READNOISE STR RDNOISE - CELL.SATURATION STR SATURATE -END - -# Default PS concepts that may be specified by value -DEFAULTS METADATA - CELL.BAD S32 0 - CELL.XPARITY S32 1 - CELL.YPARITY S32 1 -END - - -# How to translate PS concepts into database lookups -DATABASE METADATA - TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS -# CELL.GAIN dbEntry Camera gain chipId,cellId CHIP.NAME,CELL.NAME -# CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP.NAME,CELL.NAME - -# A database entry refers to a particular column (COLUMN) in a -# particular table (TABLE), given certain PS concepts (GIVENPS) that -# match certain database columns (GIVENDBCOL). - -END -\end{verbatim} - -\subsection{LRIS Blue} - -\begin{verbatim} -# The Low Resolution Imager and Spectrograph (LRIS) blue side - -# We have no choice but to hard-code the various regions, because Keck -# only stores them as: -# WINDOW = '1,0,0,2048,4096' -# PREPIX = 51 -# POSTPIX = 80 -# BINNING = '1,1 ' -# AMPPSIZE= '[1:1024,1:4096]' - -# I don't know how we would get the IPP to react to changes in the -# windowing on the fly --- we have no mechanism for setting the region -# sizes on the basis of the above keywords. Therefore, we hard-code -# the regions and assert on our assumptions in the RULE. - - -# How to identify this type -RULE METADATA - TELESCOP STR Keck I - INSTRUME STR LRISBLUE - AMPLIST STR 1,4,0,0 - WINDOW STR 1,0,0,2048,4096 - PREPIX S32 51 - POSTPIX S32 80 - BINNING STR 1,1 - AMPPSIZE STR [1:1024,1:4096] - NAXIS1 S32 4620 - NAXIS2 S32 4096 -END - -# How to read this data -PHU STR FPA # The FITS file represents an entire FPA -EXTENSIONS STR NONE # There are no extensions - -# What's in the FITS file? -CONTENTS METADATA - LeftChip STR amp1 amp2 - RightChip STR amp3 amp4 -END - -# Specify the cell data -CELLS METADATA - amp1 METADATA - CELL.BIASSEC STR VALUE:[1:51,1:4096];[4301:4380,1:4096] - CELL.TRIMSEC STR VALUE:[205:1228,1:4096] - CELL.GAIN STR VALUE:1.2 - CELL.READNOISE STR VALUE:5.6 - END - - amp2 METADATA - CELL.BIASSEC STR VALUE:[52:102,1:4096];[4381:4460,1:4096] - CELL.TRIMSEC STR VALUE:[1229:2252,1:4096] - CELL.GAIN STR VALUE:1.3 - CELL.READNOISE STR VALUE:6.7 - END - - amp3 METADATA - CELL.BIASSEC STR VALUE:[103:153,1:4096];[4461:4540,1:4096] - CELL.TRIMSEC STR VALUE:[2253:3276,1:4096] - CELL.GAIN STR VALUE:1.4 - CELL.READNOISE STR VALUE:7.8 - END - - amp4 METADATA - CELL.BIASSEC STR VALUE:[154:204,1:4096];[4541:4620,1:4096] - CELL.TRIMSEC STR VALUE:[3277:4300,1:4096] - CELL.GAIN STR VALUE:1.5 - CELL.READNOISE STR VALUE:8.9 - END -END - -# How to translate PS concepts into FITS headers -TRANSLATION METADATA - FPA.AIRMASS STR AIRMASS - FPA.FILTER STR BLUFILT - FPA.POSANGLE STR ROTPOSN - FPA.RA STR RA - FPA.DEC STR DEC - CELL.EXPOSURE STR EXPOSURE - CELL.DARKTIME STR EXPOSURE // No special darktime header; use exposure time - CELL.DATE STR DATE // NOTE: There are TWO keywords called "DATE" (creation, exp)! - CELL.TIME STR UT -END - -# Default PS concepts that may be specified by value -DEFAULTS METADATA - FPA.RADECSYS STR ICRS -END -\end{verbatim} - -\subsection{LRIS Red} - -\begin{verbatim} -# The Low Resolution Imager and Spectrograph (LRIS) red side - -# We have no choice but to hard-code the various regions, because Keck -# only stores them as: -# WINDOW = '0,0,0,2048,2048' -# PREPIX = 20 -# POSTPIX = 80 -# BINNING = '1,1 ' -# AMPPSIZE= '[1:1024,1:4096]' - -# I don't know how we would get the IPP to react to changes in the -# windowing on the fly --- we have no mechanism for setting the region -# sizes on the basis of the above keywords. Therefore, we hard-code -# the regions and assert on our assumptions in the RULE. - - -# How to identify this type -RULE METADATA - TELESCOP STR Keck I - INSTRUME STR LRIS - AMPLIST STR 2,1,0,0 - WINDOW STR 0,0,0,2048,2048 - PREPIX S32 20 - POSTPIX S32 80 - BINNING STR 1, 1 - CCDPSIZE STR [1:2048,1:2048] - NAXIS1 S32 2248 - NAXIS2 S32 2048 - IMTYPE STR TWOAMPTOP -END - -# How to read this data -PHU STR CHIP # The FITS file represents a single chip -EXTENSIONS STR NONE # There are no extensions - -# What's in the FITS file? -CONTENTS STR LeftSide RightSide - -# Specify the cell data -CELLS METADATA - LeftSide METADATA - CELL.BIASSEC STR VALUE:[1:20,1:2048];[2089:2168,1:2048] - CELL.TRIMSEC STR VALUE:[41:1064,1:2048] - CELL.GAIN STR VALUE:1.2 - CELL.READNOISE STR VALUE:5.6 - END - - RightSide METADATA - CELL.BIASSEC STR VALUE:[21:40,1:2048];[2169:2248,1:2048] - CELL.TRIMSEC STR VALUE:[1065:2088,1:2048] - CELL.GAIN STR VALUE:1.3 - CELL.READNOISE STR VALUE:6.5 - END -END - -# How to translate PS concepts into FITS headers -TRANSLATION METADATA - FPA.AIRMASS STR AIRMASS - FPA.FILTER STR FILTER - FPA.POSANGLE STR POSANG - FPA.RA STR OBJ-RA - FPA.DEC STR OBJ-DEC - CELL.EXPOSURE STR EXPTIME - CELL.DARKTIME STR DARKTIME - CELL.DATE STR DATE-OBS - CELL.TIME STR TIME-OBS -END - -# Default PS concepts that may be specified by value -DEFAULTS METADATA - FPA.RADECSYS STR ICRS -END -\end{verbatim} - -\subsection{GPC OTA} - -\begin{verbatim} -# The raw GPC data comes off the telescope with each of the chips stored in separate files - -# How to identify this type -RULE METADATA -# TELESCOP STR PS1 -# DETECTOR STR GPC1 - EXTEND BOOL T - NEXTEND S32 64 - NAMPS S32 64 -END - -# How to read this data -PHU STR CHIP # The FITS file represents a single chip -EXTENSIONS STR CELL # The extensions represent cells - -# What's in the FITS file? -CONTENTS METADATA - # Extension name, type - xy00 STR pitch10u - xy01 STR pitch10u - xy02 STR pitch10u - xy03 STR pitch10u - xy04 STR pitch10u - xy05 STR pitch10u - xy06 STR pitch10u - xy07 STR pitch10u - xy10 STR pitch10u - xy11 STR pitch10u - xy12 STR pitch10u - xy13 STR pitch10u - xy14 STR pitch10u - xy15 STR pitch10u - xy16 STR pitch10u - xy17 STR pitch10u - xy20 STR pitch10u - xy21 STR pitch10u - xy22 STR pitch10u - xy23 STR pitch10u - xy24 STR pitch10u - xy25 STR pitch10u - xy26 STR pitch10u - xy27 STR pitch10u - xy30 STR pitch10u - xy31 STR pitch10u - xy32 STR pitch10u - xy33 STR pitch10u - xy34 STR pitch10u - xy35 STR pitch10u - xy36 STR pitch10u - xy37 STR pitch10u - xy40 STR pitch10u - xy41 STR pitch10u - xy42 STR pitch10u - xy43 STR pitch10u - xy44 STR pitch10u - xy45 STR pitch10u - xy46 STR pitch10u - xy47 STR pitch10u - xy50 STR pitch10u - xy51 STR pitch10u - xy52 STR pitch10u - xy53 STR pitch10u - xy54 STR pitch10u - xy55 STR pitch10u - xy56 STR pitch10u - xy57 STR pitch10u - xy60 STR pitch10u - xy61 STR pitch10u - xy62 STR pitch10u - xy63 STR pitch10u - xy64 STR pitch10u - xy65 STR pitch10u - xy66 STR pitch10u - xy67 STR pitch10u - xy70 STR pitch10u - xy71 STR pitch10u - xy72 STR pitch10u - xy73 STR pitch10u - xy74 STR pitch10u - xy75 STR pitch10u - xy76 STR pitch10u - xy77 STR pitch10u -END - -# Specify the cell data -CELLS METADATA - pitch10u METADATA - CELL.BIASSEC STR VALUE:[575:606,1:594] - CELL.TRIMSEC STR VALUE:[1:574,1:594] - # CELL.BIASSEC STR HEADER:BIASSEC - # CELL.TRIMSEC STR HEADER:DATASEC - END - - # This is just in here for fun - pitch12u METADATA - CELL.BIASSEC STR VALUE:[1:10,1:512];[523:574,1:512] - CELL.TRIMSEC STR VALUE:[11:522,1:512] - # CELL.BIASSEC STR HEADER:BIASSEC - # CELL.TRIMSEC STR HEADER:TRIMSEC - END -END - - -# How to translate PS concepts into FITS headers -TRANSLATION METADATA - CELL.BIN STR CCDSUM - CELL.SATURATION STR SATURATE -END - -# Default PS concepts that may be specified by value -DEFAULTS METADATA - FPA.AIRMASS F32 0.0 - FPA.FILTER STR NONE - FPA.POSANGLE F32 0.0 - FPA.RA STR 0:0:0 - FPA.DEC STR 0:0:0 - FPA.RADECSYS STR ICRS - FPA.NAME S32 0 - FPA.MJD F32 12345.6789 - CELL.EXPOSURE F32 0.0 - CELL.DARKTIME F32 0.0 - CELL.GAIN F32 1.0 - CELL.READNOISE F32 0.0 - CELL.BAD S32 0 - CELL.BIN S32 1 - CELL.XPARITY S32 1 - CELL.YPARITY S32 1 -END - -# How to translate PS concepts into database lookups -DATABASE METADATA - TYPE dbEntry TABLE COLUMN GIVENDBCOL GIVENPS - CELL.GAIN dbEntry Camera gain chipId,cellId CHIP,CELL - CELL.READNOISE dbEntry Camera readNoise chipId,cellId CHIP,CELL - -# A database entry refers to a particular column (COLUMN) in a -# particular table (TABLE), given certain PS concepts (GIVENPS) that -# match certain database columns (GIVENDBCOL). - -END -\end{verbatim} - - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - \section{Revision Change Log} \input{ChangeLogSDRS.tex}