Context Navigation

← Previous Change
Next Change →

design

Timestamp:

Jun 2, 2004, 4:56:58 PM (22 years ago)

Author:

eugene

Message:

major modifications to split SRS into SRS + SCD
too numerous to detail: check the diff

File:

: 1 edited

trunk/doc/design/ippSRS.tex (modified) (29 diffs)

Legend:

: Unmodified
: Added
: Removed

trunk/doc/design/ippSRS.tex

-              r810
+              r837
 %%% $Id: ippSRS.tex,v 1.2 2004-05-29 00:56:14 eugene Exp $
+%%% $Id: ippSRS.tex,v 1.3 2004-06-03 02:56:58 eugene Exp $
 \documentclass[panstarrs]{panstarrs}
 …
 \label{req:system-capabilities}
 \tbd{distinguish data products in commissioning, during PA survey,
 after PA survey}
+\tbd{distinguish data products in commissioning, during AP survey,
+after AP survey}
 The IPP must perform the following tasks:
 …
  portions of the IPP.
 \item {\bf Photometry \& Astrometry Database (PnA):} This component is
+\item {\bf Astrometry \& Photometry Database (AP):} This component is
   required to store and manipulate astronomical objects detected in
   various images, as identified above, including individual
 …
 MB/sec.
+\subsubsection{PA Database}
+\subsubsection{AP Database}
 \begin{table}
 \begin{center}
 \caption{PA Detection Classes \& Object Parameters\label{PAdetections}}
+\caption{AP Detection Classes \& Object Parameters\label{APdetections}}
 \begin{tabular}{lrrrr}
 \hline
 …
 \end{table}
 The PA Database must accept and store individual detections and
+The AP Database must accept and store individual detections and
 collections of detections along with information about the image which
 provided the detections.
 Detections must be saved as one of several detection classes (P2, P4S,
 P4D, SS) and the PA Database must store the appropriate parameters,
 listed in Table~\ref{PAdetections}, for each class.
 The PA Database must identify the image which provided the detection,
+P4D, SS) and the AP Database must store the appropriate parameters,
+listed in Table~\ref{APdetections}, for each class.
+The AP Database must identify the image which provided the detection,
 or in the case of external references, an identifier specific to the
 reference source.
 The PA Database must group detections into objects and measure average
+The AP Database must group detections into objects and measure average
 parameters of those objects.
 The PA Database must store parallax and proper motion parameters for a
+The AP Database must store parallax and proper motion parameters for a
 subset of the average objects.
 The PA Database must store image and filter calibration information
+The AP Database must store image and filter calibration information
 necessary to convert between instrumental magnitudes and calibrated
 magnitudes in standard systems.
 The PA Database must perform at least the follow queries, with
+The AP Database must perform at least the follow queries, with
 constraints on the output based on at least time ranges, magnitude
 limits, error limits:
 \begin{enumerate}
 \item given (RA,DEC) and a Radius, return all objects and/or
+\item given $(RA,DEC)$ and a Radius, return all objects and/or
 detections in the region.
 \item given (RA,DEC)_0 - (RA,DEC)_1, return all objects and/or
+\item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all objects and/or
   detections in the region.
 \item given (RA,DEC), return closest object.
+\item given $(RA,DEC)$, return closest object.
 \item given object ID, return all detections
 …
 \item given detection, return source image data.
 \item given (RA,DEC), return all images overlapping coordinate.
 \item given (RA,DEC) and a Radius, return all images overlapping region.
 \item given (RA,DEC)_0 - (RA,DEC)_1, return all images overlapping
+\item given $(RA,DEC)$, return all images overlapping coordinate.
+\item given $(RA,DEC)$ and a Radius, return all images overlapping region.
+\item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping
   region.
 …
 \item given a region, return all possible combinations of the object
   or detection magnitudes (M1 - M2).
 \item given a list of (RA,DEC) entries, return all nearest objects.
+  or detection magnitudes $(M1 - M2)$.
+\item given a list of $(RA,DEC)$ entries, return all nearest objects.
 \item given a filter, telescope, or detector, return all calibration
 …
 \end{enumerate}
 The PA Database must accept detection IDs of moving objects and label
+The AP Database must accept detection IDs of moving objects and label
 the detections with the identified object.
 \begin{table}
 \begin{center}
 \caption{PA Detection Classes \& Object Parameters\label{PAdetections}}
+\caption{AP Detection Classes \& Object Parameters\label{APdetections}}
 \begin{tabular}{lrrrr}
 \hline
 …
 \end{table}
 The PA Database must accept new detections at the rate generated by
+The AP Database must accept new detections at the rate generated by
 the telescope from the Phase 2 and Phase 4 analysis.  Except within 10
 degrees of the galactic plane, the PA Database must keep up with the
 incoming rates.  The expected rates are listed in Table~\ref{PArates},
+degrees of the galactic plane, the AP Database must keep up with the
+incoming rates.  The expected rates are listed in Table~\ref{APrates},
 along with the total data volume required for storage space over the
 PS-1 lifetime.  The PA Database must be able to keep up with these
+PS-1 lifetime.  The AP Database must be able to keep up with these
 rates.
 \subsubsection{Metadata Database}
+\subsubsection{Metadata Database -- FIX ME}
 \tbd{this section needs to be reviewed and revised}
 …
   location of images is in the Image server}
 \paragraph{Configuration Database}
+\paragraph{Configuration Database -- FIX ME}
 The IPP requires a Configuration Database to store and provide access to
 …
 \paragraph{Flat-field correction}
+The object image (after bias correction and non-linearity correction)
+must be corrected for sensitivity variations as a function of
+position, dividing by a flat-field image.
+The Phase 2 analysis must divide by the provided flat-field image.
+The division must handle zero-valued pixels in the flat-field image
+without raising floating point exceptions.
 The flat-field images must be appropriately normalized (see section
+\ref{mkcal}).  The flat-fielded image must have a consistent
+photometric zero-point across the chip, and across the full FPA, to
+within 0.2\% with peak-to-peak deviations of \tbr{0.5\%}.
+\ref{mkcal}).
+The flat-fielded image must have a consistent photometric zero-point
+across the chip, and across the full FPA, to within 0.2\% with
+peak-to-peak deviations of \tbr{0.5\%}.
 \paragraph{Sky \& Fringe subtraction}
+The flux contribution of the sky (from both continuum emission and the
+line emission that causes fringing) must be subtracted from the
+flat-fielded object image.  The subtraction must remove background
+(technically, foreground) variations which are not celestial but
+generated in the atmosphere or by more localized scattering.  This
+background subtraction does not address the artifacts generated by
+bright stars: bleeding columns, ghosts, or other localized reflection
+effects.  The background subtraction must remove the variations with
+an accuracy such that the residual variations do not introduce, on
+average, more than \tbd{0.2\%} photometric scatter or more than
+\tbd{1\%} extremely deviant outlier stars (stars for which the
+photometry is in error by more than 3\%).  \tbd{what is the
+requirement on galaxy photometry? morphology determinations?}
+\tbd{What is allowed power-spectrum of background variations?}
+The Phase 2 analysis must subtract the sky (and fringe where needed)
+contributions from the images.
+The residual after the background subtraction must have an average
+offset of 0 counts, excluding the signal from astronomical features.
+The background residuals must have peak-to-peak variations of less
+than \tbr{1\%} of the input background amplitude.
+The background residuals must have a scatter of less than \tbr{1\%} of
+the input background amplitude for apertures of less than
+\tbr{10~arcsec}.\comment{derived from the need for systematic errors
+of better than 0.5\% and known background ranges.}
 \paragraph{Identify `cosmic rays'}
+Charged particles in the detector frequently cause features which do
+not have the morphology of astronomical objects.  In a well-sampled
+image, these may be easily identified by the sharpness of the image.
 In a near critically-sampled image, these `cosmic rays' may be
+indistinguishable from stellar objects.  The IPP must have the
+capability of making the morphological identification of cosmic rays
+if the imaging data is suitable.  The identified cosmic rays must be
+masked with a configurable growth factor (additional pixels beyond the
+detected pixels in the feature).  \tbd{The determination if the image
+can be treated with morphological cosmic ray rejection must be made by
+Phase~2.}
+The Phase 2 analysis must detect cosmic rays in single images which
+are brighter than a user-configurable threshold.
+The Phase 2 analysis must mask detected cosmic rays with a unique
+bit value in the mask.
+The Phase 2 analysis must extend the masked region be a
+user-configurable growth factor.
+The Phase 2 analysis must perform the cosmic ray detection only if it
+is required by the analysis recipe.
 \paragraph{Find objects in the image}
+Objects on the flat-fielded object image must be found, and general
+parameters, including the object centroid, instrumental magnitude,
+local background level, and basic shape parameters ($\sigma_{\rm min},
+\sigma_{maj}$) measured.  The detection threshold must be
+configurable, and be a function of the average background flux or the
+image noise map.  Minimal object classification must be performed to
+distinguish objects which are consistent with a single PSF, objects
+which are inconsistent, and objects which are saturated.  The
+resulting collection of detected objects must be saved along with the
+relevant image metadata (\ie filter, exposure time, etc).
+The Phase 2 analysis must perform object detection on the processed
+images.
+The object detection must detect all objects above a user-configured
+threshold. \tbd{valid range for the threshold?}  The detection
+threshold must be a function of the average background flux or the
+image noise map.
+The object detection must measure the following object parameters:
+\begin{enumerate}
+\item object centroid and position errors
+\item an extended object position ($x_g, y_g$)
+\item instrumental PSF magnitude and error
+\item local background level and error
+\item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their
+  covarience matrix
+\end{enumerate}
+Minimal object classification must be performed to distinguish objects
+which are consistent with a single PSF, objects which are
+inconsistent, and objects which are saturated.
+The resulting collection of detected objects must be saved along with
+the relevant image metadata (\ie filter, exposure time, etc).
 \paragraph{Astrometry}
+Objects detected in Phase~2 must be matched with known astrometric
+reference objects, using reference object coordinates which have been
+adjusted for proper motion.  The matched objects must be used to
+improve the astrometric solutions for the individual OTAs.  At this
+stage, a user-defined collection of OTA astrometry parameters must be
+fitted on the basis of the matched stars.  The Cell astrometric
+parameters must not be allowed to vary at this stage.  The fit must be
+robust, rejecting outlier matches (either stars with poorly determined
+proper motion or spurious matches).  The resulting astrometric
+solution must be consistent across the OTA field to within \tbd{0.2
+arcsec}.
+The Phase 2 analysis must match the detected objects with known
+astrometric reference objects.
+The astrometric reference object coordinates must be adjusted for
+proper motion.
+The reference and detected object coordinates must be fit to determine
+astrometric parameters for the individual OTAs.
+The OTA astrometric parameters must include Chebychev polynomials of the
+coordinates up to 3rd order.
+The fitted number of polynomial orders must be a user-configured
+parameter.
+The Cell astrometric parameters must not be allowed to vary in the
+fit.
+The fit must be robust, rejecting outlier matches (either stars with
+poorly determined proper motion or spurious matches).
+The resulting astrometric solution must be consistent across the OTA
+field to within \tbd{0.2 arcsec}.
 \paragraph{Postage Stamps}
 The IPP must have the capability of extracting regions surrounding a
 specified subset of objects from the flattened images.  These postage
+stamp images must be saved for additional use by client science
+pipelines.  The identification of these objects must be on the basis
 of a set of rules applied to the object magnitude and position.
+The Phase 2 analysis must extract subrasters (`postage stamps')
+surrounding a user-specified list of coordinates from the flattened
+images.
+The postage stamp images must be saved in the IPP Image Server.
 \subsubsection{Phase 3 : exposure analysis}
 The Phase 3 analysis stage works with the results from a complete FPA
+obtained during Phase 2 to improve the photometric and astrometric
+calibrations.
+Phase 3 must use the objects detected in Phase 2, matched with an
+appropriate reference catalog, to determine the image photometric zero
+point and zero-point variations across the field.  If zero-point
+variations are significant \tbd{level TBD}, the zero-point variations
+must be modeled with a chebychev polynomial correction of order 3 or
+less.  The complete FPA image must be categorized as photometric or
+not \tbd{numerical scale?} on the basis of the zero-point consistency,
 the transparency compared with recent long-term measurements in the
+The Phase 3 analysis must use the objects detected in Phase 2, matched
+with a user-specified reference photometry catalog, to determine the
+image photometric zero point and zero-point variations across the
+field.
+If zero-point variations are significant \tbd{level TBD}, the
+zero-point variations must be modeled with a chebychev polynomial
+correction of order 3 or less.
+The photometric nature of the FPA image must be categorized
+\tbd{numerical scale?} on the basis of the zero-point consistency, the
+transparency compared with recent long-term measurements in the
 filter, and the external indicators of photometricity.
+Phase 3 must use the objects detected in Phase 2, matched with an
+appropriate reference catalog, to determine improvements to the
+astrometric solutions.  The distortion model appropriate to this image
+must be determined.  The resulting astrometric accuracy must be
+limited by the astrometric reference catalog \tbd{30 mas for USNO?}
+The Phase 3 analysis must use the objects detected in Phase 2, matched
+with an appropriate reference catalog, to improve the distortion model
+used for this image.
+The resulting astrometric accuracy must be limited by the astrometric
+reference catalog \tbd{30 mas for USNO?}
 \subsubsection{Phase 4 : image combination}
 …
 Phase 4 is the image combination stage, in which multiple images of
 the same portion of the sky are merged and confronted with the static
+sky image.  Phase 4 operates on the smallest data unit of the static
+sky, the sky cell, along with the associated pixels from a collection
+of images which have been processed through phases 1--3.  For each sky
+cell, the corresponding pixels are extracted from the exposures being
+processed and mapped to the projection of the sky cell. The pixels
+from the multiple input processed images are combined into a single,
+cleaned image.  This image is then confronted with the static sky cell
+data to produce a difference image.  Residual objects in the
+difference image, above a threshold are detected and excised from the
+original cleaned image.  The remaining pixels are added to the
+existing static sky image.  Object detection must be performed on the
+difference and cleaned images.  \tbd{when is static sky object
+detection \& classification performed?}  Phase 4 naturally divides
+into several stages, each of which are discussed in detail below.
+sky image.  Requirements for the different steps of the process are
+given below.
 \paragraph{Extract image pixels}
+For the given sky cell, the corresponding set of image pixels must be
+determined and extracted from the input images.  This process must use
+the astrometric information for each OTA and Cell to determine the
+exact overlaps.  It must not miss any pixels, and it must read no more
+than 20\% more pixels than necessary from the input images.
+The Phase 4 analysis must determine the corresponding set of image
+pixels for a given sky cell.
+The corresponding image pixels must be extracted from the input
+images, using the astrometric information for each OTA and Cell to
+determine the exact overlaps.
+The Phase 4 analysis must not miss any pixels in this match, and it
+must read no more than 20\% more pixels than necessary from the input
+images.
+The Phase 4 analysis must skip any sky cells with fewer than 5\% of
+their pixels overlapping the input images.
 \paragraph{Transform pixel coordinates}
 Pixels which have been extracted from the input images must be mapped
+to the corresponding pixels in the sky image.  The tranformation must
+be based on the measured astrometric solution for the input images
+relative to the reference catalog used to generate the static sky
+image.  This warping must use a locally linear astrometric solution to
+minimize computational effort. The output image must maintain be
+photometric consistent with the input image to within 0.2\%.
+\tbd{interpolation method?}
+to the corresponding pixels in the sky image.
+The tranformation must be based on the measured astrometric solution
+for the input images relative to the reference catalog used to
+generate the static sky image.
+This warping must use a locally-linear astrometric solution.
+The output image must maintain photometric consistency with the input
+image to within 0.2\%.  \tbd{interpolation method?}
 \paragraph{Flux matching}
+The multiple input images must have their object fluxes intercompared
+using the stars measured in Phase 2 in order to determine the
+appropriate photometry scaling factors needed to properly combine them
+photometrically.
+The Phase 4 analysis must determine appropriate photometry scaling
+factors needed to combine the images photometrically.
 \paragraph{Image outlier pixel rejection}
+Pixels from the group of images which are inconsistent with the
+ensemble of pixel values must be identified and flagged.  The
+resulting collection of pixels must be used to construct a single
+output image, cleaned of the outliers.  This outlier rejection must be
+performed optionally since moving objects will be rejected in images
+obtained over a wide range of times.
+Pixels from the group of images which are inconsistent \tbd{how much?}
+with the ensemble of pixel values must be identified and flagged.
+This outlier rejection must be performed optionally.
+\paragraph{Initial cleaned image}
+The resulting collection of pixels must be used to construct a single
+output image, cleaned of the outliers.
 \paragraph{PSF matching}
+The multiple input images must have their PSF mutually matched to
+allow for proper image subtraction.
+The cleaned, combined image must be PSF matched with the static sky image.
 \paragraph{Image Subtraction}
 The static sky image must be subtracted from the stacked, cleaned
+image.  All objects in the difference image must be detected and the
+pixels belonging to variable sources flagged in the input image.
+Object detection at this stage is the same as that used for Phase 2.
+image.
+\paragraph{Find objects in the image}
+The Phase 4 analysis must perform object detection on the difference
+images.
+All objects in the difference image must be detected and the pixels
+belonging to variable sources flagged in the input image.
+The object detection must detect all objects above a user-configured
+threshold. \tbd{valid range for the threshold?}  The detection
+threshold must be a function of the average background flux or the
+image noise map.
+The object detection must measure the following object parameters:
+\begin{enumerate}
+\item object centroid and position errors
+\item instrumental PSF magnitude and error
+\item local background level and error
+\item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$
+\item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covarience matrix
+\end{enumerate}
+Minimal object classification must be performed to distinguish objects
+which are consistent with a single PSF, objects which are
+inconsistent, and objects which are saturated.
+The resulting collection of detected objects must be saved along with
+the relevant image metadata (\ie filter, exposure time, etc).
 \paragraph{Cleaned Input Image}
+The flagged pixels must be excluded from the input images and a new,
+cleaned image constructed.  This image must have object detection
+applied to it.  \tbd{parameters}
+The pixels flagged as being from the difference image sources must be
+masked in the input images.
+A new, cleaned image must be constructed from the masked input images.
+\paragraph{Find objects in the image}
+The Phase 4 analysis must perform object detection on the cleaned,
+summed image.
+The object detection must detect all objects above a user-configured
+threshold. \tbd{valid range for the threshold?}  The detection
+threshold must be a function of the average background flux or the
+image noise map.
+The object detection must measure the following object parameters:
+\begin{enumerate}
+\item object centroid and position errors
+\item an extended object position ($x_g, y_g$)
+\item instrumental PSF magnitude and error
+\item local background level and error
+\item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their
+  covarience matrix
+\item the Petrosian radius, magnitude, axis ratio, and angle
+\item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$.
+\end{enumerate}
+Minimal object classification must be performed to distinguish objects
+which are consistent with a single PSF, objects which are
+inconsistent, and objects which are saturated.
+The resulting collection of detected objects must be saved along with
+the relevant image metadata (\ie filter, exposure time, etc).
 \paragraph{Update static sky}
 …
 The final, cleaned input image must be added to the static sky so that
 an incrementally-deeper static sky image may be made.
 \tbd{parameters, weight map}
-\paragraph{Products}
-Phase 4 must produce the following data products at a minimum:
-\begin{enumerate}
-\item Subtracted image --- the combined image using each of the
-telescopes, with the static sky subtracted;
-\item New static sky image --- the combined image using each of the
-telescopes, with the (old) static sky added;
-\item Metadata about the quality of each of these images; and
-\item A catalog of variable sources.
-\item A catalog of sources from the combined image.
-\end{enumerate}
 \paragraph{Timing}
 …
 \paragraph{Robustness}
+\tbd{what are the corresponding requirements?}
 It is essential that the static sky image (which may have been
 painstakingly accumulated over many months) not be corrupted by adding
 …
 \label{mkcal}
+The Calibration analysis stages may be performed on whatever
+timescales are appropriate and necessary to maintain the quality and
+relevance of the calibration images.  We distinguish two major classes
+of calibration images which require significantly different techniques
+for their construction.  We list the specific calibration images which
+must be constructed in the calibration analysis stages. The
+requirements for each of these stages are discussed in more detail
+below.
+\subsubsection{Basic Calibration Stages}
+The IPP must generate basic calibration images using the raw bias,
+dark, and flat-field (dome or twilight) images obtained by the
+telescope as the input.  The analysis of these images requires
+relatively simple stacking of the input set of images.  Outlier
+rejection, both of complete input images as well as pixels within the
+input stack, must be performed.  In addition, each type of image
+requires an appropriate normalization which may depend on the data
+levels in other detectors in the input set.  Each of these calibration
+stages must be able to determine from the input stack if the relevant
+calibration image needs to be updated and perform an initial test to
+see which input images are consistent and valid.
+The Calibration analysis stages must construct the various types of
+calibration frames needed by the IPP.  The requirements for each of
+these stages are discussed in detail below.
 \paragraph{bias images}
+Bias images may be needed to correct for structure in the bias.  The
+IPP must have the capability of constructing a master bias image from
+a stack of raw bias frames.  The input bias images, representing
+offsets from the overscan level, must have the overscan removed,
+including 1D structure if needed.  The bias construction must
+incorporate outlier image and outlier pixel rejection.  The statistic
+used to determine pixel values must optionally be derived from the
+sample mean, median, and mode, robust mean, median, and mode, and the
+clipped mean and median.  Residual images, in which the master bias is
+applied to the input images must be constructed and their statistics
+used to exclude any significant outlier input images.
+The \code{bias} calibration stage must construct a master bias image
+from a collection of raw bias images.
+The \code{bias} calibration stage must correct the input images based
+on the overscan region.
+The \code{bias} calibration stage must combine the input images using
+the statistic specified by the user, selected from one of the
+following: sample mean, median, and mode, robust mean, median, and
+mode, and the clipped mean and median.
+The \code{bias} calibration stage must construct residual images, in
+which the master bias is applied to the input images.
 \paragraph{dark images}
+Dark images may be needed to correct for structure in the dark
+current.  The IPP must have the capability of constructing a master
+dark image from a stack of raw dark frames.  The input dark images
+must first be corrected for the bias using whatever method is
+appropriate for the science images.  The master dark frame must be
+specified for a particular exposure time.  As such, the input dark
+frames must have a limited range of exposure times.  The dark frame
+construction must incorporate outlier image and outlier pixel
+rejection.  The statistic used to determine pixel values must
+optionally be derived from the sample mean, median, and mode, robust
+mean, median, and mode, and the clipped mean and median.  Residual
+images, in which the master dark image is applied to the input images
 must be constructed and their statistics used to exclude any
+significant outlier input images.  \tbd{The dark frames must be
+examined to determine the non-linearity of the measured dark current
 -- by what component?}.
+The \code{dark} calibration stage must construct a master dark image
+from a collection of raw dark images.
+The \code{dark} calibration stage must raise an error if the input
+images have exposure time which deviate by more than \tbr{2\%}.
+The \code{dark} calibration stage must correct the input dark images
+for the bias.
+The \code{dark} calibration stage must combine the input images using
+the statistic specified by the user, selected from one of the
+following: sample mean, median, and mode, robust mean, median, and
+mode, and the clipped mean and median.
+The \code{dark} calibration stage must construct residual images, in
+which the master dark is applied to the input images.
 \paragraph{flat-field images}
+Master flat-field images must be constructed from a collection of
+input flat-field images.  An appropriate set of input images must be
+selected on the basis of their flux levels, time of observations, and
+the observing conditions.  The input flat-field images must be
+processed (bias and dark corrected if needed) and the resulting images
+stacked.  The master flat-field construction must incorporate image
+and pixel outlier rejection.  The statistic used to determine pixel
+values must optionally be derived from the sample mean, median, and
+mode, robust mean, median, and mode, and the clipped mean and median.
+Residual images, in which the master flat-field image is applied to
+the input images must be constructed and their statistics used to
+exclude any significant outlier input images.
+\subsubsection{Other Calibration Stages}
+The \code{flat-field} calibration stage must construct a master
+flat-field image from a collection of raw flat-field images.
+The \code{flat-field} calibration stage must accept a group of images
+from one of the following flat-field sources: dome, twilight,
+night-sky.
+The \code{flat-field} calibration stage must raise an error if the
+input images in a single stack used more than one of the above
+flat-field sources or multiple filters.
+The \code{flat-field} calibration stage must correct the input
+flat-field images for the bias and dark.
+The \code{flat-field} calibration stage must combine the input images
+using the statistic specified by the user, selected from one of the
+following: sample mean, median, and mode, robust mean, median, and
+mode, and the clipped mean and median.
+The \code{flat-field} calibration stage must construct residual
+images, in which the master flat-field is applied to the input images.
 \paragraph{mask images}
+Initial bad-pixel mask images must be generated on the basis of
+comparison between raw flat-field images and a cleaned, stacked
+master.  The mask creation analysis stage must accept a collection of
+flat-field images and identify pixels which are repeatedly
+inconsistent from image to image.  If too many pixels are
+inconsistent, an error must be raised.
+The \code{mask} calibration stage must construct a bad-pixel mask from
+a stack of raw flat-field images and a master flat-field image.
+The \code{mask} calibration stage must mask the pixels which are
+inconsistent in the input flats by more than \tbr{1\%}, given
+sufficient signal-to-noise in the input flats.
+The \code{mask} calibration stage must mask the pixels which are
+consistently low or high in the input flats by more than a factor of
+\tbr{3} beyond the typical pixel.
+The \code{mask} calibration stage must mask the pixels identified in a
+table of bad pixels generated externally to the calibration stage.
+The \code{mask} calibration stage must use multiple bit values to
+identify the different types of masked pixels.
+The \code{mask} calibration stage must raise an error if the input
+images generate too many bad pixels in the mask.
 \paragraph{fringe frames}
+Fringe-correction frames must be generated to remove the fringe
+pattern caused by thin-film interference in the top layers of CCDs,
+particularly in the redder passbands.  Fringe correction frames must
+be constructed on the basis of observations of the night-sky in the
+appropriate filters.  The images must first be flattened to remove the
+pixel-to-pixel sensitivity variations of the detector.  The
+combination of multiple input fringe frames may not be simply stacked
+since the amplitude of the fringe pattern varies independently of
+other variations in the image.  The amplitude of the fringe frames
+must be measured and the images scaled to normalize the fringe
+amplitude to the range -1 to +1 before combining with one of the
+standard combination statistics (mean, median, mode, etc).
+The \code{fringe} calibration stage must construct a master fringe
+frame from a stack of raw night-time sky images or from a stack of
+dome fringe frames.
+The \code{fringe} calibration stage must raise an error if the input
+stack consists is images generated with more than one type of fringe
+source or with multiple filters.
+The \code{fringe} calibration stage must flatten the input images
+to remove the pixel-to-pixel sensitivity variations of the detector.
+The \code{fringe} calibration stage must measure the fringe amplitude
+on the input fringe images.
+The \code{fringe} calibration stage must scale the input fringe images
+based on the fringe amplitude.
+The \code{fringe} calibration stage must combine the scaled input
+images using the statistic specified by the user, selected from one of
+the following: sample mean, median, and mode, robust mean, median, and
+mode, and the clipped mean and median.
+The \code{fringe} calibration stage must construct residual images, in
+which the master fringe image is applied to the input images, along
+with all necessary preceeding calibration images.
+The \code{fringe} calibration stage must measure the residual fringe
+amplitude on the residual images.
 \paragraph{low-k sky models}
+Large-scale background structure in images which is not caused by
+thin-film interference must also be detected and corrected.  Models of
+this background structure may be the necessary input to the correction
+proceedure.  The IPP must have the capability of generating image
+models of the large-scale structure patterns observed with the
+telescope.  \tbd{discuss principal components, SVD?}
+The \code{sky model} calibration stage must construct a sky model
+image from a stack of raw night-time sky images.
 \paragraph{Flat-field correction frame}
+Flat-field images, whether constructed from the dome, twilight, or
+night-sky images, rarely will perfectly correct the detector response
+in a consistent fashion across the full field of the camera.  The IPP
+must have the capability of generating flat-field photometric
+correction frames on the basis of the measured photometry of objects
+which are placed at a variety of locations on the detector in a
+sequence of images.
+The \code{flat-field correction} calibration stage must construct a
+flat-field correction image from dithered observations of a stellar
+field.
+The \code{flat-field correction} calibration stage must construct a
+flat-field correction image for each filter and camera independently.
+The \code{flat-field correction} calibration stage must construct a
+correction image which makes the photometry of multiple observations
+of the same stellar source consistent at different locations in the
+focal plane.
+The \code{flat-field correction} calibration stage must construct
+corrected flat-field images using the measured correction.
+The \code{flat-field correction} calibration stage must determine the
+consistency of the corrected flat-field images using the dithered
+stellar field observations flattened with the corrected flat-field
+image..
 \paragraph{Non-linearity correction frames}
+The IPP must have the capability of constructing non-linear correction
+frames.  These frames are constructed from exposures of a uniform
+source with a range of exposure times.  The non-linearity correction
+frames provide polynomial correction coefficients as a function of
+pixel to convert the observed pixel counts to the expected pixel count
+from a linear detector.
+The \code{non-linear correction} calibration stage must construct a
+non-linear correction from a collection of images of a constant source
+with varying exposure times.
+The \code{non-linear correction} calibration stage must construct a
+non-linear correction which linearizes the detector fluxes $<0.5\%$.
+The \code{non-linear correction} calibration stage must determine the
+saturation regime, in which the non-linear correction is no longer
+consistent to $<0.5\%$.
 \subsubsection{Reference Catalog Creation}
 For PS-1, one of the primary goals is the creation of photometric and astrometric
 reference catalogs for the general community and for the future
 Pan-STARRS requirements.  The generation of these catalogs is
+For PS-1, one of the primary goals is the creation of photometric and
+astrometric reference catalogs for the general community and for the
+future Pan-STARRS calibration.  The generation of these catalogs is
 inherently a research project, and will require human control and
 intervention.  The IPP will be required to provide the data access,
 …
 \subsubsection{Astrometry Reference Creation}
+The existing astrometric reference catalogs are known to have
+limitations at the level of \tbd{NN} milli-arcsec.  The internal
+accuracy of the Pan-STARRS dataset can potentially be much higher than
+the external reference catalogs.  The IPP must have the capability of
+generating an astrometric reference on the basis of the observations
+obtained by the PnA survey.  The IPP must provide the analysis tools
+needed to generate the master astometric reference catalog.  Much of
+the required functionality is covered by the PnA Database.
+The necessary ingredients for the construction of the PS-1 Astrometric
+Reference Catalog are: the observed coordinates of stars and the
+existing astrometric reference catalogs.  A variety of reference
+catalogs will be required, including:
+\begin{itemize}
+\item Hipparcos
+\item Tycho2
+\item UCAC
+\item YBx
+\item USNO-Bx
+\item 2MASS
+\end{itemize}
+These catalog must be available and accessible to the PnA Database.
+It is necessary to have the tools to convert the reference catalog
+object coordinates to all of the possible coordinate frame of
+relevance in the telescope and camera system, including:
+\begin{table}
+\begin{center}
+\caption{Astrometric Reference Catalogs\label{AstroRefs}}
+\begin{tabular}{lrrr}
+\hline
+\hline
+Name       & scatter & depth & filters \\
+           & arcsec  & mag   &         \\
+\hline
+Hipparcos  & & & \\
+Tycho2     & & & \\
+UCAC       & & & \\
+YBx        & & & \\
+USNO-Bx    & & & \\
+MASS      & & & \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+The IPP must have the capability of generating an astrometric
+reference on the basis of the observations obtained by the AP survey.
+The IPP must provide the analysis tools needed to generate the master
+astometric reference catalog.  Much of the required functionality is
+covered by the AP Database.
+The Astrometry Reference creation tools must return the match between
+stars observed with Pan-STARRS and any of several astrometric
+reference catalogs listed in Table~\ref{AstroRefs}.
+The tools must convert the reference catalog object coordinates to all
+of the coordinate frames of relevance in the telescope and camera
+system:
 \begin{itemize}
 \item Catalog to mean positions
 …
 \end{itemize}
+In addition to the reference catalogs, it will be necessary to
+determine and have available for the analysis system a variety of
+approximate calibration data, including the telescope and camera
+optical distortion models and the variation of the image PSF across
+the camera field, as a function of color.
+The final ingredient in the astrometry solution is the observation of
+stars with the PS-1 telescope.  The object detections are produced by
+several of the analysis stages discussed in the Science Analysis
+section.  The likely measurement of relevance to the astrometric
+reference catalog is the object extraction for the individual,
+detrended images (section~\ref{foo}).  \tbd{is it necessary to have
+  multiple centroiding methods available?}.  The detected objects must
+be matched against the reference catalogs, and it must be possible to
+determine fit coefficients as a function of simply position, or with
+combinations of magnitude or color.  The fitting method must include
+robust outlier rejection.  It is also necessary to have information
+about the objects which are detected in the catalog, but not the
+science image or vice-versa, as well as an assessment of the
+centroiding errors for each object.  It must be possible to plot the
+fit residuals against a wide variety of parameters, including the
+object positions, magnitudes, colors, etc, and to make subset
+selections of the objects on the basis of these parameters.  .
+An alternative measurement of the stellar positions is derived from
+the guide stars, which are much brighter than the typical saturated
+stars.  It must be possible to compare the coordinates of the guide
+stars with the coordinates of the other stars in the image.  It must
+also be possible to perform the various fitting steps for the guide
+stars rather than for the normal image data.
+The tools must provide the necessary calibration data: the telescope
+and camera optical distortion models and the variation of the image
+PSF across the camera field, as a function of color.
+The tools must fit the observed stellar coordinates to the astrometric
+reference catalog coordinates to determine improved astrometric
+solutions for both the stars and the detectors.
+The tools must determine improved telescope optical distortion models
+based on the astrometric solutions.
+The tools must optionally determine the fit coefficients as a function
+of position along, or with combinations of magnitude or color.
+The fitting method must include robust outlier rejection.
+The tools must identify objects which are detected in the catalog, but
+not the science image or vice-versa.
+The tools must determine average centroiding errors for each object.
+The tools must plot the fit residuals against a wide variety of
+parameters: the object positions, magnitudes, colors, etc.
+The tools must allow the fit to exclude subsets of objects from the
+fits on the basis of these parameters.  .
+The tools must provide coordinates of the guide stars in the same frame
+of reference as the normal image data.
+The tools must perform the various fitting steps for the guide stars
+rather than for the normal image data.
 \subsubsection{Photometry Reference Creation}
+The IPP must provide the analysis tools needed to generate a master
+photometric reference catalog.  The tools needed for generation of the
+photometric reference catalogs are similar in essence to those used
+for the astrometric reference catalog.  It is necessary to confront
+the observed objects against the existing reference catalogs to
+determine the necessary calibrations.  Again, much of the required
+functionality is covered by the PnA Database.
+The necessary ingredients for the construction of the PS-1 Photometric
+Reference Catalog are: the observed magnitudes of stars and the
+existing photometric reference catalogs.  A variety of reference
+catalogs will be required, including:
+\begin{itemize}
+\item SDSS
+\item CFHT-LS standards
+\item Landolt
+\item etc
+\end{itemize}
+These catalog must be available and accessible to the PnA Database.
+The final ingredient in the photometry solution is the observation of
+stars with the PS-1 telescope.  The object detections are produced by
+several of the analysis stages discussed in the Science Analysis
+section.  The likely measurement of relevance to the photometric
+reference catalog is the object extraction for the individual,
+detrended images (section~\ref{foo}).  It is necessary to have the
+tools to convert between different photometric systems, including:
+\begin{table}
+\begin{center}
+\caption{Photometric Reference Catalogs\label{PhotoRefs}}
+\begin{tabular}{lrrr}
+\hline
+\hline
+Name       & scatter & depth & filters \\
+           & mmag    & mag   &         \\
+\hline
+SDSS       & & & \\
+CFHT-LS    & & & \\
+Landolt    & & & \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+The IPP must have the capability of generating a photometric reference
+on the basis of the observations obtained by the AP survey.  The IPP
+must provide the analysis tools needed to generate a master
+photometric reference catalog.  Much of the required functionality is
+covered by the AP Database.
+The Photometry Reference creation tools must return the match between
+stars observed with Pan-STARRS and any of several photometric
+reference catalogs listed in Table~\ref{PhotoRefs}.
+The tools must convert between different photometric systems, including:
 \begin{itemize}
 \item instrumental to nominal detector magnitude
 …
 \item average filter system to reference photometry system
 \end{itemize}
+These transformations are based on a set of measured coefficients for
+the color and airmass dependency of the measurement.  In addition to
+these types of transformations, it is necessary to have the ability to
+measure and apply relative photometry corrections.
+The detected objects must be matched against the reference catalogs,
+and it must be possible to determine fit coefficients as a function of
+airmass, magnitude, color and detector coordinates, or with
+combinations of the above.  The fitting method must include robust
+outlier rejection.  It is also necessary to perform exclusions on the
+basis of object locations, instrumental magnitudes, observed and
+reference errors, and in particular time of the observations. It must
+be possible to plot the fit residuals against a wide variety of
+parameters, including the object positions, magnitudes, colors, etc,
+and to make subset selections of the objects on the basis of these
+parameters.  .
+An alternative measurement of the stellar positions is derived from
+the guide stars, which are much brighter than the typical saturated
+stars.  It must be possible to relate the magnitudes of the guide
+stars with the magnitudes of the other stars in the image.  It must
+also be possible to perform the above fitting steps for the guide
+stars rather than for the normal image data.
+These transformations must account for color and airmass terms.
+The tools must measure and apply relative photometry corrections
+between images.
+The tools must determine photometric transformation fit coefficients
+as a function of airmass, magnitude, color and detector coordinates,
+or with combinations of the above.
+The fitting method must include robust outlier rejection.
+The tools must reject specific objects from the fit on the basis of
+object locations, instrumental magnitudes, observed and reference
+errors, and in particular time of the observations.
+The tools must plot the fit residuals against a wide variety of
+parameters, including the object positions, magnitudes, colors, etc.
+The tools must provide photometry from the guide stars in the same
+system as observations of stars from the normal imaging data.
+The tools must perform the above fitting steps for the guide stars
+rather than for the normal image data.
 \subsection{Modules}
 …
 \subsubsection{Image Formats}
+FITS images
+Certain IPP programs must be able to read and write standard FITS images.
+Certain IPP programs must be able to read and write files in modified
+FITS format with Pan-STARRS definitions for non-square pixel arrays.
 \subsubsection{Table Formats}
+FITS tables
+Certain IPP programs must be able to read and write FITS tables.
 \subsubsection{Other Data Formats}
+XML files
+Certain IPP program must be able to read and write XML files.
 \subsubsection{External Catalogs}
+The IPP AP Database must be able to interact with the following
+externally provided reference catalogs:
 \begin{itemize}
 …
 \subsubsection{Analysis Reference Data}
+The IPP must store reference data describing the following entities:
 \begin{itemize}
 \item Telescopes
 …
 \item Filters
 \item software basic parameters
+\item computer configuration
 \end{itemize}
-\subsubsection{Installation Reference Data}
-\begin{itemize}
-\item computers
-\end{itemize}
 \subsection{External Interfaces}
 …
 \subsubsection{Overview}
+\tbd{this section should be parred down a bit by referring more to the
+  hardware report}.
 This section discusses the Pan-STARRS Image Processing Pipeline (IPP)
 …
 \end{itemize}
+Even without the complete IPP design, it is possible to identify the
+major drivers on the hardware requirements.  The total disk volume
+requirements are dominated by the need to store raw images for a
+certain period, the need to store calibration images for a longer
+period, and the need to store the static sky images.  Of the various
+analysis stages, Phase 2 and Phase 4 present the most significant
+demands in terms of data I/O throughput on the network.  Phase 2 and
+Phase 4 also present the most significant CPU demands.  In this
+discusion, Phase 2 refers to the per-OTA image pre-processing in which
+the instrumental signature is removed and a first pass object
+detection is performed.  Phase 4 refers to the multiple OTA
+combination in which the pre-processed images are merged and combined,
+in both addition and subtraction, with the static sky image, and up to
+three object detection passes are performed.
+This document does not address the hardware requirements implied by
+Phase 1 or 3, nor the load required by the calibration or reference
+catalog creation stages.  In the first instance, the operations are
+only performed on the metadata and are extremely minimal both in terms
+of data I/O and computation requirements.  In the second case, the
+processing is less time critical than the per-image processing and is
+performed only infrequently (once per night to once per week, month or
+year).  \tbd{The software implementation for metadata storage (RDBMS,
+FITS tables, etc) will have a very large impact and will be evaluated
+along with the needed hardware at a later date.}
+We will address the various hardware requirements by referring to an
+assumed data processing and data organization scenario.  The
+We will address the various hardware requirements by referring to the
+assumed data processing and data organization scenarios discussed in
+the document \tbd{Pan-STARRS IPP Hardware Report, PSDC-4xx-xx}.  The
 organization of the data and certain aspects of the data processing
+scheme have very large implications for the hardware requirements.  In
+this analysis, we assume that data types are chosen to minimize the
+data volume and that the data is organized to minimize the I/O
+bandwidth needs, as defined below.  We address the data requirements
+scheme have very large implications for the hardware requirements.  We
+use the values from that report representing the minimum data volume
+and the optimum data organization.  We address the data requirements
 of the single-telescope Pan-STARRS-1 scenario based on the Design
 Reference Mission \tbd{REF}.
-\subsubsection{Data Organization}
-The IPP hardware system must provide both data storage and
-computational resources.  The IPP requires relativley large amounts of
-data storage space, primarily for the image data.  Image data is
-organized in two categories.  First, there is the per-OTA data -- data
-associated with specific OTAs, including the raw images, the
-calibration images, and temporary processed images at various stages.
-Second, there is the data associated with the static sky imagery,
-which is in turn organized into smaller sky-cell units.  The first
-assumption we make is that the hardware is organized into nodes which
-provide both data storage and computational resources.  The second
-assumption we make is that the data storage nodes are divided into two
-classes: those which deal with the per-OTA data and those that provide
-the static sky storage.  In addition, we assume that the computational
-tasks related to Phase 2 take place on the per-OTA storage nodes and
-the Phase 4 computation takes place on the static sky storage nodes.
-Figure~\ref{hardware} shows our basic concept for the hardware
-organization for the IPP.  This diagram shows the two types of compute
-nodes: OTA-level processing and storage nodes (dominated by Phase 2)
-and static sky processing and storage nodes (mostly Phase 4).  Also
-shown are two switches used in this configuration; although it is
-currently possible to buy a single switch with sufficient number of
-ports, this organization represents a minimal configuration for the
-PS-1 IPP hardware.  In such a case, the interswitch communication must
-also meet the required throughput needs.  We discuss the hardware
-requirements in the assumption that such an organization will be
-necessary.
-The way in which the images are distributed among the storage and
-compute nodes will largely determine the I/O bandwidth requirements.
-For data bandwidth requirements calculations, it is necessary to make
-some assumptions about the data organization.  We make the assumption
-that the OTA data is optimally distributed to the OTA nodes such that
-the OTA processing is always on a machine with local OTA data.  This
-implies that all OTA data from a specific OTA are targetted to a
-specific machine.  (see below for discussion of data duplication).
-A second factor which will have a significant impact on the I/O
-requirements is the image storage format for the processed and
-calibration images.  We have two basic choices: 32 bit floating point
-format or 16 bit integer format with appropriate scaling.  In the
-former case, additional dynamic range is retained, while in the latter
-case, we reduce the data volume by a factor of 2.  Since the science
-requirements for PS-1 do not specify a need for dynamic range greater
-than 16 bits, we assume all images are stored as 16 bit data.
-A third determining factor is the number of calibration images needed
-by the processing system.  Since the complete analysis is not yet
-defined, this number is difficult to ascertain.  However, we can make
-a reasonable guess at the total number for scaling purposes.  We
-assume that each frame requires a total of 4 calibration frames on
-average
 \begin{table}[b]
 \begin{center}
 \caption{Data Storage Requirements \label{storage}}
 \begin{tabular}{lrrrr}
+\begin{tabular}{lr}
 \hline
 \hline
 …
 governed by the number of nights' worth of data we are required to
 keep online.  \tbd{for the first year, we are required to keep all
 images from the PnA and IPV surveys.  This amounts to a total of 200
+images from the AP and IPV surveys.  This amounts to a total of 200
 TB of data}.
 …
 \begin{center}
 \caption{Data I/O (MB per OTA or Sky-cell) \label{scenarios}}
 \begin{tabular}{lrrrr}
+\begin{tabular}{lr}
 \hline
 \hline
 …
 output:        &                            96 MB  \\
 \hline
 \multicolumn{5}{l}{\em Bold-faced entries are access to local-disk} \\
 \multicolumn{5}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\
+\multicolumn{2}{l}{\em Bold-faced entries are access to local-disk} \\
+\multicolumn{2}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\
 \end{tabular}
 \end{center}
 …
 \begin{center}
 \caption{Data Throughput \label{throughput}}
 \begin{tabular}{lrrrr}
+\begin{tabular}{lr}
 \hline
 \hline
 …
 summit-to-Phase 2 switch load is 70 MB/s.
-\begin{table}
-\begin{center}
-\caption{Hardware Throughput Tests \label{existing-hardware}}
-\begin{tabular}{lrrrr}
-\hline
-\hline
-Test        & where \& when     & model                & result                             \\
-\hline
-node I/O    & CFHT 11/2002      & Intel 1000 Gigabit   & 35 - 40 MB/s sustained             \\
-node I/O    & CFHT 2/2004       & Intel 1000 Gigabit   & 65 - 70 MB/s sustained             \\
-RAID write  & CFHT 2/2004       & 3ware RAID cntl + IDE & 110 MB/s sustained                 \\
-Switch Load & VeriTest          & Cisco                & 3 GB/s (for 32 ports)              \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
-\subsubsection{Existing Hardware Throughput}
-We have collected a few representative tests of various pieces of
-modern hardware to give a reference for the throughput capabilities.
-A number of hardware configurations have been tested at CFHT for the
-Elixir project, and we include here their recent reported hardware
-RAID-5 I/O speeds and GigE card speeds.  We also have included data
-from VeriTest studies of Cisco switch throughput, commissioned by
-Cisco for a 32 port GigE switch.  These tests are summarized in
-Table~\ref{existing-hardware}.
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 …
 \bibliography{panstarrs}
 \end{document}
-Requirements Trace Matrix
-active state \ref{req:active-state}
-paused state \ref{req:paused-state}
-interactive state \ref{req:interactive-state}
-system capabilities
-C for source code \ref{req:languages}
-Python for scripts \ref{req:languages}
-SWIG interfaces
-C APIs
-POSIX
-Pan-STARRS Coding Standard
-Naming Conventions

Note: See TracChangeset for help on using the changeset viewer.

Context Navigation

Changeset 837 for trunk/doc/design

Legend:

trunk/doc/design/ippSRS.tex

Download in other formats: