Index: /trunk/doc/design/ippSRS.tex =================================================================== --- /trunk/doc/design/ippSRS.tex (revision 881) +++ /trunk/doc/design/ippSRS.tex (revision 882) @@ -1,3 +1,3 @@ -%%% $Id: ippSRS.tex,v 1.3 2004-06-03 02:56:58 eugene Exp $ +%%% $Id: ippSRS.tex,v 1.4 2004-06-05 00:49:48 eugene Exp $ \documentclass[panstarrs]{panstarrs} @@ -84,6 +84,8 @@ \DocumentsInternalSection PSDC-130-001 & PS-1 Design Reference Mission \\ \hline +PSDC-130-xxx & PS-1 SCD \\ \hline PSDC-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline PSDC-430-006 & Pan-STARRS IPP ADD \\ \hline +PSDC-430-006 & Pan-STARRS IPP SDRS \\ \hline PSDC-430-007 & Pan-STARRS IPP PSLib SDRS \\ \hline \DocumentsExternalSection @@ -101,66 +103,85 @@ after AP survey} +\tbd{define astronomy concepts: cleaned, pre-processed, significant, +outlier} + The IPP must perform the following tasks: \begin{enumerate} -\item Accept raw images from OATS at a sustained rate of 1 exposure - per 30 seconds. - -\item Accept metadata from OATS at a sustained rate of \tbd{XXX MB / sec}. - -\item Produce high-quality calibration images from the raw calibration - images. The calibration images must not introduce systematic - uncertainties greater than \tbd{0.2\%}. \tbd{Requirements on the - speed of processing the calibration images.} - -\item Pre-process the science images with the high-quality calibration - images. +\item Accept raw images from the summit at a sustained rate of 1 + exposure per 30 seconds. + +\item Accept metadata from the summit at a sustained rate of \tbd{XXX +MB / sec}. + +\item Produce high-quality master calibration images from the raw + calibration images. The master calibration images must not + introduce systematic uncertainties greater than \tbd{0.2\%}. + \tbd{Requirements on the speed of processing the calibration + images.} + +\item Pre-process the science images with the high-quality master + calibration images. \item Merge multiple pre-processed science images -- from multiple telescopes or from sequential, dithered exposures -- into single, - cleaned, stacked images. + cleaned, stacked images with corresponding signal-to-noise maps. \item Subtract a static sky image from the cleaned, stacked images to - produce an image of only the transient events. + produce an image of only the transient objects. \item Excise the significant transients and outliers from the - pre-processed science images and merge the cleaned images into the - static sky image. - -\item Detect objects on the four types of images: pre-processed - images, the stacked image, the difference image, and the static sky - image. + pre-processed science images. \tbd{how to handle variable stars?} + +\item merge the cleaned images into the static sky image, and + update the corresponding exposure (S/N) maps. + +\item Detect and measure parameters of objects on the four types of + images: pre-processed images, the stacked image, the difference + image, and the static sky image. \item Determine astrometry of the detected objects relative to an - astrometric reference to an accuracy of \tbd{30 mas}. + astrometric reference to an accuracy of \tbd{30 mas}, with a limit of + \tbd{xxx} on the outliers. \item Determine photometry of the detected objects relative to a photometric reference to an accuracy of \tbd{5 millimag} relative photometry and \tbd{10 millimag} absolute photometry in photometric - weather. + weather. \tbd{before vs after PS-1 AP Surver} \tbd{bright vs faint + errors} with a limit of \tbd{xxx} on the outliers. \tbd{limit + depends on filter} \item Produce a high-quality astrometric reference catalog from the extracted objects on a time-scale of 6 months. The astrometric reference must have an absolute accuracy of \tbd{30 mas} and a local - relative accuracy of \tbd{10 mas}. Proper motions of all nearly - stationary objects must be determined with an accuracy of \tbd{XXX - mas / year}. + relative accuracy of \tbd{10 mas}. Proper motions of detected + objects with distances greater than 1000 AU must be determined with + an accuracy of \tbd{XXX mas / year}. \item Produce a high-quality photometric reference catalog from the - extracted objects on a time-scale of 6 months. The photometric - reference must have an consistency across the sky of \tbd{5 - millimag} and an absolute calibration to the external system defined - by \tbd{SDSS} of \tbd{10 millimag}. + extracted point-source objects on a time-scale of 6 months. The + photometric reference must have an consistency across the sky of + \tbd{5 millimag} and an absolute calibration to the external system + defined by \tbd{SDSS} of \tbd{10 millimag}. \item Publish the static sky images to the Pan-STARRS published static - sky server on a time-scale of \tbd{1 month}. + sky server on a time-scale of \tbr{1 month}. \item Publish the detected objects to the Pan-STARRS published object - database on a time-scale of \tbd{1 week}. + database on a time-scale of \tbr{1 week}. \item Provide access to external Pan-STARRS clients to the detected - objects on time-scales of \tbd{1 minute} after the image is - processed. + objects on time-scales of \tbr{10 minute} after the image is + obtained.\comment{this is a top-level science requirement.} + +\item Store the raw images for a particular period of, depending on + the survey source of the data. In PS-1, the AP and IVP Survey data + must be stored for the lifetime of the project. Other raw data must + be store for \tbr{1 month}. + +\item Store the detected objects for a period of time, depending on + the type of detection. Transients from the P4$\Delta$ images may be + excised after \tbr{6 monts}. \end{enumerate} @@ -198,5 +219,6 @@ Source code must be in C. All source code must be compiled with `gcc' -version v2.95 or higher. +version v2.95 or higher, and the tested compiler version must be +defined for the delivered software product. Scripting language must be \tbd{Python, version TBD}. @@ -355,11 +377,12 @@ \subsection{Architectural Components} -As discussed in the Pan-STARRS System Concept Definition, the IPP is -organized into a number of clearly-defined software elements. The SCD -provides a detailed description of the roles and responsibilities of -these subsystems. In brief, the IPP consists of a collection of -science analysis stages, a set of architectural components which -provide the infrastructure needed to run the analysis programs, and a -collection of hardware on which all of the software elements exist. +As discussed in the Pan-STARRS System Concept Definition +(PSDC-xxx-xxx), the IPP is organized into a number of clearly-defined +software elements. The SCD provides a detailed description of the +roles and responsibilities of these subsystems. In brief, the IPP +consists of a collection of science analysis stages, a set of +architectural components which provide the infrastructure needed to +run the analysis programs, and a collection of hardware on which all +of the software elements exist. The architectural components consist of: @@ -383,6 +406,8 @@ \item {\bf Metadata Database:} This component is required to store the - data which is not directly related to images or astronomical objects - as needed to perform the analysis specified above. + all other data which are neither image files nor astronomical object + data. The Metadata Database is the authoratative source for all + metadata data, including metadata which may be duplicated elsewhere, + such as in the headers of images in the image database. \item {\bf Controller:} In order to perform the analysis stages @@ -424,6 +449,6 @@ machine and notify the requesting agent of the new locations. -The IPP Image Server be able to maintain multiple copies of each -image, as specified by the user. +The IPP Image Server store multiple copies of each image, the number +of copies specified independently for each by the user. The IPP Image Server must maintain a record of all image copies @@ -551,5 +576,5 @@ \begin{table} \begin{center} -\caption{AP Detection Classes \& Object Parameters\label{APdetections}} +\caption{AP Data Volume Requirements\label{APrates}} \begin{tabular}{lrrrr} \hline @@ -557,12 +582,12 @@ Quantity & P2 & P4$\Sigma$ & P4$\Delta$ & SS \\ \hline -detection limit & $20 \sigma$ & $5 \sigma$ & $3 \sigma$ & \\ -depth (r') & 20.8 & 23.0 & & \\ -stars deg$^{-2}$ ($|b|>10$) & $1 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ \\ -stars FPA$^{-1}$ ($|b|>10$) & $7 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ \\ -stars sec$^{-1}$ ($|b|>10$) & $2.3 \times 10^4$ & $1 \times 10^5$ & $1 \times 10^5$ & $1 \times 10^5$ \\ -bytes star$^{-1}$ & 64 & 100 & 64 & \\ -MB sec$^{-1}$ & 1.4 & 2.2 & 0.7 & \\ -PS-1 total TB & 8 & 12 & 4 & \\ +detection limit & $20 \sigma$ & $5 \sigma$ & $3 \sigma$ & \\ +depth (r') & 20.8 & 23.0 & ? & \\ +stars deg$^{-2}$ ($|b|>10$) & $1 \times 10^5$ & $4 \times 10^5$ & $2 \times 10^5$ & \\ +stars FPA$^{-1}$ ($|b|>10$) & $7 \times 10^5$ & $2.8 \times 10^6$ & $1.4 \times 10^6$ & \\ +stars sec$^{-1}$ ($|b|>10$) & $2.3 \times 10^4$ & $2.3 \times 10^4$ & $1.2 \times 10^4$ & \\ +bytes star$^{-1}$ & 64 & 100 & 64 & \\ +MB sec$^{-1}$ & 1.4 & 2.2 & 0.7 & \\ +PS-1 total TB & 8 & 12 & 4 & \\ \hline \end{tabular} @@ -578,5 +603,11 @@ rates. -\subsubsection{Metadata Database -- FIX ME} +\tbd{archive lifetime} + +\tbd{reliability} + +\tbd{backups} + +\subsubsection{Metadata Database} \tbd{this section needs to be reviewed and revised} @@ -605,14 +636,18 @@ \tbd{queries} -{\bf note: description of images belong in the Metadata database, - location of images is in the Image server} - -\paragraph{Configuration Database -- FIX ME} - -The IPP requires a Configuration Database to store and provide access to -information about the IPP itself. Examples of data in the +\tbd{description of images belong in the Metadata database, location + of images is in the Image server} + +\paragraph{Configuration Database} + +The IPP requires a Configuration Database to store and provide access +to information about the IPP itself. Examples of data in the configuration database include the default parameters for the various analysis programs, the description of the computing environment, the process status information, etc. \tbd{part of metadata database?}. + +\tbd{some information must have access limited to specific responsible + people. ie, software / hardware configuration $\rightarrow$ sysadmin; + science parameters $\rightarrow$ science team.} \subsubsection{Controller} @@ -880,4 +915,6 @@ peak-to-peak deviations of \tbr{0.5\%}. +\tbd{color of stars in flat-field correction?} + \paragraph{Sky \& Fringe subtraction} @@ -1029,5 +1066,6 @@ The output image must maintain photometric consistency with the input -image to within 0.2\%. \tbd{interpolation method?} +image to within 0.2\%. \tbd{does interpolation method choice risk +losing flux?} \paragraph{Flux matching} @@ -1036,10 +1074,20 @@ factors needed to combine the images photometrically. +\tbd{is flux matched automatically by calibration?} + \paragraph{Image outlier pixel rejection} -Pixels from the group of images which are inconsistent \tbd{how much?} -with the ensemble of pixel values must be identified and flagged. +When multiple images are combined, the group of input pixels which +contribute to an output pixel must be examined and pixels from the +group of images which are inconsistent with the ensemble \tbd{how +much?} must be identified and flagged. This outlier rejection must be performed optionally. + +\tbd{for moving objects and images which are not simultaneous, do we + identify the moving objects?} + +\tbd{use the spatial information? fit a 2-D Nth order polynomial to + the collection of pixels and then look for outliers} \paragraph{Initial cleaned image} @@ -1056,4 +1104,6 @@ The static sky image must be subtracted from the stacked, cleaned image. + +\tbd{what about different stellar colors?} \paragraph{Find objects in the image} @@ -1122,4 +1172,8 @@ the relevant image metadata (\ie filter, exposure time, etc). +\paragraph{Image Processing Q/A} + +Before the image is added to the static sky, it must pass Q/A tests. + \paragraph{Update static sky} @@ -1372,4 +1426,5 @@ \item Catalog to mean positions \item Mean to apparent positions +\item Apparent positions + pointing to tangent plane coordinates \item Apparent positions + pointing to focal plane coordinates \item focal plane to specific detector (OTA) @@ -1552,4 +1607,18 @@ \subsection{External Interfaces} +Images from OATS / Camera data store. + +Summit Metadata from OATS. + +Image Q/A assessment to OATS + +3$\sigma$ transient non-orphaned detections to MOPS + +3$\sigma$ transient detections to Transient Science Client + +Published static sky images to Science Database + +Published objects (P2, P4S, P4D, SS) to Science Database + \subsection{Internal Interfaces} @@ -1562,4 +1631,7 @@ \tbd{this section should be parred down a bit by referring more to the hardware report}. + +\tbd{switch to passive voice (we will address foo $\rightarrow$ foo is +addressed)} This section discusses the Pan-STARRS Image Processing Pipeline (IPP) @@ -1597,5 +1669,5 @@ object db & 4 TB \\ \hline -total & 116 TB \\ +total & 466 TB \\ \hline \end{tabular} @@ -1778,6 +1850,6 @@ input images & $1.5 \times 4 \times 32$ MB \\ input masks & $1.5 \times 4 \times 8$ MB \\ -static sky & 32 MB \\ -static weight & 32 MB \\ +static sky & 32x9/4 MB \\ +static weight & 32x9/4 MB \\ \hline input: & 304 MB \\ @@ -1913,6 +1985,7 @@ \subsection{Software Integrity Tests} -MHPCC must test the integrity of the software, specifically to check -that the code does not produce memory leaks, segmentation faults. +MHPCC must test that the code does not produce memory leaks. + +MHPCC must test that the code does not produce segmentation faults. \subsection{Basic Unit Tests}