Index: trunk/doc/design/ippSRS.tex =================================================================== --- trunk/doc/design/ippSRS.tex (revision 1399) +++ trunk/doc/design/ippSRS.tex (revision 2114) @@ -1,3 +1,3 @@ - %%% $Id: ippSRS.tex,v 1.7 2004-08-06 19:06:01 eugene Exp $ + %%% $Id: ippSRS.tex,v 1.8 2004-10-14 05:06:32 eugene Exp $ \documentclass[panstarrs,spec]{panstarrs} @@ -154,4 +154,23 @@ \section{Requirements} +\begin{table} +\begin{center} +\caption{Valid Moon Conditions for the 6 PS filters\label{moonconditions}} +\begin{tabular}{lrrrr} +\hline +\hline +filter & phase (days) & min. distance (degrees) \\ +\hline +g & 6 & 60 \\ +r & 5 & 40 \\ +i & 4 & 30 \\ +z & 2 & 20 \\ +y & 1 & 10 \\ +w & 5 & 50 \\ +\hline +\end{tabular} +\end{center} +\end{table} + \subsection{Top-Level Requirements} \label{req:system-capabilities} @@ -162,84 +181,111 @@ \begin{enumerate} -\item Produce reduced science images for each full camera exposure - which are photometrically consistent across the field to within 1\%.\VER{ANALYSIS}{SCD:3.2.2.5} +\item For images obtained in photometric weather, produce reduced + science images for each full camera exposure with photometric + zero-point scatter less than 1\% across the full + field. \VER{ANALYSIS}{SCD:3.2.2.5} \label{TLR:1} -\item Produce reduced science images for each full camera exposure - which are photometrically calibrated to within 1\%.\VER{ANALYSIS}{SCD:3.2.2.5} +\item For images obtained in photometric weather, produce reduced + science images for each full camera exposure which are + photometrically calibrated with respect to the Pan-STARRS filter + system with a 1$\sigma$ accuracy of 1\%.\VER{ANALYSIS}{SCD:3.2.2.5} \label{TLR:2} -\item Produce reduced science images for each full camera exposure - which are astrometrically calibrated to 100 milliarcseconds to an - absolute reference.\VER{ANALYSIS}{SCD:3.2.2.6} +\item For images obtained under normal seeing conditions and optical + distortion, produce reduced science images for each full camera + exposure with an astrometric calibration providing $< 30$ + milliarcsecond scatter (1$\sigma$) for sequential images of the same + location.\VER{ANALYSIS}{SCD:3.2.2.7} + \label{TLR:4} + +\item For images obtained under normal seeing conditions and optical + distortion, produce reduced science images for each full camera + exposure with an astrometric calibration providing $< 100$ + milliarcsecond scatter (1$\sigma$) relative to the ICRS reference + system.\VER{ANALYSIS}{SCD:3.2.2.6} \label{TLR:3} -\item Produce reduced science images for each full camera exposure - which are astrometrically consistent to 30 - milliarcseconds.\VER{ANALYSIS}{SCD:3.2.2.7} - \label{TLR:4} - -\item Produce reduced science images for each full camera exposure - which have foreground emission subtracted with no more than 1\% - variation in the non-astronomical background.\VER{ANALYSIS}{SCD:3.5.12} +\item In photometric weather and under moon conditions listed in + Table~\ref{moonconditions}, produce reduced science images for each + full camera exposure which have background variations of less than + 1\% in regions free of large ($> 30$ pixels diameter) astronomical + structures.\VER{ANALYSIS}{SCD:3.5.12} \label{TLR:5} -\item Merge all $g$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} +\item In photometric weather, produce reduced science images for each + full camera exposure which have background deviations from the + static sky in the same filter of less than \tbd{1\%} for the median + in large ($> 30$ pixels diameter) regions.\VER{ANALYSIS}{SCD:3.5.12} + \label{TLR:5a} + +\item Merge all $g$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10} \label{TLR:6} -\item Merge all $r$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} +\item Merge all $r$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10} \label{TLR:7} -\item Merge all $i$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} +\item Merge all $i$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10} \label{TLR:8} -\item Merge all $z$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} +\item Merge all $z$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10} \label{TLR:9} -\item Merge all $y$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} +\item Merge all $y$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10} \label{TLR:10} -\item Merge all $w$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} +\item Merge all $w$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10} \label{TLR:11} -\item Detect and classify objects on the individual processed science images.\VER{TEST}{SCD:3.2.2.16} +\item Detect and classify objects on the individual processed science + images.\VER{TASK}{SCD:3.2.2.16} \label{TLR:12} -\item Detect and classify objects on the stacked groups of science images.\VER{TEST}{SCD:3.2.2.16} +\item Detect and classify objects on the stacked groups of science + images.\VER{TASK}{SCD:3.2.2.16} \label{TLR:13} -\item Detect and classify objects on the static sky image.\VER{TEST}{SCD:3.2.2.16} +\item Detect and classify objects on the static sky image.\VER{TASK}{SCD:3.2.2.16} \label{TLR:14} -\item Detect all significant transients in the individual science - images relative to the static sky image.\VER{TEST}{SCD:3.2.2.16} +\item Detect transients with significance $>3\sigma$ in the individual + science images relative to the static sky + image.\VER{ANALYSIS}{SCD:3.2.2.16} \label{TLR:15} -\item Degrade the stacked image by no more than \tbr{10 milliarcseconds}.\VER{ANALYSIS}{SCD:3.5.2} +\item Degrade the stacked image by no more than \tbr{10 + milliarcseconds} (FWHM added in quadrature) over the theoretical + limit for the stack under infinite + sampling.\VER{ANALYSIS}{SCD:3.5.2} \label{TLR:16} \item Perform the processing of science images to the level of transient detection and static sky inclusion at a rate such that - exposures taken at a cadence of \tbr{40} seconds do not accumulate - in the processing buffer.\VER{TEST}{SCD:3.2.2.3} + exposures taken at an average cadence of \tbr{40} seconds do not + accumulate in the processing buffer (average throughput + requirement).\VER{TEST}{SCD:3.2.2.3} \label{TLR:17} \item Limit the false alarm rate (FAR) to less than \tbr{5\%} for - transient detections $> 5\sigma$ sent to the preferred client science - pipelines.\footnote{note difference with PS-4: 1\%} - \VER{ANALYSIS}{SCD:3.2.2.13} + transient detections $> 5\sigma$ sent to the preferred client + science pipelines.\footnote{note difference with PS-4: 1\%} + \VER{ANALYSIS}{SCD:3.2.2.13} \label{TLR:18} + +\item Perform transient detection to a completeness of \tbr{99\%} at + the completeness for transient detections with a significant $> + 5\sigma$.\VER{ANALYSIS}{SCD:xxx} \item Publish the static sky images to the Pan-STARRS Published Science Products Subsystem (PSPS) once per \tbr{6 - months}.\VER{TEST}{SCD:3.2.2.18} + months}.\VER{TASK}{SCD:3.2.2.18} \label{TLR:19} \item Publish the detected objects to the Pan-STARRS Published Science - Products Subsystem (PSPS) once per month.\VER{TEST}{SCD:3.2.2.18} + Products Subsystem (PSPS) once per month.\VER{TASK}{SCD:3.2.2.18} \label{TLR:20} \item Send the IPP metadata and received OTIS metadata to the - Pan-STARRS Published Science Products Subsystem (PSPS) weekly.\VER{TEST}{SCD:3.2.2.18} + Pan-STARRS Published Science Products Subsystem (PSPS) weekly.\VER{TASK}{SCD:3.2.2.18} \label{TLR:21} @@ -441,5 +487,5 @@ Timing requirements specified in this document shall be achieved on the -deployed Pan-STARRS analysis computers.\VER{TEST}{allocated} +deployed Pan-STARRS analysis computers.\VER{INSPECT}{allocated} \subsubsection{Software Configuration} @@ -526,42 +572,61 @@ \subsubsection{Image Server} +%% IPP Image Server T & F + +Image Server tasks and functions: + +\begin{itemize} + +\item The IPP Image Server stores images on a distributed collection + of computer disks. Individual instances of a file are only required + to be stored on a single machine (striping across computers is not a + requirement). + +\item The IPP Image Server attempts to store an image on a specific + machine if requested by the user. + +\item If such a request cannot be honored (ie, the machine is down), + the IPP Image Server selects an appropriate machine and notifies the + requesting agent of the new location. + +\item The IPP Image Server stores multiple copies of each image upon + request, the number of copies specified independently for each file + by the user. + +\item The IPP Image Server maintains a record of all image copies + currently available in the repository. This record includes at + least the image name, location (which machine), the image size, and + the state of the image (available, locked, + deleted). + +\item The IPP Image Server locks images in the repository on request. + Both read (shared) and write (exclusive) locks are provided. A read + lock prevents write access to the file; a write lock prevents both + read and write access. Access prevention may be advisory rather + than rigorously enforced. + +\item The IPP Image Server return the image location (the computer or + computers on which it resides) upon request. + +\item The IPP Image Server provides a specified image upon request. + +\item The IPP Image Server deletes images in the repository on + request. +\end{itemize} + +%% IPP Image Server Requirements + +IPP Image Server requirements: + \begin{enumerate} \item The IPP Image Server shall accept raw images from the summit at - a sustained rate of 1 exposure (2~GB) per \tbr{40 - seconds}. \VER{TEST}{TLR:17, TLR:23} - -\item The IPP Image Server shall store images on a distributed - collection of computer disks. Individual instances of a file are - only required to be stored on a single machine (striping across - computers is not a requirement).\VER{TEST}{TLR:17, TLR:23} - -\item The IPP Image Server shall attempt to store an image on a - specific machine if requested by the user.\VER{TEST}{TLR:17, TLR:23} - -\item If such a request cannot be honored (ie, the machine is down), - the IPP Image Server shall select an appropriate machine and notify - the requesting agent of the new location.\VER{TEST}{TLR:17, TLR:23} - -\item The IPP Image Server shall store multiple copies of each image - upon request, the number of copies specified independently for each - file by the user.\VER{TEST}{TLR:17, TLR:23} - -\item The IPP Image Server shall maintain a record of all image copies - currently available in the repository. This record shall include at - least the image name, location (which machine), the image size, and - the state of the image (available, locked, deleted).\VER{INSPECT}{TLR:17, TLR:23} - -\item The IPP Image Server shall lock images in the repository on - request. Both read (shared) and write (exclusive) locks shall be - provided. A read lock shall prevent write access to the file; a - write lock shall prevent both read and write access. \tbr{Access - prevention may be advisory rather than enforced.} \VER{TEST}{TLR:17, TLR:23} - -\item The IPP Image Server shall return the image location (the - computer or computers on which it resides) upon request.\VER{TEST}{TLR:17, TLR:23} - -\item The IPP Image Server shall provide a specified image upon request.\VER{TEST}{TLR:17, TLR:23} - -\item The IPP Image Server shall delete images in the repository on request.\VER{TEST}{TLR:17, TLR:23} + a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds.} + \VER{TEST}{TLR:17, TLR:23} + +\item The IPP Image Server nodes shall not be offline for more than 12 hours + consecutively or 36 hours per year.\VER{ANALYSIS}{TLR:17} + +\item The IPP Image Server shall provide a seekable, unix file-system + reference to the specified image.\VER{TEST}{allocated} \end{enumerate} @@ -569,16 +634,5 @@ \subsubsection{AP Database} -The purpose of the AP Database is: -\begin{itemize} -\item to enable the photometric calibration of images -\item to enable the astrometric calibration of images -\item to enable the construction of flat-field correction frames -\item to enable the construction of a photometric calibration catalog -\item to enable the construction of an astrometric calibration catalog -\item to monitor the system photometry calibration parameters -\item to monitor the system astrometry calibration parameters -\item to perform the identification of single-occurance transients -\end{itemize} - +%%% Table: AP DB parameters \begin{table} \begin{center} @@ -603,90 +657,5 @@ \end{table} -\begin{enumerate} -\item The AP Database shall accept and store individual detections and - collections of detections along with information about the image - which provided the detections.\VER{TEST}{TLR:2, TLR:3, TLR:22, TLR:24} - -\item Detections shall be saved as one of several detection classes - (P2, P4$\Sigma$, P4$\Delta$, SS) and the AP Database shall store the - appropriate parameters, listed in Table~\ref{APdetections}, for each - class.\VER{TEST}{TLR:2, TLR:3, TLR:22, TLR:24} - -\item The AP Database shall identify the image which provided the - detection, or in the case of external references, an identifier - specific to the reference source.\VER{TEST}{TLR:2, TLR:3} - -\item The AP Database shall group detections into objects and measure - average parameters of those objects.\VER{ANALYSIS}{TLR:2, TLR:3, TLR:22} - -\item The AP Database shall store parallax and proper motion parameters - for a subset of the average objects.\VER{TEST}{TLR:2, TLR:3, TLR:22} - -\item The AP Database shall store image and filter calibration - information necessary to convert between instrumental magnitudes and - calibrated magnitudes in standard systems.\VER{INSPECT}{TLR:3} - -\item The AP Database shall perform at least the follow queries, with - constraints on the output based on at least time ranges, magnitude - limits, error limits: - - \begin{enumerate} - \item given $(RA,DEC)$ and a Radius, return all objects and/or - detections in the region.\VER{TEST}{TLR:2, TLR:3} - - \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all objects and/or - detections in the region.\VER{TEST}{TLR:2, TLR:3} - - \item given $(RA,DEC)$, return closest object.\VER{ANALYSIS}{TLR:2, TLR:3, TLR:22} - - \item given object ID, return all detections\VER{TEST}{TLR:2, TLR:3} - - \item given detection, return source image data.\VER{TEST}{TLR:2, TLR:3} - - \item given detection, return object.\VER{TEST}{TLR:2, TLR:3, TLR:22} - - \item given $(RA,DEC)$, return all images overlapping coordinate.\VER{ANALYSIS}{TLR:2, TLR:3} - - \item given $(RA,DEC)$ and a Radius, return all images overlapping region.\VER{ANALYSIS}{TLR:2, TLR:3} - - \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping region.\VER{ANALYSIS}{TLR:2, TLR:3} - - \item given detection instrumental magnitude, return derived - magnitudes based on calibration information.\VER{TEST}{TLR:2, TLR:3} - - \item given a collection of detections in a filter, determine the - object average magnitude in that filter.\VER{ANALYSIS}{TLR:2, TLR:3} - - \item given a collection of objects and detections, determine the - individual image zero-points.\VER{ANALYSIS}{TLR:2, TLR:3} - - \item given a region, return all possible combinations of the object - or detection magnitudes $(M_1 - M_2)$.\VER{TEST}{TLR:2, TLR:3} - - \item given a list of $(RA,DEC)$ entries, return all nearest objects.\VER{ANALYSIS}{TLR:2, TLR:3} - - \item given a filter, telescope, or detector, return all calibration - terms and history.\VER{TEST}{TLR:2, TLR:3} - - \item given a detection, return all non-detections from images which - overlapped the detection coordinates.\VER{ANALYSIS}{TLR:2, TLR:3, TLR:22} - \end{enumerate} - -\item The AP Database shall accept detection IDs of moving objects and - label the detections with the identified object.\VER{TEST}{TLR:2, TLR:3, TLR:22} - -\item The AP Database shall accept new detections at the rate - generated by the telescope from the Phase 2 and Phase 4 analysis. - \tbr{Except within 10 degrees of the galactic plane, the AP Database - shall keep up with the incoming rates.} The expected rates are - listed in Table~\ref{APrates}, along with the total data volume - required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2, TLR:3, TLR:22} - -\item The AP Database shall provide access to external Pan-STARRS - clients to the detected objects within \tbr{5 minute} after the - image is obtained.\VER{TEST}{TLR:22} -\label{IPP:DeReq:29c} -\end{enumerate} - +%%% Table: AP DB Throughput \begin{table} \begin{center} @@ -713,6 +682,115 @@ \end{table} +%% IPP AP DB T & F + +The purpose of the AP Database is: +\begin{itemize} +\item to enable the photometric calibration of images +\item to enable the astrometric calibration of images +\item to enable the construction of flat-field correction frames +\item to enable the construction of a photometric calibration catalog +\item to enable the construction of an astrometric calibration catalog +\item to monitor the system photometry calibration parameters +\item to monitor the system astrometry calibration parameters +\item to perform the identification of single-occurance transients +\end{itemize} + +The tasks and functions of the AP Database include: + +\begin{itemize} +\item The AP Database accepts and stores individual detections and + collections of detections along with information about the image + which provided the detections. + +\item Detections are saved as one of several detection classes (P2, + P4$\Sigma$, P4$\Delta$, SS) and the AP Database stores the + appropriate parameters, listed in Table~\ref{APdetections}, for each + class. + +\item The AP Database identifies the image which provided the + detection, or in the case of external references, an identifier + specific to the reference source. + +\item The AP Database groups detections into objects on the basis of + positional coincidence and measures average parameters of those + objects. + +\item The AP Database stores parallax and proper motion parameters for + a subset of the average objects. + +\item The AP Database stores image and filter calibration information + necessary to convert between instrumental magnitudes and calibrated + magnitudes in standard systems. + +\item The AP Database performs at least the follow queries, with + constraints on the output based on at least time ranges, magnitude + limits, error limits: + + \begin{itemize} + \item given $(RA,DEC)$ and a Radius, return all objects and/or + detections in the region. + + \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all objects and/or + detections in the region. + + \item given $(RA,DEC)$, return closest object. + + \item given object ID, return all detections. + + \item given detection, return source image data. + + \item given detection, return object. + + \item given $(RA,DEC)$, return all images overlapping coordinate. + + \item given $(RA,DEC)$ and a Radius, return all images overlapping region. + + \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping region. + + \item given detection instrumental magnitude, return derived + magnitudes based on calibration information. + + \item given a collection of detections in a filter, determine the + object average magnitude in that filter. + + \item given a collection of objects and detections, determine the + individual image zero-points. + + \item given a region, return all possible combinations of the object + or detection magnitudes $(M_1 - M_2)$. + + \item given a list of $(RA,DEC)$ entries, return all nearest objects. + + \item given a filter, telescope, or detector, return all calibration + terms and history. + + \item given a detection, return all non-detections from images which + overlapped the detection coordinates. + \end{itemize} + +\item The AP Database shall accept detection IDs of moving objects and + label the detections with the identified object. +\end{itemize} + +%% IPP AP DB Requirements +The IPP AP Database has the following performance requirements: + +\begin{enumerate} +\item The AP Database shall accept new detections at the rate + generated by the telescope from the Phase 2 and Phase 4 analysis. + \tbr{Except within 10 degrees of the galactic plane, the AP Database + shall keep up with the incoming rates.} The expected rates are + listed in Table~\ref{APrates}, along with the total data volume + required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2, TLR:3, TLR:22} + +\item The AP Database shall provide access to external Pan-STARRS + clients to the detected objects within \tbr{5 minute} after the + image is obtained.\VER{TEST}{TLR:22} + \label{IPP:DeReq:29c} +\end{enumerate} + \subsubsection{Metadata Database} +%% Table: Metadata data classes \begin{table} \begin{center} @@ -739,52 +817,114 @@ \end{table} -\begin{enumerate} -\item The IPP Metadata Database shall accept metadata from the summit - at a sustained rate of \tbr{1 MB per second}.\VER{TEST}{TLR:17, TLR:21, TLR:25} - -\item The Metadata Database shall store the classes of data listed in - Table~\ref{metadata}. Thus, the Metadata Database shall store and - provide metadata for all raw images, for processed images, for the +%% Metadata DB T & F + +The Metadata Database tasks and functions: + +\begin{itemize} +\item The Metadata Database stores the classes of data listed in + Table~\ref{metadata}. Thus, the Metadata Database stores and serves + metadata for all raw images, for processed images, for the calibration images (both raw and master), for the extracted object lists. Metadata describing the environmental conditions at the - telescope shall also be stored and provided as needed. - Database.\VER{INSPECT}{TLR:21, TLR:25} - -\item The Metadata Database queries shall have a latency of $< 0.1$ seconds.\VER{TEST}{TLR:17} - -\item The Metadata Database shall be capable of at least 100 queries per second.\VER{TEST}{TLR:17} + telescope is also stored and provided as needed. + +\item The Metadata Database responds to simple queries which return + the data in the categories listed in Table~\ref{metadata} based on + the primary data key and with basic constraints of time ranges and + other simple conditional constraints. + +\item The Metadata database stores the configuration information with + restricted access so that only specific people may change the + information (eg, science parameters available to the science team; + software configuration parameters available to the system + maintainers). +\end{itemize} + +%% Metadata DB Requirements + +The Metadata Database has the following requirements: + +\begin{enumerate} +\item The IPP Metadata Database shall accept metadata from the summit + at a sustained rate of \tbr{1 MB per 40 second.}\VER{TEST}{TLR:17, + TLR:21, TLR:25} + +\item The Metadata Database queries shall have a latency of $< 0.1$ + seconds.\VER{TEST}{TLR:17} + +\item The Metadata Database shall be capable of at least 100 queries + per second.\VER{TEST}{TLR:17} \item The Metadata Database shall be capable of accepting a total data volume after 2 years of operation of 280 GB. \VER{INSPECT}{TLR:25} - -\item The Metadata Database shall respond to simple queries which - return the data in the categories listed in Table~\ref{metadata} - based on the primary data key and with basic constraints of time - ranges and other simple conditional constraints.\VER{TEST}{TLR:17} - -\item The Metadata shall store descriptive information about the raw - images received from the summit and the current state of the data - processing.\TASK - -\item The Metadata shall also store descriptive information for each of - the static sky images currently available.\TASK - -\item Software configuration parameters shall be stored in and - extracted from the Metadata Database.\TASK - -\item The Metadata database shall store the configuration information - with restricted access so that only specific people may change the - information.\VER{TEST}{allocated} - -\item User-configurable software parameters shall be stored in and - extracted from the Metadata Database.\TASK \item The Metadata Database shall restrict write access of the scientific parameters to a different group from the software and hardware configuration parameters.\VER{TEST}{allocated} - \end{enumerate} \subsubsection{Controller} + +%% IPP Controller T & F + + IPP Controller tasks and functions: + +\begin{itemize} + +\item On startup, the IPP Controller attempts to establish + communication with all of its computers and set their state to be + {\tt alive} or {\tt dead} based on the success of the + connection. + +\item The IPP Controller detects computers which crash or stop + responding and set their state to {\tt dead}. + +\item The IPP Controller attempts to re-establish communication with + {\tt dead} computers. + +\item The IPP Controller accepts tasks from external users and + systems, which may specify a desired CPU (node) and priority in + addition to the task command. + +\item The IPP Controller attempts to run pending tasks on the desired + node, if available (not {\tt dead} or {\tt off}). + +\item If the node is unavailable, the IPP Controller attempts to run + the task on another node. + +\item If the node is available, the IPP Controller attempts to run a + given task only if no higher-priority tasks are available and no + task is currently being executed. + +\item The IPP Controller monitors the output from the task and writes + it to an associated log destination. + +\item The IPP Controller monitors the execution status of each task + currently executing on a node and performs the following actions: + + \begin{itemize} + \item identify the task as successful if it has a valid exit status. + \item identify the task as unsuccessful if it has an error exit status. + \item identify the task as unattempted if the computer crashed. + \end{itemize} + +\item The IPP Controller accepts and performs the following external + commands: + \begin{itemize} + \item add a task to the pending task list. + \item delete a specific task from the pending task list. + \item return the current status of a specific task. + \item return a list of all pending and non-pending tasks. + \item set a specified computer state to {\tt off} or {\tt dead}. + \item restrict a specified CPU to a class of tasks. + \item halt execution of a specified task. + \item set the IPP Controller state to {\tt finish}, {\tt abort}, or {\tt stop}. + \end{itemize} +\end{itemize} + +%% IPP Controller Requirements + +IPP Controller requirements: + \begin{enumerate} @@ -792,54 +932,4 @@ computers.\VER{TEST}{TLR:17} -\item On startup, the IPP Controller shall attempt to establish - communication with all of its computers and set their state to be - {\tt alive} or {\tt dead} based on the success of the connection.\VER{TEST}{TLR:17} - -\item The IPP Controller shall detect computers which crash or stop - responding and set their state to {\tt dead}.\VER{TEST}{TLR:17} - -\item The IPP Controller shall attempt to re-establish communication - with {\tt dead} computers.\VER{TEST}{TLR:17} - -\item The IPP Controller shall accept tasks from external users and - systems, which may specify a desired CPU (node) and priority in - addition to the task command.\VER{TEST}{TLR:17} - -\item The IPP Controller shall attempt to run pending tasks on the - desired node, if available (not {\tt dead} or {\tt off}).\VER{TEST}{TLR:17} - -\item If the node is unavailable, the IPP Controller shall attempt to - run the task on another node.\VER{TEST}{TLR:17} - -\item If the node is available, the IPP Controller shall attempt to run - a given task only if no higher-priority tasks are available and no - task is currently being executed.\VER{TEST}{TLR:17} - -\item The IPP Controller shall monitor the output from the task and - write it to an associated log destination.\VER{TEST}{TLR:17} - -\item The IPP Controller shall monitor the execution status of each - task currently executing on a node and perform the following - actions: - - \begin{enumerate} - \item identify the task as successful if it has a valid exit status.\VER{TEST}{TLR:17} - \item identify the task as unsuccessful if it has an error exit status.\VER{TEST}{TLR:17} - \item identify the task as unattempted if the computer crashed.\VER{TEST}{TLR:17} - \end{enumerate} - -\item The IPP Controller shall accept and perform the following - external commands: - \begin{enumerate} - \item add a task to the pending task list.\VER{TEST}{TLR:17} - \item delete a specific task from the pending task list.\VER{TEST}{TLR:17} - \item return the current status of a specific task.\VER{TEST}{TLR:17} - \item return a list of all pending and non-pending tasks.\VER{TEST}{TLR:17} - \item set a specified computer state to {\tt off} or {\tt dead}.\VER{TEST}{TLR:17} - \item restrict a specified CPU to a class of tasks.\VER{TEST}{TLR:17} - \item halt execution of a specified task.\VER{TEST}{TLR:17} - \item set the IPP Controller state to {\tt finish}, {\tt abort}, or {\tt stop}.\VER{TEST}{TLR:17} - \end{enumerate} - \item The IPP Controller shall limit command latency to \tbr{$< 0.1$} seconds.\VER{TEST}{TLR:17} @@ -856,11 +946,38 @@ \subsubsection{Scheduler} -\begin{enumerate} -\item The IPP Scheduler shall send the analysis tasks which it - initiates to the IPP Controller.\VER{TEST}{TLR:17} - -\item All analysis tasks sent by the IPP Scheduler shall include a - complete UNIX command with necessary arguments, the priority of the - task, and optionally the desired processing node.\VER{INSPECT}{TLR:17} + +%% IPP Scheduler T & F + +The IPP Scheduler tasks and functions: + +\begin{itemize} +\item The IPP Scheduler sends the analysis tasks which it initiates to + the IPP Controller. + +\item All analysis tasks sent by the IPP Scheduler include a complete + UNIX command with necessary arguments, the priority of the task, and + optionally the desired processing node. + +\item When the IPP Scheduler is placed in the {\em paused state}, it + only initiates User-requested tasks. + +\item When the IPP Scheduler is placed in the {\em interactive state}, + it initiates User-requested tasks as well as data transfer tasks. + +\item When the IPP Scheduler is placed in the {\em automatic state}, + it initiates the most appropriate task based on the inputs and + dependency rules. + +\item The IPP Scheduler sends the exit status of the analysis tasks to + the appropriate destination as defined by the task dependency table. +\end{itemize} + +%% IPP Scheduler Requirements + +The IPP Scheduler requirements: + +\begin{enumerate} +\item The IPP Scheduler shall publish the static sky images to the + Pan-STARRS PSPS on a time-scale of \tbr{6 month}.\VER{TEST}{TLR:19} \item The IPP Scheduler shall query the Databases on a regular basis @@ -869,23 +986,5 @@ \item The IPP Scheduler shall accept new User input in real-time: -within 0.1 seconds of the request.\VER{TEST}{TLR:17} - -\item When the IPP Scheduler is placed in the {\em paused state}, it - shall only initiate User-requested tasks.\VER{TEST}{TLR:17} - -\item When the IPP Scheduler is placed in the {\em interactive state}, - it shall initiate User-requested tasks as well as data transfer - tasks.\VER{TEST}{TLR:17} - -\item When the IPP Scheduler is placed in the {\em automatic state}, - it shall initiate the most appropriate task based on the inputs and - dependency rules.\VER{TEST}{TLR:17} - -\item The IPP Scheduler shall send the exit status of the analysis - tasks to the appropriate destination as defined by the task - dependency table.\VER{TEST}{TLR:17} - -\item The IPP Scheduler shall publish the static sky images to the - Pan-STARRS PSPS on a time-scale of \tbr{6 month}.\VER{TEST}{TLR:19} + within 0.1 seconds of the request.\VER{TEST}{TLR:17} \item The IPP Scheduler shall publish the detected objects to the @@ -903,21 +1002,29 @@ to the MOPS subsystem within 5 minutes of the image exposure time.\VER{TEST}{TLR:22} - -\end{enumerate} - -\subsection{Analysis Stages} - -We now consider the requirements of the analysis tasks which shall be -performed by the IPP. These tasks represent the core of the required -IPP functionality; the architectural components discussed above can be -viewed as primarily supporting infrastructure to enable the analysis -tasks to be executed on the appropriate data and to store the results. - -\subsubsection{Science Image Analysis} +\end{enumerate} + +%%%%%% Analysis Stages + +%%%% Science Image Analysis Stages + +\subsection{Science Image Analysis Stages} + +We now consider the requirements of the science image analysis tasks +which are performed by the IPP. These tasks represent the core of the +required IPP functionality; the architectural components discussed +above can be viewed as primarily supporting infrastructure to enable +the analysis tasks to be executed on the appropriate data and to store +the results. The Science Image analysis stages together represent the basic data -analysis required by the IPP. There are several requirements which -shall be met by the collection of science image analysis stages as a -group. +analysis required by the IPP. Integral to our operational concept for +the IPP is the division of the science image analysis into four +phases, each of which represents a complete analysis on a particular +unit of data. The tasks and functions of these separate stages are +discussed below. + +\subsubsection{General Science Image Analysis Requirements} +There are several requirements which shall be met by the collection of +science image analysis stages as a group. \begin{enumerate} @@ -937,52 +1044,4 @@ static sky image, and update the corresponding exposure (S/N) maps, at a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17} - -\item The IPP Science Analysis shall detect and measure parameters of -objects on the pre-processed science images.\VER{TEST}{TLR:12} - -\item The IPP Science Analysis shall detect and measure parameters of -objects on the stacked science images.\VER{TEST}{TLR:13} - -\item The IPP Science Analysis shall detect and measure parameters of -objects on the static sky images.\VER{TEST}{TLR:14} - -\item The IPP Science Analysis shall detect and measure parameters of -objects on the difference images.\VER{TEST}{TLR:15} - -\item The IPP Science Analysis shall determine astrometry of the - detected objects relative to an external astrometric reference with - an accuracy of \tbr{750 mas} (for bright objects) in the - Commissioning phase of the telescope.\VER{TEST}{TLR:4, TLR:3} - -\item The IPP Science Analysis shall determine astrometry of the - detected objects relative to an external astrometric reference with - an accuracy of \tbr{250 mas} (for bright objects) during the - construction of the Pan-STARRS Astrometric Reference Catalog.\VER{ANALYSIS}{TLR:4, TLR:3} - -\item The IPP Science Analysis shall determine astrometry of the - detected objects relative to the Pan-STARRS Astrometric Reference - with an accuracy of \tbr{100 mas} (for bright objects) during normal - operations.\VER{ANALYSIS}{TLR:4, TLR:3} - -\item The IPP Science Analysis shall determine photometry of the - detected objects within an internal photometric system with scatter - less than \tbr{25 millimags} (for bright objects) during the - Commissioning phase of the telescope in photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2} - -\item The IPP Science Analysis shall determine photometry of the - detected objects within an internal photometric system with scatter - less than \tbr{10 millimags} (for bright objects) during the - construction of the Pan-STARRS Photometric Reference Catalog in - photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2} - -\item The IPP Science Analysis shall determine photometry of the - detected objects within an internal photometric system with scatter - less than \tbr{5 millimags} (for bright objects) during normal - operations in photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2} - -\item The IPP Science Analysis shall determine photometry of the - detected objects in an external photometric system with scatter less - than \tbr{10 millimags} (for bright objects) during normal operations - in photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2} \item The maximum latency between the acquisition of an image and the @@ -997,283 +1056,155 @@ \end{enumerate} +%% Phase 1 \subsubsection{Phase 1 : image processing preparation} -\begin{enumerate} -\item the Phase 1 analysis shall execute within 2 seconds for a - complete FPA image.\VER{TEST}{TLR:17} - -\item The Phase 1 analysis stage shall determine the astrometric - solution of the complete camera (FPA image) with an accuracy of - \tbr{1 arcsec} peak-to-peak deviation.\VER{TEST}{TLR:3} - -\item The Phase 1 analysis stage shall load the guide star pixel and - celestial coordinates.\TASK - -\item If guide stars are not available, the Phase 1 analysis stage - shall extract bright stars from the image.\TASK - -\item This extraction shall be done in less than \tbr{1 second}.\VER{TEST}{TLR:17} - +Phase 1 is the image processing preparation stage. The analysis is +performed on a complete FPA. At the end of this analysis, the FPA is +ready to be analysed in detail in Phase 2. The Phase 1 tasks and +functions are: + +\begin{itemize} + +\item Extract FPA guide stars to determine astrometry across the full FPA + +\item If no guide stars are available, phase 1 must measure the pixel + coordinates of known bright stars expected in the field from the + image data. + \item The total number of stars and size of the bright-star acquisition box shall be a user-configurable parameter in the range - 20 - 250.\TASK - + 20 - 250. + +\item Calculate the Image cell / Sky cell overlaps for each image. + Sky cells which do not have sufficient science image overlap \tbr{$< + 5\%$} are excluded from the overlap table. + +\end{itemize} + +The Phase 1 requirements are: + +\begin{enumerate} +\item the Phase 1 analysis shall execute within 2 seconds for a + complete FPA image.\VER{TEST}{TLR:17, allocated} + +\item The Phase 1 analysis stage shall determine the astrometric + solution of the complete camera (FPA image) with an accuracy of 1 + arcsec peak-to-peak deviation.\VER{TEST}{TLR:3} + +\item Bright-star extraction from the image data shall be performed in + less than \tbr{1 second}.\VER{TEST}{TLR:17} + \item In order for blind astrometry of an image to succeed, it is necessary that approximate image coordinates be known. The Phase 1 - analysis shall be able to succeed despite initial coordinate errors - as large as \tbr{20\arcsec}.\VER{TEST}{TLR:3} - -\item The Phase 1 analysis stage shall construct a table of the - overlaps between the science image to be processed and the static - sky images.\TASK - -\item The overlaps shall be overestimated by a small amount so that - errors in astrometry at Phase 1 will not cause any valid static sky - / science image pairs to be missed.\TASK - -\item The amount of overlap shall be a user-configurable parameter.\VER{TEST}{TLR:6, TLR:11} + analysis shall succeed despite initial coordinate errors as large as + \tbr{20\arcsec}.\VER{TEST}{TLR:3} -\item Sky cells which do not have sufficient science image overlap - \tbr{$< 5\%$} shall be excluded from the overlap table.\VER{TEST}{TLR:6, TLR:11} - -\item It is not unusual for an image to be obtained with invalid - coordinates or without any valid stars. For example, the telescope - control system may make an error and report the wrong time or - coordinates. Or, the image may be obtained in exceptionally poor - conditions with no detected stars. Phase 1 shall return a - descriptive error message in these conditions.\TASK -\end{enumerate} - +\end{enumerate} + +%% Phase 2 \subsubsection{Phase 2 : image reduction} -The Phase~2 analysis is the detrend stage, in which the images from -the detector are processed to remove instrumental signatures. - -\paragraph{Timing} -The complete Phase~2 analysis shall be performed in $< 38$ seconds for -up to 4 complete FPA images at one time. \VER{TEST}{TLR:17} - -\paragraph{Processing Recipe} -\begin{enumerate} -\item The Phase 2 analysis stage shall consult the processing recipe - to define the necessary analysis steps performed by the Phase 2 - stage.\TASK - -\item Phase 2 shall perform the analysis steps only if required by the - processing recipe.\TASK - -\item The processing recipe shall define the stages to be executed with - optional exposure time and background flux limits to require or - exclude select certain stages.\TASK -\end{enumerate} - -\paragraph{Detrend Image Convolutions} -\begin{enumerate} - -\item The Phase 2 analysis stage shall convolve the flat-field and - high-spatial-frequency fringe images with the OT kernel.\VER{TEST}{TLR:1} - -\item The Phase 2 analysis stage shall determine the OT kernel from the - IPP Metadata Database.\TASK - -\item If no OT kernel exists, this step shall be silently skipped.\TASK -\end{enumerate} - -\paragraph{Flag bad and saturated pixels} -\begin{enumerate} - -\item The Phase 2 analysis shall load the basic bad pixel map appropriate to -the detector of interest.\VER{TEST}{TLR:18} - -\item The Phase 2 analysis shall use the OT kernel to grow the traps in the -raw bad pixel map. \VER{TEST}{TLR:18} - -\item The Phase 2 analysis shall mask saturated pixels and a user-specified -number of surrounding pixels.\VER{TEST}{TLR:18} - -\item The Phase 2 analysis shall mask ghosts of bright stars.\VER{TEST}{TLR:18} - -\item Different bits shall be set to identify different reasons for masking -the pixels.\VER{TEST}{TLR:21} -\end{enumerate} - -\paragraph{Bias correction via overscan subtraction} -\begin{enumerate} - -\item Phase 2 shall perform bias subtraction on the image.\VER{TEST}{TLR:1} - -\item Phase 2 shall choose the bias subtraction method and analysis statistic -based on the user-configured parameters.\TASK - -\item The bias correction shall be measured from the image overscan region.\TASK - -\item The overscan region shall be determined from the Metadata DB.\TASK - -\item The bias subtraction shall be capable of using one of following -bias corrections, depending on the user parameters: - -\begin{enumerate} -\item subtract a single constant from the image. \VER{TEST}{TLR:1} - -\item subtract a 1-D bias which varies along the overscan. The function to be used shall include -a spline or a Chebychev polynomial derived from the data values along -the overscan, as specified by the user parameters. \VER{TEST}{TLR:1} - -\item correct the overscan {\em and} subtract a 2-D bias image which - has been overscan corrected using one of the two methods above.\VER{TEST}{TLR:1} -\end{enumerate} - -\item The statistic used to calculate the overscan constant or the -inputs to the spline and polynomial fits shall be derived from groups -of pixels on the basis of one of several possible statistics, as -specified by the user parameters.\VER{TEST}{TLR:1} - -\item The choice of statistics shall include the sample and robust -mean, median, and modes.\VER{TEST}{TLR:1} - -\item In the case of a single constant, all of the overscan pixel -values are used in the calculation of this statistic.\VER{TEST}{TLR:1} - -\item In the case of the 1D functional representation, the input -values to the fit shall represent the coordinate along the overscan, -with the statistic derived from the pixels in the perpendicular -direction at each location.\VER{TEST}{TLR:1} - -\item If specified in the user parameters, sigma-clipping shall be -performed on the input data values.\VER{TEST}{TLR:1} - -\item The bias subtraction shall leave no residuals greater than \tbr{1 DN} -peak-to-peak.\VER{TEST}{TLR:1} -\end{enumerate} - -\paragraph{Trim object image} -\begin{enumerate} - -\item The Phase 2 analysis shall trim the non-imaging pixels from the -image.\TASK - -\item The definition of the imaging area shall be determined from the -Metadata Database.\TASK - -\item Phase 2 shall trim pixel near the edges that have been -compromised due to OT operation.\VER{TEST}{TLR:1} -\end{enumerate} - -\paragraph{Correct for non-linearity} - -If required by the recipe, each chip shall be independently corrected for the -effects of non-linearity.\VER{TEST}{TLR:1} - -\paragraph{Flat-field correction} -\begin{enumerate} - -\item The Phase 2 analysis shall divide the science image by the - provided flat-field image.\VER{TEST}{TLR:1} - -\item The division shall handle zero-valued pixels in the flat-field - image without raising floating point exceptions, setting the - corresponding bit value in the mask.\VER{TEST}{TLR:1} - -\item The flat-field images shall be appropriately normalized (see - section \ref{mkcal}).\VER{TEST}{TLR:1} +Phase 2 is the detrend stage, in which each detector is separately +processed to remove instrumental signatures. The result of Phase 2 is +an image with high-quality astrometric and photometric calibrations, a +collection of objects detected in the image and characterized in a +rudimentary way (star / non-stellar), and a measurement of the PSF +across the detector. + +The tasks and functions of Phase 2 are as follows: + +\begin{itemize} + +\item Convolve the flat-field and high-spatial-frequency fringe images + with the OT kernel. + +\item Mask ghosts of bright stars which introduce residual feature + more significant than \tbr{1\%} of the background. + +\item Bias subtract the image. + +\item Correct each chip independently for non-linearity. + +\item Flat-field correct the image. + +\item Subtract a fit to the detector-dependent fringing pattern. + +\item Subtract a fit to the low-spatial frequency sky background. + +\item Identify `cosmic rays' on the basis of morphology. + +\item Perform (positive) object detection on the processed images, + down to a user-configured threshold, likely to be $\sim 20\sigma$. + The detection threshold may optionally be a function of the average + background flux or the local noise level. + +\item Measure the following object parameters: + + \begin{itemize} + \item object centroid and position errors. + \item an extended object position ($x_g, y_g$). + \item instrumental PSF magnitude and error. + \item local background level and error. + \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) of the object + and their covariance matrix. + \end{itemize} + +\item Perform minimal object classification to distinguish objects + which are consistent with a single PSF, objects which are + inconsistently large, objects which are inconsistently small, and + objects which are saturated. + +\item Match the detected objects with known astrometric reference + objects, including proper-motion compensation. + +\item Fit the reference and detected object coordinates to determine + astrometric parameters for the individual OTAs, including + polynomials of the coordinates up to 3rd order (user-specified + parameter). The Cell astrometric parameters are not allowed to vary + in the fit, which uses outlier rejection to determine a robust + solution. + +\item Extract subrasters (`postage stamps') surrounding a + user-specified list of coordinates from the flattened + images, to be saved in the IPP Image Server. + +\item measure the PSF variation as a function of detector position. + +\end{itemize} + +The Phase 2 requirements are: + +\begin{enumerate} +\item The complete Phase~2 analysis shall be performed in $< 38$ + seconds for up to 4 complete FPA images at one + time. \VER{TEST}{TLR:17} + +\item The bias subtraction shall leave no residuals greater than + \tbr{1 DN} peak-to-peak for images within the normal range of bias + variations.\VER{TEST}{TLR:1} + +\item The Phase 2 flat-field correction shall handle zero-valued + pixels in the flat-field image without raising floating point + exceptions, setting the corresponding bit value in the + mask.\VER{TEST}{TLR:1} \item The flat-fielded image shall have a consistent photometric - zero-point across the chip, and across the full FPA, to within 0.2\% - with peak-to-peak deviations of \tbr{0.5\%}.\VER{TEST}{TLR:1} -\end{enumerate} - -\paragraph{Sky \& Fringe subtraction} -\begin{enumerate} - -\item The Phase 2 analysis shall subtract the sky (and fringe where - needed) contributions from the images.\VER{TEST}{TLR:1, TLR:5} + zero-point across the chip, and across the full FPA, with scatter $< + 0.2\%$ and peak-to-peak deviations of $< 0.5\%$.\VER{ANALYSIS}{TLR:1} \item The residual after the background subtraction shall have an average offset of 0 counts, excluding the signal from astronomical - features.\VER{TEST}{TLR:5} + features.\VER{ANALYSIS}{TLR:5} \item The background residuals shall have peak-to-peak variations of - less than \tbr{1\%} of the input background amplitude.\VER{TEST}{TLR:5} + less than \tbr{1\%} of the input background amplitude.\VER{ANALYSIS}{TLR:5} \item The background residuals shall have a scatter of less than \tbr{1\%} of the input background amplitude for apertures of less - than \tbr{10~arcsec}.\VER{TEST}{TLR:1} -\end{enumerate} - -\paragraph{Identify `cosmic rays'} -\begin{enumerate} - -\item The Phase 2 analysis shall detect cosmic rays with flux $> - 5\sigma$ by morphology in single images with an efficiency of $> 95$\%. - \VER{TEST}{TLR:18} - -\item The Phase 2 analysis shall mask detected cosmic rays with a - unique bit value in the mask.\TASK - -\item The Phase 2 analysis shall extend the masked region by a - user-configurable growth factor.\TASK - -\item The Phase 2 analysis shall perform the cosmic ray detection only - if it is required by the analysis recipe.\TASK -\end{enumerate} - -\paragraph{Find objects in the image} -\begin{enumerate} - -\item The Phase 2 analysis shall perform object detection on the - processed images.\VER{TEST}{TLR:12} - -\item The object detection process shall detect all objects above a - user-configured threshold.\TASK - -\item The threshold shall be a positive value; negative values shall - invoke an error.\TASK - -\item The detection threshold shall optionally be a function of the - average background flux or the local noise level.\TASK - -\item The object detection shall measure the following object - parameters: - \begin{enumerate} - \item object centroid and position errors\VER{TEST}{TLR:12} - \item an extended object position ($x_g, y_g$)\VER{TEST}{TLR:12} - \item instrumental PSF magnitude and error\VER{TEST}{TLR:12} - \item local background level and error\VER{TEST}{TLR:12} - \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) of the object - and their covariance matrix\VER{TEST}{TLR:12} - \end{enumerate} - -\item Minimal object classification shall be performed to distinguish - objects which are consistent with a single PSF, objects which are - inconsistently large, objects which are inconsistently small, and - objects which are saturated.\VER{TEST}{TLR:12} - -\item The resulting collection of detected objects shall be saved along - with the relevant image metadata (\ie filter, exposure time, etc).\VER{TEST}{TLR:20} -\end{enumerate} - -\paragraph{Astrometry} -\begin{enumerate} - -\item The Phase 2 analysis shall match the detected objects with known - astrometric reference objects.\VER{TEST}{TLR:3} - -\item The astrometric reference object coordinates shall be adjusted - for proper motion.\VER{TEST}{TLR:3} - -\item The reference and detected object coordinates shall be fit to - determine astrometric parameters for the individual OTAs.\VER{TEST}{TLR:3} - -\item The OTA astrometric parameters shall include polynomials of the -coordinates up to 3rd order.\VER{TEST}{TLR:3} - -\item The fitted number of polynomial orders shall be a user-configured - parameter.\TASK - -\item The Cell astrometric parameters shall not be allowed to vary in - the fit.\VER{}{} - -\item The fit shall be robust, rejecting outlier matches (either stars - with poorly determined proper motion or spurious matches).\VER{TEST}{TLR:3} + than \tbr{10~arcsec}.\VER{ANALYSIS}{TLR:1} + +\item The Phase 2 analysis shall detect cosmic rays with flux $> 5\sigma$ by + morphology in single images with an efficiency of $> 95$\% for + images which are not undersampled. \VER{TEST}{TLR:18} \item The resulting astrometric solution shall be consistent across the @@ -1281,230 +1212,215 @@ \end{enumerate} -\paragraph{Postage Stamps} -\begin{enumerate} - -\item The Phase 2 analysis shall extract subrasters (`postage stamps') - surrounding a user-specified list of coordinates from the flattened - images.\VER{TEST}{TLR:12} - -\item The postage stamp images shall be saved in the IPP Image Server.\VER{TEST}{TLR:12} -\end{enumerate} - +%% Phase 3 \subsubsection{Phase 3 : exposure analysis} -\begin{enumerate} - -\item The Phase 3 analysis shall use the objects detected in Phase 2, - matched with a user-specified reference photometry catalog, to - determine the image photometric zero point and zero-point variations - across the field.\VER{TEST}{??} + +The Phase 3 analysis uses the objects detected in Phase 2 and external +reference catalogs to determine improved photometric and astrometric +calibrations for the FPA as a whole, and to improve the measurement of +the PSF and sky variations across the field. The Phase 3 tasks and +functions are as follows: + +\begin{itemize} + +\item Phase 3 uses the objects detected in Phase 2, matched with a + user-specified reference photometry catalog, to determine the image + photometric zero point and zero-point variations across the field. \item If zero-point variations are significant (\tbr{$> 0.01$ mag - peak-to-peak}), the zero-point variations shall be modeled with a - polynomial correction of order 3 or less.\VER{TEST}{TLR:1} - -\item The photometric nature of the FPA image shall be categorized on - the basis of the zero-point consistency, the transparency compared - with recent long-term measurements in the filter, and the external - indicators of photometricity.\VER{TEST}{TLR:2} - -\item The Phase 3 analysis shall use the objects detected in Phase 2, - matched with an appropriate astrometric reference catalog, to - improve the distortion model used for the image.\VER{TEST}{TLR:3} - -\item The resulting astrometric accuracy shall be consistent across -the field to 30 mas.\VER{TEST}{TLR:4} - -\item The resulting astrometric accuracy shall be limited by the + peak-to-peak}), the zero-point variations are modeled with a + polynomial correction of order 3 or less. + +\item The photometric nature of the FPA image is categorized on the + basis of the zero-point consistency, the transparency compared with + recent long-term measurements in the filter, and the external + indicators of photometricity. + +\item Phase 3 uses the objects detected in Phase 2, matched with an + appropriate astrometric reference catalog, to improve the distortion + model used for the image. The resulting astrometric accuracy is + consistent across the field to 30 mas, and is limited by the astrometric reference catalog, (eg, 100 - 250 mas for - USNO-B1.0).\VER{TEST}{TLR:3} - -\item The Phase 3 analysis shall modify the background correction of -Phase 2 based on the full-field statistics to achieve an accuracy of 1\% -of the background.\VER{TEST}{TLR:5} + USNO-B1.0). + +\item The Phase 3 analysis modifies the background correction of Phase + 2 based on the full-field statistics to achieve an accuracy of 1\% + of the background. + +\end{itemize} + +The Phase 3 requirements are: + +\begin{enumerate} \item The complete Phase~3 analysis shall be performed in $< 2$ seconds for up to 4 complete FPA images at one time. \VER{TEST}{TLR:17} -\end{enumerate} - +\item For images obtained under normal observing conditions, the + resulting astrometric solution shall have a residual scatter of $< + 30$ milliarcseconds when calibrated with the AP Survey reference + catalog and $< 100$ milliarcseconds when calibrated with the USNO-B + catalog.\VER{ANALYSIS}{TLR:} + +\item For images obtained under normal observing conditions, the + resulting astrometric solution shall have a precision relative to + ICRS of better than 100 milliarcseconds.\VER{ANALYSIS}{TLR:} + +\item For images obtained under photometric conditions or minimal + cirrus conditions ($< 0.1$ mag total extinction), the resulting + photometric calibration shall have a relative accuracy of 5 + millimagnitudes.\VER{ANALYSIS}{TLR:} + +\item For images obtained under photometric conditions or minimal + cirrus conditions ($< 0.1$ mag total extinction), the resulting + photometric calibration shall have an absolution photometric + accuracy of 10 millimagnitudes when calibrated relative to the AP + Survey reference catalog.\VER{ANALYSIS}{TLR:} + +\item For images obtained under photometric conditions or minimal + cirrus conditions ($< 0.1$ mag total extinction) and under the moon + conditions listed in Table~\ref{moonconditions}, the resulting sky + background subtraction shall leave behind peak-to-peak residuals $< + 1$\% of the input sky flux.\VER{ANALYSIS}{TLR:} + +\end{enumerate} + +%% Phase 4 \subsubsection{Phase 4 : image combination} Phase 4 is the image combination stage, in which multiple images of the same portion of the sky are merged and confronted with the static -sky image. Requirements for the different steps of the process are -given below. - -\paragraph{Extract image pixels} -\begin{enumerate} - -\item The Phase 4 analysis shall determine the corresponding set of - image pixels for a given sky cell.\TASK - -\item The corresponding image pixels shall be extracted from the input - images, using the astrometric information for each OTA and Cell to - determine the exact overlaps.\TASK - +sky image. The Phase 4 tasks and functions are as follows: + +\begin{itemize} + +\item The Phase 4 analysis determines the corresponding set of image + pixels for a given sky cell. + +\item These pixels are extracted from the input images, using the + astrometric information for each OTA and Cell to determine the exact + overlaps. + +\item The Phase 4 analysis skips any sky cells with fewer than 5\% of + their pixels overlapping the input images. + +\item Pixels which have been extracted from the input images are + geometrically warped to match the corresponding pixels in the sky + image. This transformation is based on the measured astrometric + solution for the input images relative to the reference catalog used + to generate the static sky image. The warping may use a + locally-linear astrometric solution to speed the processing. + +\item Phase 4 determines the appropriate photometry scaling factors + needed to combine the images photometrically. + +\item When multiple images are combined, the group of input pixels + which contribute to an output pixel are examined and pixels from the + group of images which are inconsistent with the ensemble (by an + amount defined by the user-configurable parameters) are identified + and flagged, though this outlier rejection shall be performed + optionally. + +\item The resulting collection of pixels is used to construct a single + output image, cleaned of the outliers. + +\item The cleaned, combined image is PSF matched with the static sky + image. + +\item The static sky image is subtracted from the stacked, cleaned + image, resulting in the difference image (P4$\Delta$ image) + +\item The Phase 4 analysis performs object detection on the difference + images. All objects in the difference image above a user-configured + signficance threshold are detected, including both positive and + negative flux objects. The detection threshold may optionally be a + function of the average background flux or the local noise + level. The likely significance threshold is $\sim 3\sigma$. + +\item P4$\Delta$ objects have the following object parameters + measured: + \begin{itemize} + \item object centroid and position errors. + \item instrumental PSF magnitude and error. + \item local background level and error. + \item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$. + \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix. + \end{itemize} + +\item Minimal object classification is performed to distinguish + objects which are consistent with a single PSF, objects which are + inconsistent, and objects which are saturated. + +\item The pixels belonging to variable sources are masked in the + input image. + +\item A new, cleaned image is constructed from the masked input images + (P4$\Sigma$ image) + +\item Object detection is performed on the cleaned, summed image to a + user-configured significance threshold ($\sim 7\sigma$). Only + positive flux object are considered. The detection threshold may + optionally be a function of the average background flux or the local + noise level. + +\item P4$\Sigma$ objects have the following object parameters + measured: + \begin{itemize} + \item object centroid and position errors. + \item an extended object position ($x_g, y_g$). + \item instrumental PSF magnitude and error. + \item local background level and error. + \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their + covariance matrix. + \item the Petrosian radius, magnitude, axis ratio, and angle. + \item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$. + \end{itemize} + +\item Minimal object classification is performed to distinguish + objects which are consistent with a single PSF, objects which are + inconsistent, and objects which are saturated. + +\item Before the image is added to the static sky, it must pass Q/A + tests: + \begin{itemize} + \item the measured photometry scatter for the image must be less + than \tbr{1\%}. + + \item the measured astrometry scatter for the image must be less + than \tbr{30 mas}. + \end{itemize} + +\item The final, cleaned input image is added to the static sky so + that an incrementally-deeper static sky image may be + made. +\end{itemize} + +The Phase 4 requirements are: + +\begin{enumerate} \item The Phase 4 analysis shall not miss any pixels in this match, and it shall read no more than 20\% more pixels than necessary from the input images.\VER{TEST}{TLR:17} -\item The Phase 4 analysis shall skip any sky cells with fewer than 5\% - of their pixels overlapping the input images.\VER{TEST}{TLR:17} -\end{enumerate} - -\paragraph{Transform pixel coordinates} -\begin{enumerate} - -\item Pixels which have been extracted from the input images shall be - mapped to the corresponding pixels in the sky image.\TASK - -\item The transformation shall be based on the measured astrometric - solution for the input images relative to the reference catalog used - to generate the static sky image.\VER{TEST}{TLR:3} - -\item This warping shall use a locally-linear astrometric solution.\VER{TEST}{TLR:17} - -\item The output image shall maintain photometric consistency with the - input image to within 0.2\%.\VER{TEST}{TLR:1} -\end{enumerate} - -\paragraph{Flux matching} - -The Phase 4 analysis shall determine appropriate photometry scaling -factors needed to combine the images photometrically.\TASK - -\paragraph{Image outlier pixel rejection} -\begin{enumerate} - -\item When multiple images are combined, the group of input pixels - which contribute to an output pixel shall be examined and pixels from - the group of images which are inconsistent with the ensemble (by an - amount defined by the user-configurable parameters) shall be - identified and flagged.\VER{TEST}{TLR:18} - -\item This outlier rejection shall be performed optionally.\TASK - -\end{enumerate} - -\paragraph{Initial cleaned image} - -The resulting collection of pixels shall be used to construct a single -output image, cleaned of the outliers.\VER{TEST}{TLR:18} - -\paragraph{PSF matching} - -The cleaned, combined image shall be PSF matched with the static sky image.\VER{TEST}{TLR:15} - -\paragraph{Image Subtraction} - -The static sky image shall be subtracted from the stacked, cleaned -image. \VER{TEST}{TLR:15} - -\paragraph{Find objects in the image} -\begin{enumerate} - -\item The Phase 4 analysis shall perform object detection on the - difference images.\VER{TEST}{TLR:15} - -\item All objects in the difference image shall be detected and the - pixels belonging to variable sources flagged in the input image.\VER{TEST}{TLR:15} - -\item The object detection shall detect all objects above a - user-configured threshold.\VER{TEST}{TLR:15} - -\item Both positive and negative objects shall be detected: the - specified threshold shall define the absolute value of the detection - thresholds.\VER{TEST}{TLR:15} - -\item The detection threshold shall optionally be a function of the - average background flux or the local noise level.\VER{TEST}{TLR:15} - -\item The object detection shall measure the following object parameters: - \begin{enumerate} - \item object centroid and position errors\VER{TEST}{TLR:15} - \item instrumental PSF magnitude and error\VER{TEST}{TLR:15} - \item local background level and error\VER{TEST}{TLR:15} - \item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$\VER{TEST}{TLR:15} - \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix\VER{TEST}{TLR:15} - \end{enumerate} - -\item Minimal object classification shall be performed to distinguish - objects which are consistent with a single PSF, objects which are - inconsistent, and objects which are saturated.\VER{TEST}{TLR:15, TLR:18} - -\item The resulting collection of detected objects shall be saved along - with the relevant image metadata (\ie filter, exposure time, etc).\VER{TEST}{TLR:22} -\end{enumerate} - -\paragraph{Cleaned Input Image} -\begin{enumerate} - -\item The pixels flagged as being from the difference image sources - shall be masked in the input images.\VER{TEST}{TLR:6, TLR:11} - -\item A new, cleaned image shall be constructed from the masked input - images.\VER{TEST}{TLR:6, TLR:11} - -\end{enumerate} - -\paragraph{Find objects in the image} -\begin{enumerate} - -\item The Phase 4 analysis shall perform object detection on the - cleaned, summed image.\VER{TEST}{TLR:13} - -\item The object detection shall detect all objects above a - user-configured threshold.\VER{TEST}{TLR:13} - -\item The threshold shall be a positive value; negative values shall - invoke an error.\VER{TEST}{TLR:13} - -\item The detection threshold optionally shall be a function of the - average background flux or the local noise level.\VER{TEST}{TLR:13} - -\item The object detection shall measure the following object parameters: - \begin{enumerate} - \item object centroid and position errors\VER{TEST}{TLR:13} - \item an extended object position ($x_g, y_g$)\VER{TEST}{TLR:13} - \item instrumental PSF magnitude and error\VER{TEST}{TLR:13} - \item local background level and error\VER{TEST}{TLR:13} - \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their - covariance matrix\VER{TEST}{TLR:13} - \item the Petrosian radius, magnitude, axis ratio, and angle\VER{TEST}{TLR:13} - \item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$.\VER{TEST}{TLR:13} - \end{enumerate} - -\item Minimal object classification shall be performed to distinguish - objects which are consistent with a single PSF, objects which are - inconsistent, and objects which are saturated.\VER{TEST}{TLR:13} - -\item The resulting collection of detected objects shall be saved along - with the relevant image metadata (\ie filter, exposure time, etc).\VER{TEST}{TLR:20} -\end{enumerate} - -\paragraph{Image Processing Q/A} - -Before the image is added to the static sky, it shall pass Q/A tests: - -\begin{enumerate} -\item the measured photometry scatter for the image shall be less than - \tbr{1\%}.\VER{TEST}{TLR:1} - -\item the measured astrometry scatter for the image shall be less than - \tbr{30 mas}.\VER{TEST}{TLR:3} - -\end{enumerate} - -\paragraph{Update static sky} - -The final, cleaned input image shall be added to the static sky so that -an incrementally-deeper static sky image may be made.\VER{TEST}{TLR:6, TLR:11} - -\paragraph{Timing} -The complete Phase~4 analysis shall be performed in $< 38$ seconds for -up to 4 complete FPA images at one time. \VER{TEST}{TLR:17} - -\subsubsection{Calibration Stages} +\item The warped images shall maintain photometric consistency with + the input image to within 0.2\%.\VER{TEST}{TLR:1} + +\item The sky representation shall degrade the image quality by less + than 10 milliarcseconds added in quadrature to the input image + quality.\VER{TEST}{TLR:1} + +\item The complete Phase~4 analysis shall be performed in $< 38$ + seconds for up to 4 complete FPA images at one + time. \VER{TEST}{TLR:17} + +\item completeness? + +\item contamination? + +\end{enumerate} + +\subsection{Calibration Stages} \label{mkcal} + +\subsubsection{General Calibration Construction Requirements} The Calibration analysis stages construct the various types of @@ -1521,393 +1437,227 @@ \end{enumerate} -Requirements for each of the individual calibration analysis stages -are discussed in detail below. - -\paragraph{bias images} -\begin{enumerate} - -\item The \code{bias} calibration stage shall construct a master bias - image from a collection of raw bias images.\TASK - -\item The \code{bias} calibration stage shall correct the input images - based on the overscan region.\TASK - -\item The \code{bias} calibration stage shall combine the input images - using the statistic specified by the user, selected from one of the +\subsubsection{Bias Image Creation} + +The Bias calibration stage constructs a master bias image from a +collection of raw bias images. The tasks and functions include: + +\begin{itemize} + +\item The Bias calibration stage corrects the input images based on + the overscan region, determined from either the header or from + metadata. + +\item The Bias calibration stage combines the input images using the + statistic specified by the user, selected from one of the following: + sample mean, median, and mode, robust mean, median, and mode, and + the clipped mean and median. + +\item The Bias calibration stage construct residual images, in which + the master bias is applied to the input images. + +\item Outlier residual images, those for which the residual bias and + variance in the bias image are excessive, are excluded from the + input image stack and the bias image reconstructed. +\end{itemize} + +\subsubsection{Dark Image Creation} + +The Dark calibration stage shall construct a master dark image from a + collection of raw dark images. The tasks and functions include: + +\begin{itemize} + +\item The Dark calibration stage raises an error if the input images + have exposure times which deviate by more than + \tbr{2\%}. + +\item The Dark calibration stage corrects the input dark images for + the bias. + +\item The Dark calibration stage combines the input images using the + statistic specified by the user, selected from one of the following: + sample mean, median, and mode, robust mean, median, and mode, and + the clipped mean and median. + +\item The Dark calibration stage constructs residual images, in which + the master dark is applied to the input images. + +\item Outlier residual images, those for which the residual level and + variance are excessive, are excluded from the input image stack and + the dark image reconstructed. +\end{itemize} + +\subsubsection{Flat-field Image Creation} + +The Flat-field calibration stage constructs a master flat-field image +from a collection of raw flat-field images. The tasks and functions +include: + +\begin{itemize} + +\item The Flat-field calibration stage accepts a group of images from + one of the following flat-field sources: dome, twilight, + night-sky. + +\item The flat-field calibration stage raises an error if the + input images in a single stack used more than one of the above + flat-field sources or multiple filters. + +\item The Flat-field calibration stage corrects the input flat-field + images for the bias and dark. + +\item The Flat-field calibration stage combines the input images using + the statistic specified by the user, selected from one of the following: sample mean, median, and mode, robust mean, median, and - mode, and the clipped mean and median.\TASK - -\item The \code{bias} calibration stage shall construct residual - images, in which the master bias is applied to the input images.\TASK - -\item Outlier residual images, those for which the residual bias and - variance in the bias image are excessive ($> 1DN$), shall be excluded - from the input image stack the the bias image reconstructed.\VER{TEST}{TLR:1} -\end{enumerate} - -\paragraph{dark images} -\begin{enumerate} - -\item The \code{dark} calibration stage shall construct a master dark - image from a collection of raw dark images.\TASK - -\item The \code{dark} calibration stage shall raise an error if the - input images have exposure time which deviate by more than - \tbr{2\%}.\VER{TEST}{TLR:1} - -\item The \code{dark} calibration stage shall correct the input dark - images for the bias.\TASK - -\item The \code{dark} calibration stage shall combine the input images - using the statistic specified by the user, selected from one of the - following: sample mean, median, and mode, robust mean, median, and - mode, and the clipped mean and median.\VER{TEST}{TLR:1} - -\item The \code{dark} calibration stage shall construct residual - images, in which the master dark is applied to the input images.\TASK - -\item Outlier residual images, those for which the residual level and - variance are excessive ($> 1DN$), shall be excluded from the input - image stack the the dark image reconstructed.\VER{TEST}{TLR:1} -\end{enumerate} - -\paragraph{flat-field images} -\begin{enumerate} - -\item The \code{flat-field} calibration stage shall construct a master - flat-field image from a collection of raw flat-field images.\VER{TEST}{TLR:1} - -\item The \code{flat-field} calibration stage shall accept a group of - images from one of the following flat-field sources: dome, twilight, - night-sky.\VER{TEST}{TLR:1} - -\item The \code{flat-field} calibration stage shall raise an error if - the input images in a single stack used more than one of the above - flat-field sources or multiple filters.\TASK - -\item The \code{flat-field} calibration stage shall correct the input - flat-field images for the bias and dark.\TASK - -\item The \code{flat-field} calibration stage shall combine the input - images using the statistic specified by the user, selected from one - of the following: sample mean, median, and mode, robust mean, - median, and mode, and the clipped mean and median.\VER{TEST}{TLR:1} - -\item The \code{flat-field} calibration stage shall construct residual - images, in which the master flat-field is applied to the input - images.\TASK + mode, and the clipped mean and median. + +\item The Flat-field calibration stage constructs residual images, in + which the master flat-field is applied to the input images. \item Outlier residual images, those for which the residual level and variance are excessive ($> 0.1$\%, or 1.02 times the Poisson limit - of the flat-field image), shall be excluded from the input image - stack the the flat-field image reconstructed.\VER{TEST}{TLR:1} -\end{enumerate} - -\paragraph{mask images} -\begin{enumerate} - -\item The \code{mask} calibration stage shall construct a bad-pixel - mask from a stack of raw flat-field images and a master flat-field - image.\VER{TEST}{TLR:1} - -\item The \code{mask} calibration stage shall mask the pixels which are + of the flat-field image), are excluded from the input image stack + and the flat-field image reconstructed. +\end{itemize} + +\subsubsection{Mask Image Creation} + +The Mask calibration stage constructs a bad-pixel mask from a stack of +raw flat-field images and a master flat-field image. The tasks and +functions include: + +\begin{itemize} + +\item The Mask calibration stage masks the pixels which are inconsistent in the input flats by more than \tbr{1\%}, given - sufficient signal-to-noise in the input flats.\VER{TEST}{TLR:1} - -\item The \code{mask} calibration stage shall mask the pixels which are + sufficient signal-to-noise in the input flats. + +\item The Mask calibration stage mask the pixels which are consistently low or high in the input flats by more than a factor of - \tbr{3} beyond the typical pixel.\VER{TEST}{TLR:1} - -\item The \code{mask} calibration stage shall mask the pixels - identified in a table of bad pixels generated externally to the - calibration stage.\TASK - -\item The \code{mask} calibration stage shall use multiple bit values - to identify the different types of masked pixels.\TASK - -\item The \code{mask} calibration stage shall raise an error if the - input images generate too many bad pixels in the mask.\TASK -\end{enumerate} - -\paragraph{fringe frames} -\begin{enumerate} - -\item The \code{fringe} calibration stage shall construct a master fringe + \tbr{3} beyond the typical pixel. + +\item The Mask calibration stage masks the pixels identified in a + table of bad pixels generated externally to the calibration stage. + +\item The Mask calibration stage uses multiple bit values to identify + the different types of masked pixels. + +\item The Mask calibration stage raises an error if the input images + generate too many bad pixels in the mask. +\end{itemize} + +\subsubsection{Fringe-frame Creation} + +The Fringe-frame Creation calibration stage constructs a master fringe frame from a stack of raw night-time sky images or from a stack of -dome fringe frames.\VER{TEST}{TLR:1, TLR:5} - -\item The \code{fringe} calibration stage shall raise an error if the input -stack consists is images generated with more than one type of fringe -source or with multiple filters.\TASK - -\item The \code{fringe} calibration stage shall flatten the input images -to remove the pixel-to-pixel sensitivity variations of the detector.\VER{TEST}{TLR:1} - -\item The \code{fringe} calibration stage shall measure the fringe amplitude -on the input fringe images.\TASK - -\item The \code{fringe} calibration stage shall scale the input fringe images -based on the fringe amplitude.\TASK - -\item The \code{fringe} calibration stage shall combine the scaled input -images using the statistic specified by the user, selected from one of -the following: sample mean, median, and mode, robust mean, median, and -mode, and the clipped mean and median.\VER{TEST}{TLR:5} - -\item The \code{fringe} calibration stage shall construct residual images, in -which the master fringe image is applied to the input images, along -with all necessary preceding calibration images.\TASK - -\item The \code{fringe} calibration stage shall measure the residual fringe -amplitude on the residual images.\TASK -\end{enumerate} - -\paragraph{low-spatial-frequency sky models} - -The \code{sky model} calibration stage shall construct a sky model -image from a stack of raw night-time sky images.\VER{TEST}{TLR:5} - -\paragraph{Flat-field correction frame} -\begin{enumerate} - -\item The \code{flat-field correction} calibration stage shall construct a -flat-field correction image from dithered observations of a stellar -field.\VER{TEST}{TLR:1} - -\item The \code{flat-field correction} calibration stage shall construct a -flat-field correction image for each filter and camera independently.\TASK - -\item The \code{flat-field correction} calibration stage shall construct a -correction image which makes the photometry of multiple observations -of the same stellar source consistent at different locations in the -focal plane.\VER{TEST}{TLR:1} - -\item The \code{flat-field correction} calibration stage shall construct -corrected flat-field images using the measured correction.\VER{TEST}{TLR:1} - -\item The \code{flat-field correction} calibration stage shall determine the -consistency of the corrected flat-field images using the dithered -stellar field observations flattened with the corrected flat-field -image.\TASK -\end{enumerate} - -\paragraph{Non-linearity correction frames} -\begin{enumerate} - -\item The \code{non-linear correction} calibration stage shall construct a -non-linear correction from a collection of images of a constant source -with varying exposure times.\VER{TEST}{TLR:1} - -\item The \code{non-linear correction} calibration stage shall construct a -non-linear correction which linearizes the detector fluxes $<0.5\%$.\VER{TEST}{TLR:1} - -\item The \code{non-linear correction} calibration stage shall determine the -saturation regime, in which the non-linear correction is no longer -consistent to $<0.5\%$.\VER{TEST}{TLR:1} -\end{enumerate} - -\paragraph{Telescope Astrometry Parameters} - -\begin{enumerate} -\item The IPP Calibration system shall construct static models of the +dome fringe frames. The tasks and functions include: + +\begin{itemize} + +\item The Fringe-frame Creation calibration stage raises an error if + the input stack consists is images generated with more than one type + of fringe source or with multiple filters. + +\item The Fringe-frame Creation calibration stage flattens the input + images to remove the pixel-to-pixel sensitivity variations of the + detector. + +\item The Fringe-frame Creation calibration stage measures the fringe + amplitude on the input fringe images. + +\item The Fringe-frame Creation calibration stage scales the input + fringe images based on the fringe amplitude. + +\item The Fringe-frame Creation calibration stage combines the scaled + input images using the statistic specified by the user, selected + from one of the following: sample mean, median, and mode, robust + mean, median, and mode, and the clipped mean and median. + +\item The Fringe-frame Creation calibration stage constructs residual + images, in which the master fringe image is applied to the input + images, along with all necessary preceding calibration images. + +\item The Fringe-frame Creation calibration stage measures the + residual fringe amplitude on the residual images. +\end{itemize} + +\subsubsection{Low-spatial-frequency Sky Models} + +The Sky Model calibration stage constructs a sky model image set from +a stack of raw night-time sky images. + +\subsubsection{Flat-field correction Frame Creation} + +The Flat-field correction calibration stage constructs a flat-field +correction image from dithered observations of a stellar field. The +tasks and functions include: + +\begin{itemize} + +\item The Flat-field correction calibration stage constructs a + flat-field correction image from dithered observations of a stellar + field. + +\item The Flat-field correction calibration stage constructs a + flat-field correction image for each filter and camera + independently. + +\item The Flat-field correction calibration stage constructs a + correction image which makes the photometry of multiple observations + of the same stellar source consistent at different locations in the + focal plane. + +\item The Flat-field correction calibration stage constructs corrected + flat-field images using the measured correction. + +\item The Flat-field correction calibration stage determines the + consistency of the corrected flat-field images using the dithered + stellar field observations flattened with the corrected flat-field + image. +\end{itemize} + +\subsubsection{Non-linearity correction} + +The Non-linear correction calibration stage constructs a correction +model for low-level non-linearity effects in the detector. The tasks +and functions include: + +\begin{itemize} + +\item The Non-linear correction calibration stage constructs a + non-linear correction from a collection of images of a constant + source with varying exposure times. + +\item The Non-linear correction calibration stage construct a + non-linear correction which linearizes the detector fluxes + $<0.5\%$. + +\item The Non-linear correction calibration stage determines the + saturation regime, in which the non-linear correction is no longer + consistent to $<0.5\%$. +\end{itemize} + +\subsubsection{Telescope Astrometry Parameters} + +\begin{itemize} +\item The IPP Calibration system constructs static models of the telescope astrometry parameters (e.g., distortion, detector warps) - once per week.\VER{INSPECT}{TLR:4} - -\item The IPP Calibration system shall construct static models of the + once per week. + +\item The IPP Calibration system constructs static models of the telescope astrometry parameters (e.g., distortion, detector warps) with an accuracy to produce astrometry consistent to 30 - milliarcsec.\VER{TEST}{TLR:4} - -\item The IPP Calibration system shall monitor changes in the - telescope astrometry parameters and issue a warning if the - parameters change by more than \tbr{2\%}.\VER{INSPECT}{TLR:4} -\end{enumerate} - -\paragraph{Zero-Point Monitoring} - -The IPP Calibration system shall determine telescope filter and camera -zero-points on a \tbd{timescale} with an accuracy sufficient to + milliarcsec. + +\item The IPP Calibration system monitors changes in the telescope + astrometry parameters and issue a warning if the parameters change + by more than \tbr{2\%}. +\end{itemize} + +\subsubsection{Zero-Point Monitoring} + +The IPP Calibration system determines telescope filter and camera +zero-points on a nightly basis with an accuracy sufficient to determine photometry in the native filter systems to 5 millimags. - -\subsubsection{Reference Catalog Creation} - -For PS-1, one of the primary goals is the creation of photometric and -astrometric reference catalogs for the general community and for the -future Pan-STARRS calibration. The generation of these catalogs is -inherently a research project, and will require human control and -intervention. The IPP shall provide the data access, manipulation and -visualization tools needed to construct these reference catalogs and -to assess their quality. In this section, we list the requirements of -the tools needed for this effort. - -\paragraph{Astrometry Reference Creation} - -\begin{table} -\begin{center} -\caption{Astrometric Reference Catalogs\label{AstroRefs}} -\begin{tabular}{lrrrrl} -\hline -\hline -Name & scatter limit & proper & depth & Nstars & filters \\ - & (milli-arcsec) & motion? &(mag) & (millions) & \\ -\hline -Hipparcos & 1 & 2 & 7.3 & 0.1 & V \\ -Tycho2 & 10 & 1 & 11.5 & 2.5 & B,V \\ -UCAC-2 & 20 & 1 & 16.0 & 48.0 & R \\ -USNO-A2.0 & 250 & N/A & 19.0? & 526.2 & B,R \\ -USNO-B1.0 & 200 & 20? & 21.0 & 1042.6 & B,R \\ -2MASS & 70 & N/A & 15.0? & 470.0 & J,H,K \\ -\hline -\end{tabular} -\end{center} -\end{table} - -The IPP will generate an astrometric reference on the basis of the -observations obtained by the AP survey. The IPP shall provide the -analysis tools needed to generate the master astrometric reference -catalog. Much of the required functionality is covered by the AP -Database. The specific requirements for the Astrometric Reference -creation are listed below: - -\begin{enumerate} -\item The IPP Reference Creation System shall produce an astrometric - reference catalog from the extracted objects within 6 months of the - end of the AP Survey.\VER{TEST}{TLR:3, TLR:4} - -\item The IPP Reference Creation System shall produce an astrometric - reference catalog with an absolute accuracy of \tbr{100 mas} and a - local relative accuracy of \tbr{30 mas} for bright objects.\VER{TEST}{TLR:3} - -\item The IPP Reference Creation System shall produce an astrometric - reference catalog with proper motions measurements for - non-solar-system objects with an accuracy of \tbr{20 mas / year} for - unsaturated, bright stars.\VER{TEST}{TLR:3} - -\item The Astrometry Reference creation tools shall return the match between -stars observed with Pan-STARRS and any of several astrometric -reference catalogs listed in Table~\ref{AstroRefs}.\TASK - -\item The tools shall convert the reference catalog object coordinates to all -of the coordinate frames of relevance in the telescope and camera -system: -\begin{enumerate} -\item Catalog to mean positions\VER{TEST}{TLR:3} -\item Mean to apparent positions\VER{TEST}{TLR:3} -\item Apparent positions + pointing to tangent plane coordinates\VER{TEST}{TLR:3} -\item Apparent positions + pointing to focal plane coordinates\VER{TEST}{TLR:3} -\item focal plane to specific detector (OTA)\VER{TEST}{TLR:3} -\item specific detector to detector cell\VER{TEST}{TLR:3} -\end{enumerate} - -\item The tools shall provide the necessary calibration data: the telescope -and camera optical distortion models and the variation of the image -PSF across the camera field, as a function of color.\TASK - -\item The tools shall fit the observed stellar coordinates to the astrometric -reference catalog coordinates to determine improved astrometric -solutions for both the stars and the detectors. \TASK - -\item The tools shall determine improved telescope optical distortion models -based on the astrometric solutions. \VER{TEST}{TLR:3} - -\item The tools shall optionally determine the fit coefficients as a function -of position along, or with combinations of magnitude or color. \VER{TEST}{TLR:3} - -\item The fitting method shall include robust outlier rejection. \VER{TEST}{TLR:3} - -\item The tools shall identify objects which are detected in the catalog, but -not the science image or vice-versa.\TASK - -\item The tools shall determine average centroiding errors for each object.\TASK - -\item The tools shall plot the fit residuals against a wide variety of -parameters: the object positions, magnitudes, colors, etc.\TASK - -\item The tools shall allow the fit to exclude subsets of objects from the -fits on the basis of these parameters.\TASK - -\item The tools shall provide coordinates of the guide stars in the -same frame of reference as the normal image data to within 30 -mas.\VER{TEST}{TLR:3} - -\item The tools shall perform the various fitting steps for the guide stars -rather than for the normal image data.\TASK -\end{enumerate} - -\paragraph{Photometry Reference Creation} - -\begin{table} -\begin{center} -\caption{Photometric Reference Catalogs\label{PhotoRefs}} -\begin{tabular}{lrrr} -\hline -\hline -Name & scatter & depth & filters \\ - & mmag & mag & \\ -\hline -SDSS & 15? & 16? & {\em u,g,r,i,z} \\ -CFHT-LS & 10? & 18 & {\em u,g,r,i,z} \\ -Landolt & 10-20 & 15 & {\em U,B,V,R,I} \\ -\hline -\end{tabular} -\end{center} -\end{table} - -The IPP will generate a photometric reference catalog on the basis of -the observations obtained by the AP survey. The IPP shall provide the -analysis tools needed to generate a master photometric reference -catalog. Much of the required functionality is covered by the AP -Database. The specific requirements for the Photometric Reference -creation are listed below: - -\begin{enumerate} -\item The IPP Reference Creation System shall produce a photometric - reference catalog from the extracted point-source objects within 6 - months of the end of the AP Survey.\VER{TEST}{TLR:1} - -\item The IPP Reference Creation System shall produce a photometric - reference catalog with a consistency across the sky of \tbr{5 - millimag}.\VER{TEST}{TLR:1} - -\item The IPP Reference Creation System shall produce a photometric - reference catalog with an absolute calibration to the external - system (defined by \tbr{SDSS} and the CFHT Legacy Survey Standards) - with an accuracy of \tbr{10 millimag} (for bright objects).\VER{TEST}{TLR:1} - -\item The Photometry Reference creation tools shall return the match between -stars observed with Pan-STARRS and any of several photometric -reference catalogs listed in Table~\ref{PhotoRefs}.\TASK - -\item The tools shall convert between different photometric systems, including: -\begin{enumerate} -\item instrumental to nominal detector magnitude\VER{TEST}{TLR:1} -\item nominal detector magnitude to average filter system\VER{TEST}{TLR:1} -\item average filter system to reference photometry system\VER{TEST}{TLR:1} -\end{enumerate} - -\item These transformations shall account for color and airmass terms. \VER{TEST}{TLR:1} - -\item The tools shall measure and apply relative photometry corrections -between images.\VER{TEST}{TLR:1} - -\item The tools shall determine photometric transformation fit coefficients -as a function of airmass, magnitude, color and detector coordinates, -or with combinations of the above.\TASK - -\item The fitting method shall include robust outlier rejection.\VER{TEST}{TLR:1} - -\item The tools shall reject specific objects from the fit on the basis of -object locations, instrumental magnitudes, observed and reference -errors, and in particular time of the observations. \TASK - -\item The tools shall plot the fit residuals against a wide variety of -parameters, including the object positions, magnitudes, colors, etc.\TASK - -\item The tools shall provide photometry from the guide stars in the same -system as observations of stars from the normal imaging data.\VER{TEST}{TLR:1} - -\item The tools shall perform the above fitting steps for the guide stars -rather than for the normal image data.\TASK -\end{enumerate} \subsection{Modules} @@ -1968,13 +1718,7 @@ \subsubsection{External Catalogs} -The IPP AP Database shall be able to interact with the following -externally provided reference catalogs listed in Table~\ref{AstroRefs} -and Table~\ref{PhotoRefs}.\VER{TEST}{TLR:1, TLR:3} - -\subsubsection{Analysis Reference Data} - -The IPP shall store reference data describing the relevant Pan-STARRS -and IPP components, including the telescope, camera, detectors, -filters, clustered computers, and IPP software parameters. +The IPP AP Database shall be able to interact with the externally +provided reference catalogs listed in Table~\ref{AstroRefs} and +Table~\ref{PhotoRefs}.\VER{TEST}{TLR:1, TLR:3} \subsubsection{Static Sky Pixel Size} @@ -2137,13 +1881,13 @@ images obtained at a cadence of 1 image per 40 seconds.\VER{TEST}{TLR:17} -\item The IPP shall perform the Phase 2 analysis within an average -time of 40 seconds per single Gigapixel camera image. The Phase 2 -analysis has been measured to require 3200 GHz-sec on a Pentium-4 -machine.\VER{TEST}{TLR:17} - -\item The IPP shall perform the Phase 4 analysis on a set of 4 input -frames within an average time of 180 seconds. The Phase 4 analysis -has been measured to require a total of 7800 GHz-sec on a Pentium-4 -machine for a major frame of 4 input Gigapixel camera +\item The IPP shall perform the Phase 1 and Phase 2 analyses within an +average time of 40 seconds per single Gigapixel camera image. The +Phase 2 analysis has been measured to require 3200 GHz-sec on a +Pentium-4 machine.\VER{TEST}{TLR:17} + +\item The IPP shall perform the Phase 3 and Phase 4 analyses on a set +of 4 input frames within an average time of 180 seconds. The Phase 4 +analysis has been measured to require a total of 7800 GHz-sec on a +Pentium-4 machine for a major frame of 4 input Gigapixel camera images.\VER{TEST}{TLR:17} \end{enumerate} @@ -2198,5 +1942,5 @@ assumptions of bandwidth and CPU speeds to estimate the number of nodes required for the IPP. Each CPU is matched with one network -adapter and one disk array. : +adapter and one disk array. \begin{enumerate} \item The IPP requires at least 40 Phase 2 Nodes (OTA Nodes)\VER{TEST}{TLR:17} @@ -2206,9 +1950,4 @@ \item The IPP requires at least 10 AP DB Nodes\VER{TEST}{TLR:17} \end{enumerate} - -\subsubsection{Availability} - -The IPP Image Server nodes shall not be offline for more than 12 hours - consecutively or 36 hours per year.\VER{ANALYSIS}{TLR:17} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%