Changeset 1399
- Timestamp:
- Aug 6, 2004, 9:06:01 AM (22 years ago)
- Location:
- trunk/doc/design
- Files:
-
- 3 edited
-
Makefile (modified) (1 diff)
-
ippSDRS.tex (modified) (4 diffs)
-
ippSRS.tex (modified) (64 diffs)
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/design/Makefile
r1098 r1399 1 # $Id: Makefile,v 1. 6 2004-06-25 22:06:34eugene Exp $1 # $Id: Makefile,v 1.7 2004-08-06 19:06:01 eugene Exp $ 2 2 3 3 PDFLATEX = pdflatex 4 4 PSLATEX = latex 5 6 srs : 7 trace.pl ippSRS.tex ippSRSout.tex ippSRStrace.tex 8 ltx ippSRSout 5 9 6 10 all : ippSCD.pdf ippSRS.pdf ippSDRS.pdf -
trunk/doc/design/ippSDRS.tex
r1091 r1399 1 %%% $Id: ippSDRS.tex,v 1. 3 2004-06-25 03:05:31 eugene Exp $1 %%% $Id: ippSDRS.tex,v 1.4 2004-08-06 19:06:01 eugene Exp $ 2 2 \documentclass[panstarrs]{panstarrs} 3 3 … … 2029 2029 \subparagraph{Combine Images} 2030 2030 2031 \tbd{for moving objects and images which are not simultaneous, do we 2032 identify the moving objects?} 2033 2034 \tbd{use the spatial information? fit a 2-D Nth order polynomial to 2035 the collection of pixels and then look for outliers} 2036 2031 2037 The first module for Phase 4 is to combine the images from each 2032 2038 telescope, rejecting artifacts such as cosmic rays and low altitude … … 2086 2092 \subparagraph{Transient Identification} 2087 2093 2094 \tbd{what about different stellar colors?} 2095 2088 2096 This module identifies variable/moving sources. The inputs are: 2089 2097 \begin{enumerate} … … 2134 2142 2135 2143 \subparagraph{Add to Static Sky} 2144 2145 \tbd{how to handle variable stars?} 2136 2146 2137 2147 This module adds the combined sky cell image into the static sky, so -
trunk/doc/design/ippSRS.tex
r1084 r1399 1 %%% $Id: ippSRS.tex,v 1.6 2004-06-24 20:24:27eugene Exp $2 \documentclass[panstarrs ]{panstarrs}1 %%% $Id: ippSRS.tex,v 1.7 2004-08-06 19:06:01 eugene Exp $ 2 \documentclass[panstarrs,spec]{panstarrs} 3 3 4 4 % basic document variables … … 7 7 \shorttitle{IPP SRS} 8 8 \author{Eugene Magnier, Paul A. Price, Josh Hoblitt} 9 \audience{Pan-STARRS PMO} 9 10 \group{Pan-STARRS Algorithm Group} 10 11 \project{Pan-STARRS Image Processing Pipeline} … … 12 13 \version{DR} 13 14 \docnumber{PSDC-430-005} 15 16 \newcommand\FRM[2]{\parbox[t]{#1}{\raggedright #2}} 17 \newcommand\FRA{100pt} 18 \newcommand\FRB{260pt} 19 \newcommand\FRC{40pt} 20 \newcommand\FRD{80pt} 21 \newcommand\FRN[1]{\FRM{50pt}{#1}} 22 \newcommand\FRS[1]{\FRM{190pt}{#1}} 23 \newcommand\VER[2]{\\ {\scriptsize QUALIFICATION METHOD: #1, TRACE: #2}} 24 \newcommand\TASK{\\ {\scriptsize TASK}} 14 25 15 26 % allow paragraphs to be listed in TOC for now … … 30 41 31 42 \TBDsStart 32 % section page TBR number Description 33 section & page & TBR & description \\ \hline 43 Section & Page & Number & Description \\ \hline 44 & & & choice of scripting language \\ \hline 45 & & & coding standards for scripting language \\ \hline 46 & & & low-spatial frequency sky model details \\ \hline 34 47 \TBDsEnd 35 48 … … 47 60 \subsection{Identification} 48 61 49 This document establishes the software requirements for the Pan-STARRS 50 Image Processing Pipeline (IPP) as applied to Pan-STARRS 1 (PS-1), the 51 initial demonstration telescope to be constructed on Haleakala by Jan 52 2006. 62 This document is the Software Requirements Specification (SRS) for the 63 Panoramic Survey Telescope and Rapid Response System (Pan-STARRS) 64 Image Processing Pipeline (IPP) for the prototype telescope PS-1, and 65 is a System-level controlled specification/design description document 66 in the official Pan-STARRS engineering specification tree. 53 67 54 68 \subsection{System Overview} 55 69 56 \tbd{description of the Pan-STARRS System and PS-1.} 70 The Institute for Astronomy at the University of Hawaii is developing 71 a large optical synoptic survey telescope system, the Panoramic Survey 72 Telescope and Rapid Response System (Pan-STARRS). The science goals, 73 priorities, top-level concept of operations with associated 74 operational requirements, and system performance drivers with 75 associated system performance requirements are described in the 76 Pan-STARRS Science Goals Statement (SGS). As described in this 77 document, The system conceptual design for Pan-STARRS utilizes an 78 array of four 1.8m telescopes each with a 7 degree$^2$ field of view, 79 giving the system an \'etendue larger than all existing survey 80 instruments combined (defined as the product of the collecting area 81 $A$ multiplied by the field-of-view solid angle $\Omega$). Each 82 telescope will be equipped with a 1 billion pixel CCD camera with low 83 noise and rapid read-out, and the data will be reduced in near real 84 time to produce both cumulative static sky and difference images from 85 which transient, moving, and variable objects can be 86 detected. Pan-STARRS will be able to survey up to $\approx 6,000$ 87 degree$^{2}$ per night to a detection limit of approximately 24$^{th}$ 88 magnitude. This unique combination of sensitivity and sky coverage 89 will open up many new possibilities in time domain astronomy including 90 a major goal of surveying the Potentially Hazardous Object (PHO) 91 population down to a diameter of $\approx 300$ meters. In addition, 92 the Pan-STARRS data will be used to investigate a broad range of 93 astronomical problems of extreme current interest concerning the Solar 94 System, the Galaxy, and the Cosmos at large. A prototype single 95 telescope system, PS-1, is being developed as a preliminary step 96 before construction of the complete four telescope system. 97 98 \begin{tabular}{ll} 99 Project sponsor:& AFRL, United States Air Force \\ 100 Acquirer: & University of Hawaii Institute for Astronomy \\ 101 User: & Astronomical community \\ 102 Developer: & University of Hawaii Institute for Astronomy, participating \\ 103 & institutions, and associated subcontractors 104 \end{tabular} 57 105 58 106 \subsection{Document Overview} 107 108 The Pan-STARRS IPP Software Requirements Specification contains the 109 complete system requirements of the Pan-STARRS PS-1 IPP in order to 110 achieve the top-level performance and operational requirements 111 specified by the SCD. The requirements flow begun in the SGS and 112 continued in the SCD is further developed in this SRS to provide 113 additional derived system and subsystem requirements. 114 115 \subsection{Requirements Definitions} 59 116 60 117 The Pan-STARRS document naming scheme is PSDC-NNN-MMM-VV, where the VV … … 63 120 that series is implied. 64 121 65 Open issues (TBDs) in this document are marked \tbd{in bold, red with 66 surrounding square brackets}. 67 68 Quantities which should be reviewed (TBRs) are marked \tbr{in bold, 69 blue with surrounding square brackets}. 70 71 \subsubsection{Requirements Definitions} 72 73 \paragraph{``Must''} When used in this specification, the word 74 ``must'' refers to an explicit requirement of a system component or 75 the complete system. In this document, the use of the word ``must'' 76 replaces, and is equivalent to, use of the word ``shall'' found in 77 many requirements documents. 78 79 \paragraph{``Should''} When used in this specification, the word 122 Open issues (TBDs) in this document are marked \tbd{in bold red}. 123 124 Quantities which should be reviewed (TBRs) are marked \tbr{in bold 125 blue}. 126 127 \subsubsection{``Shall''} When used in this specification, the word 128 ``shall'' refers to an explicit requirement of a system component or 129 the complete system. 130 131 \subsubsection{``Should''} When used in this specification, the word 80 132 ``should'' refers to a desired characteristic of a system component or 81 133 the complete system. 82 134 83 \ paragraph{``Will''} When used in this specification, the word135 \subsubsection{``Will''} When used in this specification, the word 84 136 ``will'' provides information about a characteristic of a related 85 137 system component or a complete related system. … … 102 154 \section{Requirements} 103 155 104 \subsection{ ScienceRequirements}156 \subsection{Top-Level Requirements} 105 157 \label{req:system-capabilities} 106 158 107 The IPP must perform the following tasks: 108 109 \begin{enumerate} 110 111 \item Accept raw images from the summit at a sustained rate of 1 112 exposure (2~GB) per 30 seconds. 113 114 \item Accept metadata from the summit at a sustained rate of \tbr{1 MB 115 per second}. 116 117 \item Produce master calibration images from the raw calibration 118 images. The master calibration images must not introduce systematic 119 uncertainties in the photometry greater than \tbr{0.2\%}. 120 121 \item Pre-process the science images with the master calibration 122 images. 123 124 \item Merge multiple pre-processed science images -- from multiple 125 telescopes or from sequential, dithered exposures -- into stacked 126 images with corresponding signal-to-noise maps. Pixels from the 127 input images which are outliers for the ensemble of corresponding 128 pixels must be excised. 129 130 \item Subtract a static sky image from the stacked images to produce 131 an image of only the transient objects. 132 133 \item Excise transients and outliers which exceed a user-configurable 134 threshold in the subtracted image from the pre-processed science 135 images. 136 137 \item Merge the cleaned images into the static sky image, and update 138 the corresponding exposure (S/N) maps. 139 140 \item Detect and measure parameters of objects on the four types of 141 images: pre-processed images, the stacked image, the difference 142 image, and the static sky image. 143 144 \item Determine astrometry of the detected objects relative to an 145 astrometric reference. For the Commissioning phase of PS-1, the 146 astrometric calibration will be limited by the determination of the 147 optical model of the focal plane, and may be as poor as \tbr{750 148 mas}. For the AP reference construction phase of PS-1, after the 149 optical model has been measured, the astrometry solution must be 150 limited by the reference catalog in use, and will be in the vicinity 151 of \tbr{75 mas (UCAC) - 250 mas (USNO B1.0)}. After the construction 152 of the AP astrometric reference catalog, the accuracy will be limited 153 by atmospheric variations, and must be no worse than \tbr{50 mas}, 154 with a goal of \tbr{10 mas}. 155 156 \item Determine photometry of the detected objects, both within an 157 internal photometric system and in terms of appropriate external 158 photometric reference systems. For the Commissioning phase, the 159 accuracy of the photometric calibration will be limited by the 160 quantity and quality of the standard star observations, and the 161 consistency of the flat-field images across the camera; the scatter 162 must be less than \tbr{25 millimags}. During the AP reference 163 construction phase of PS-1, after the flat-field correction has been 164 measured, the photometric accuracy will be limited by the standard 165 star observations, the zero-point determinations, and in the case of 166 calibration to the external standard, the color corrections. The 167 photometric accuracy in this stage must be better than \tbr{10 168 millimags}. After the construction of the AP Reference Catalog, the 169 photometric accuracy will be limited by knowledge of the flat-field, 170 variations in the atmosphere across the field, and the reference 171 catalogs. The photometric scatter in photometric weather must be 172 better than \tbr{5 millimag} for relative photometry (relative to the 173 internal filter system) and \tbr{10 millimag} for absolute photometry 174 (relative to other filter systems such as the SDSS filters). 175 176 \item Produce a high-quality astrometric reference catalog from the 177 extracted objects within 6 months of the end of the AP Survey. The 178 astrometric reference must have an absolute accuracy of \tbr{30 mas} 179 and a local relative accuracy of \tbr{10 mas}. Proper motions of 180 detected non-solar-system objects must be determined with an 181 accuracy of \tbr{20 mas / year} for unsaturated, bright stars. 182 183 \item Produce a high-quality photometric reference catalog from the 184 extracted point-source objects within 6 months of the end of the AP 185 Survey. The photometric reference must have an consistency across 186 the sky of \tbr{5 millimag} and an absolute calibration to the 187 external system (defined by \tbr{SDSS} and the CFHT Legacy Survey 188 Standards) with an accuracy of \tbr{10 millimag}. 189 190 \item Publish the static sky images to the Pan-STARRS published static 191 sky server on a time-scale of \tbr{1 month}. 192 193 \item Publish the detected objects to the Pan-STARRS published object 194 database on a time-scale of \tbr{1 week}. 195 196 \item Provide access to external Pan-STARRS clients to the detected 197 objects on time-scales of \tbr{10 minute} after the image is 198 obtained.\comment{this is derived from the top-level science 199 requirement.} 200 201 \item Store the raw images for a period of time which depends on the 202 survey source of the data. In PS-1, the AP and IVP Survey data must 203 be stored for the lifetime of the project. Other raw data must be 204 stored for \tbr{1 month}. 205 206 \item Store the detected objects for a period of time, depending on 207 the type of detection. Transients from the P4$\Delta$ images may be 208 excised after \tbr{6 months}. 209 159 The Pan-STARRS System Concept Definition (SCD) specifies the derived 160 top-level requirements for the IPP, which we reproduce here (with 161 numbering consistent with this document): 162 163 \begin{enumerate} 164 \item Produce reduced science images for each full camera exposure 165 which are photometrically consistent across the field to within 1\%.\VER{ANALYSIS}{SCD:3.2.2.5} 166 \label{TLR:1} 167 168 \item Produce reduced science images for each full camera exposure 169 which are photometrically calibrated to within 1\%.\VER{ANALYSIS}{SCD:3.2.2.5} 170 \label{TLR:2} 171 172 \item Produce reduced science images for each full camera exposure 173 which are astrometrically calibrated to 100 milliarcseconds to an 174 absolute reference.\VER{ANALYSIS}{SCD:3.2.2.6} 175 \label{TLR:3} 176 177 \item Produce reduced science images for each full camera exposure 178 which are astrometrically consistent to 30 179 milliarcseconds.\VER{ANALYSIS}{SCD:3.2.2.7} 180 \label{TLR:4} 181 182 \item Produce reduced science images for each full camera exposure 183 which have foreground emission subtracted with no more than 1\% 184 variation in the non-astronomical background.\VER{ANALYSIS}{SCD:3.5.12} 185 \label{TLR:5} 186 187 \item Merge all $g$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} 188 \label{TLR:6} 189 190 \item Merge all $r$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} 191 \label{TLR:7} 192 193 \item Merge all $i$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} 194 \label{TLR:8} 195 196 \item Merge all $z$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} 197 \label{TLR:9} 198 199 \item Merge all $y$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} 200 \label{TLR:10} 201 202 \item Merge all $w$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10} 203 \label{TLR:11} 204 205 \item Detect and classify objects on the individual processed science images.\VER{TEST}{SCD:3.2.2.16} 206 \label{TLR:12} 207 208 \item Detect and classify objects on the stacked groups of science images.\VER{TEST}{SCD:3.2.2.16} 209 \label{TLR:13} 210 211 \item Detect and classify objects on the static sky image.\VER{TEST}{SCD:3.2.2.16} 212 \label{TLR:14} 213 214 \item Detect all significant transients in the individual science 215 images relative to the static sky image.\VER{TEST}{SCD:3.2.2.16} 216 \label{TLR:15} 217 218 \item Degrade the stacked image by no more than \tbr{10 milliarcseconds}.\VER{ANALYSIS}{SCD:3.5.2} 219 \label{TLR:16} 220 221 \item Perform the processing of science images to the level of 222 transient detection and static sky inclusion at a rate such that 223 exposures taken at a cadence of \tbr{40} seconds do not accumulate 224 in the processing buffer.\VER{TEST}{SCD:3.2.2.3} 225 \label{TLR:17} 226 227 \item Limit the false alarm rate (FAR) to less than \tbr{5\%} for 228 transient detections $> 5\sigma$ sent to the preferred client science 229 pipelines.\footnote{note difference with PS-4: 1\%} 230 \VER{ANALYSIS}{SCD:3.2.2.13} 231 \label{TLR:18} 232 233 \item Publish the static sky images to the Pan-STARRS Published 234 Science Products Subsystem (PSPS) once per \tbr{6 235 months}.\VER{TEST}{SCD:3.2.2.18} 236 \label{TLR:19} 237 238 \item Publish the detected objects to the Pan-STARRS Published Science 239 Products Subsystem (PSPS) once per month.\VER{TEST}{SCD:3.2.2.18} 240 \label{TLR:20} 241 242 \item Send the IPP metadata and received OTIS metadata to the 243 Pan-STARRS Published Science Products Subsystem (PSPS) weekly.\VER{TEST}{SCD:3.2.2.18} 244 \label{TLR:21} 245 246 \item Provide access to preferred Pan-STARRS science clients to the 247 detected transient objects within \tbr{5 minutes}.\VER{TEST}{SCD:3.5.10} 248 \label{TLR:22} 249 250 \item Provide sufficent storage volume for \tbr{1 year} of raw images 251 from the AP and IVP Surveys.\footnote{note difference with PS-4: 1 252 month of raw images} \VER{INSPECT}{allocated} 253 \label{TLR:23} 254 255 \item Provide sufficient storage volume for all detections from the 256 AP, IVP, and MVP Surveys.\footnote{note difference with PS-4: 1 257 year of detections}\VER{INSPECT}{allocated} 258 \label{TLR:24} 259 260 \item Provide sufficient storage volume for \tbr{2 year} of 261 metadata.\footnote{note difference with PS-4: 10 262 years of metadata}\VER{INSPECT}{allocated} 263 \label{TLR:25} 210 264 \end{enumerate} 211 265 212 266 \subsection{Required States} 213 267 214 The IPP must have 3 states: active, paused, and interactive. 215 216 \subsubsection{Active State} 217 \label{req:active-state} 218 219 In active state, the IPP must: 220 221 \begin{enumerate} 222 \item Accept images and metadata from the external sources (i.e., the 223 summit) 224 225 \item Automatically perform the complete set of image processing 226 tasks, including both calibration and science image processing. 227 228 \item Respond to requests for data from client science pipelines. 229 230 \item Respond to analysis priority requests issued by the IPP users. 231 \end{enumerate} 232 233 \subsubsection{Paused State} 234 \label{req:paused-state} 235 236 In paused state, the IPP must refuse incoming data and metadata and 237 data requests from the client science pipelines. 238 239 \subsubsection{Interactive State} 240 \label{req:interactive-state} 241 242 In interactive state, the IPP must: 243 244 \begin{enumerate} 245 \item Accept incoming data and metadata from the external sources. 246 \item Not automatically process the data 247 \item Respond to user commands to initiate portions of the data 248 analysis. 249 \end{enumerate} 268 The IPP has 3 operating states: active, paused, and interactive. In active state, the IPP: 269 270 \begin{itemize} 271 \item Accepts images and metadata from the external sources (i.e., the summit) 272 273 \item Automatically performs the complete set of image processing 274 tasks, including both calibration and science image 275 processing. 276 277 \item Responds to requests for data from client science pipelines, 278 possibly pre-registered classes of data requests. 279 280 \item Responds to analysis priority requests issued by the IPP operators. 281 \end{itemize} 282 283 In paused state, the IPP refuses incoming data and metadata and data 284 requests from the client science pipelines. 285 286 The interactive state is intermediate between these two. In 287 interactive state, the IPP: 288 289 \begin{itemize} 290 \item Accepts incoming data and metadata from the external sources. 291 \item Does {\em not} automatically process the data. 292 \item Responds to user commands to perform portions of the data analysis. 293 \end{itemize} 250 294 251 295 \subsection{Software Coding Requirements} … … 255 299 256 300 \begin{enumerate} 257 \item Source code must be in C.258 \item All source code must be compiled with `gcc' version v2.95 or higher.259 \item The tested compiler version must be defined for the delivered software product.260 \item Scripting language must be \tbd{Python}, version X.X.301 \item Source code shall be in C. \VER{INSPECT}{allocated} 302 \item All source code shall be tested with `gcc' version v2.95 or higher. \VER{INSPECT}{allocated} 303 \item The tested compiler version shall be defined for the delivered software product. \VER{INSPECT}{allocated} 304 \item Scripting language shall be Perl. \VER{INSPECT}{allocated} 261 305 \end{enumerate} 262 306 263 307 \subsubsection{Interfaces} 264 308 \begin{enumerate} 265 \item Access to low-level Library functions must be provided via C 266 APIs consisting of the function calls and the defined data structures 267 and other data types. 268 \item Access to high-level functions must be provided via C APIs as 269 well as SWIG interfaces, where specified. 270 \item Access to processing jobs must be available via the UNIX shell. 309 \item Access to low-level Library functions shall be provided via C APIs consisting of the function calls and the defined data structures and other data types. \VER{INSPECT}{allocated} 310 \item Access to high-level functions shall be provided via C APIs. \VER{INSPECT}{allocated} 311 \item Access to specified C functions in higher level languages shall employ SWIG. \VER{INSPECT}{allocated} 312 \item Access to processing jobs shall be available via the UNIX shell. \VER{INSPECT}{allocated} 271 313 \end{enumerate} 272 314 … … 274 316 275 317 \begin{enumerate} 276 \item The C code must comply with ANSI Standard C99. 277 \item Because the delivered code is required to run on UNIX machines, 278 the delivered code must be in compliance with the language-independent 279 UNIX operating system standard POSIX (Open Group Based Specifications 280 Issue 6, IEEE Std 1003.1, 2004). 281 \item Source code files must use the UNIX line-break 282 convention (line-feed only). 283 \item C coding style must adhere to the standard defined in the 284 document 'Pan-STARRS C-coding standard' (PSDC-430-004). 285 \item \tbd{Python} coding must follow the standard defined in the 286 document \tbd{TBD}. 318 \item The C code shall comply with ANSI Standard C99. \VER{INSPECT}{allocated} 319 \item Because the delivered code is required to run on UNIX machines, the delivered code shall be in compliance with the language-independent UNIX operating system standard POSIX (Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2004).\VER{INSPECT}{allocated} 320 \item Source code files shall use the UNIX line-break convention (line-feed only). \VER{INSPECT}{allocated} 321 \item C coding style shall adhere to the standard defined in the document `Pan-STARRS C-coding standard' (PSDC-430-004). \VER{INSPECT}{allocated} 322 \item Perl coding shall follow the standard defined in the document `Pan-STARRS Perl-coding standard' \tbd{(PSDC-430-0XX)}.\VER{INSPECT}{allocated} 287 323 \end{enumerate} 288 324 … … 290 326 291 327 \begin{enumerate} 292 \item Header files musthave names starting \code{ps} or \code{p_ps}293 for private interface definitions. The latter mustappear in a328 \item Header files shall have names starting \code{ps} or \code{p_ps} 329 for private interface definitions. The latter shall appear in a 294 330 subdirectory \code{private} of whichever directory is being searched 295 for the public header files. 331 for the public header files.\VER{INSPECT}{allocated} 296 332 297 333 \item Functions visible at global scope that are part of the public 298 API musthave names beginning with \code{ps} and follow the naming299 conventions in the coding standard. 334 API shall have names beginning with \code{ps} and follow the naming 335 conventions in the coding standard. \VER{INSPECT}{allocated} 300 336 301 337 \item Functions visible at global scope but which are not part of the 302 public interface must have names beginning with \code{p_ps}.338 public interface shall have names beginning with \code{p_ps}.\VER{INSPECT}{allocated} 303 339 304 \item Functions that are local to a file must\textit{not} start with305 \code{ps} or \code{p_ps}. 340 \item Functions that are local to a file shall \textit{not} start with 341 \code{ps} or \code{p_ps}.\VER{INSPECT}{allocated} 306 342 307 343 \item Variables visible at global scope which are part of the public 308 API musthave names beginning with \code{ps}, and follow the naming309 conventions in the coding standard. 344 API shall have names beginning with \code{ps}, and follow the naming 345 conventions in the coding standard. \VER{INSPECT}{allocated} 310 346 311 347 \item Variables that are visible at global scope but which are not 312 part of the public interface musthave names beginning with313 \code{p_ps}. 314 315 \item Variables that are local to a file must\textit{not} start with316 \code{ps} (or \code{p_ps}). 348 part of the public interface shall have names beginning with 349 \code{p_ps}.\VER{INSPECT}{allocated} 350 351 \item Variables that are local to a file shall \textit{not} start with 352 \code{ps} (or \code{p_ps}).\VER{INSPECT}{allocated} 317 353 318 354 \item The names of all enumerated types and C-preprocessor symbols 319 (but not variables declared \code{const}) muststart with \code{PS_},355 (but not variables declared \code{const}) shall start with \code{PS_}, 320 356 in the case of public symbols, or \code{P_PS_}, for private symbols. 321 The rest of the name mustbe uppercase with words separated by357 The rest of the name shall be uppercase with words separated by 322 358 underscores (\code{_}). An exception is the case of system utilities 323 implemented as macros, in which case the names mustconform to the324 convention for function names. 359 implemented as macros, in which case the names shall conform to the 360 convention for function names.\VER{INSPECT}{allocated} 325 361 326 362 \item When defining a function to convert from one type to another, 327 the name mustbe of the form \code{psOldToNew},363 the name shall be of the form \code{psOldToNew}, 328 364 e.g.\code{psEquatorialToEcliptic} (\emph{not} 329 \code{psEquatorial2Ecliptic}). 365 \code{psEquatorial2Ecliptic}).\VER{INSPECT}{allocated} 330 366 \end{enumerate} 331 367 … … 333 369 334 370 \begin{enumerate} 335 \item Functions that assign to a variable must list that argument 336 \textit{first}, following the pattern of \code{strcpy}. For 337 example: 338 \begin{verbatim} 339 void psVectorCopy(restrict psVector *out, const restrict psVector *in); 340 \end{verbatim} 341 342 \item Type definitions should always be accompanied by prototypes for 343 their constructors and destructors, following these guidelines: 344 345 \begin{enumerate} 346 \item The constructor name should consist of the type name followed 347 by \code{Alloc}; e.g. a type \code{psImage} would be created by a 348 function \code{psImage *psImageAlloc();}. 349 350 \item The type should be freed with a destructor named 351 \code{typeFree}, e.g. \code{void psImageFree(psImage *image);}. 352 353 \item The constructor must never return \code{NULL}, and no code 354 calling the constructor should ever check the return value. 355 356 \item The destructor must not return a value. 357 358 \item The destructor must handle being passed \code{NULL} by simply 359 returning immediately. This must not be treated as an error 360 condition. 361 362 \item Constructors and Destructors should use the memory reference 363 counter facilities of the PSLib memory management system. 364 \end{enumerate} 371 \item Functions that assign to a variable shall list that argument 372 \textit{first}, following the pattern of \code{strcpy}. \VER{INSPECT}{allocated} 373 374 Type definitions should always be accompanied by prototypes for their 375 constructors. Corresponding destructors are private functions 376 registered with the PSLib memory management system. 377 378 \item The constructor name shall consist of the type name followed by 379 \code{Alloc}; e.g. a type \code{psImage} would be created by a 380 function \code{psImage *psImageAlloc();}.\VER{INSPECT}{allocated} 381 382 \item The constructor shall never return \code{NULL}, so code calling 383 the constructor should not check the return value.\VER{INSPECT}{allocated} 384 385 \item The destructor shall not return a value.\VER{INSPECT}{allocated} 386 387 \item Constructors and Destructors shall use the memory reference 388 counter facilities of the PSLib memory management system.\VER{INSPECT}{allocated} 365 389 \end{enumerate} 366 390 … … 368 392 369 393 \begin{enumerate} 370 \item Commenting of delivered C code mustfollow the C coding394 \item Commenting of delivered C code shall follow the C coding 371 395 standards and provide tags for Doxygen interpretation of the 372 comments and program structures. 373 374 \item Commenting of delivered P ython code must follow the Python375 coding standards. 376 377 \item Source code documentation mustbe generated with Doxygen from378 the in-line comments and mustbe provided as HTML, Latex, and man379 pages. 396 comments and program structures.\VER{INSPECT}{allocated} 397 398 \item Commenting of delivered Perl code shall follow the Perl 399 coding standards.\VER{INSPECT}{allocated} 400 401 \item Source code documentation shall be generated with Doxygen from 402 the in-line comments and shall be provided as HTML, Latex, and man 403 pages. \VER{INSPECT}{allocated} 380 404 381 405 \item User documentation includes the API usage for the modules and 382 406 library functions as well as user interface description for the 383 higher-level architectural systems. User documentation mustbe384 delivered as PDF documents. 407 higher-level architectural systems. User documentation shall be 408 delivered as PDF documents.\VER{INSPECT}{allocated} 385 409 \end{enumerate} 386 410 387 411 \subsubsection{Version Control} 388 412 389 Source code version control must be implemented with CVS.413 Source code version control shall be implemented with CVS. \VER{INSPECT}{allocated} 390 414 391 415 \subsubsection{CSCI Deliverable} 392 416 393 All final source code generated for the IPP mustbe delivered via CVS,394 including the test code. CVS revision history mustbe included and395 made available via CVS. 417 All final source code generated for the IPP shall be delivered via CVS, 418 including the test code. CVS revision history shall be included and 419 made available via CVS.\VER{INSPECT}{allocated} 396 420 397 421 \subsubsection{Platform architectures and operating systems} 398 422 399 Makefiles mustbe provided with appropriate flags set so that all400 code compiles without warnings under 'gcc -Wall' for the following401 platform architectures and operating systems: 423 Makefiles shall be provided with appropriate flags set so that all 424 code compiles without warnings under `gcc -Wall' for the following 425 platform architectures and operating systems:\VER{INSPECT}{allocated} 402 426 403 427 \begin{itemize} … … 410 434 such as those caused by lex-generated code. 411 435 412 Although the code mustcompile successfully under both listed436 Although the code shall compile successfully under both listed 413 437 operating systems, unit testing should only be performed for the 414 438 x86/Linux combination. … … 416 440 \subsubsection{Timing measurements} 417 441 418 Timing requirements specified in this document mustbe achieved on the419 deployed Pan-STARRS analysis computers. 442 Timing requirements specified in this document shall be achieved on the 443 deployed Pan-STARRS analysis computers.\VER{TEST}{allocated} 420 444 421 445 \subsubsection{Software Configuration} 422 446 423 \tbd{deferred} 447 \paragraph{Version Management} 448 449 The IPP software configuration management system shall ensure that 450 validated versions of both internal and external software are used 451 when the software is compiled.\VER{TEST}{allocated} 452 453 \paragraph{Optional Modes} 454 455 The IPP software configuration management system shall provide 456 optionally selected software version sets under compilation 457 conditions. For example, compilation of the software for test 458 purposes with a non-standard FFT tool shall be an 459 option.\VER{TEST}{allocated} 424 460 425 461 \subsection{Architectural Components} 426 462 463 \begin{figure} 464 \begin{center} 465 \resizebox{6in}{!}{\includegraphics{pics/IPPoverview.ps}} 466 \caption{ \label{overview} IPP System Overview} 467 \end{center} 468 \end{figure} 469 427 470 As discussed in the Pan-STARRS System Concept Definition 428 (PSDC- xxx-xxx), the IPP is organized into a number of clearly-defined471 (PSDC-250-002), the IPP is organized into a number of clearly-defined 429 472 software elements. The SCD provides a detailed description of the 430 473 roles and responsibilities of these subsystems. In brief, the IPP … … 439 482 \begin{itemize} 440 483 441 \item {\bf I mage Server:} This component is a large data store for all484 \item {\bf IPP Image Server:} This component is a large data store for all 442 485 images used by the IPP, including the raw images from the telescope, 443 486 the master calibration images, the reference static-sky images, and 444 487 any temporary image data products produced by the IPP. The Image 445 Server is required to meet all of the image storage needs identified 446 in the top-level requirements above. The Image Server may also store 447 large data files which do not contain imaging data. The Image Server 448 must accept the incoming data and store it until it is no longer 449 needed by other portions of the IPP. 488 Server may also store large data files which do not contain imaging 489 data. The Image Server accepts the incoming data and stores it until 490 it is no longer needed by other portions of the IPP. 450 491 451 492 \item {\bf Astrometry \& Photometry Database (AP):} This component is 452 required to store and manipulate astronomical objects detected in 453 images processed by the IPP, including individual measurements of 454 objects on the images, the summary information about those objects, 455 and reference object data. 456 457 \item {\bf Metadata Database:} This component is required to store the 458 all other data which are neither image files nor astronomical object 493 used to store and manipulate astronomical objects detected in images 494 processed by the IPP, including individual measurements of objects on 495 the images, the summary information about those objects, and 496 reference object data. It includes descriptive information about the 497 images, filter, cameras, telescopes, and other aspects of the system 498 needed to interpret the object data. 499 500 \item {\bf IPP Metadata Database:} This component is used to store all 501 other data which are neither image files nor astronomical object 459 502 data. The Metadata Database is the authoritative source for all 460 503 metadata data, including metadata which may be duplicated elsewhere, 461 504 such as in the headers of images in the image database. 462 505 463 \item {\bf Controller:} In order to perform the analysis stages506 \item {\bf IPP Controller:} In order to perform the analysis stages 464 507 required by the IPP, it is necessary to use distributed computing 465 508 processes on a large number of computers. The Controller is required … … 467 510 machines. 468 511 469 \item {\bf Scheduler:} This component is a decision-making mechanism512 \item {\bf IPP Scheduler:} This component is a decision-making mechanism 470 513 required to guide the operation of the IPP: to evaluate the currently 471 514 available collection of data, to identify the necessary analysis, and … … 476 519 The relationship between these software elements is shown in 477 520 Figure~\ref{overview}. This figure also shows the interactions 478 between the IPP and other Pan-STARRS systems. 479 480 \begin{figure} 481 \begin{center} 482 \resizebox{8cm}{!}{\includegraphics{pics/overview}} 483 \caption{ \label{overview} IPP System Overview} 484 \end{center} 485 \end{figure} 521 between the IPP and other Pan-STARRS systems. The following sections 522 identify requirements of these five software elements. 486 523 487 524 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% … … 490 527 491 528 \begin{enumerate} 492 \item The IPP Image Server must store images on a distributed 529 \item The IPP Image Server shall accept raw images from the summit at 530 a sustained rate of 1 exposure (2~GB) per \tbr{40 531 seconds}. \VER{TEST}{TLR:17, TLR:23} 532 533 \item The IPP Image Server shall store images on a distributed 493 534 collection of computer disks. Individual instances of a file are 494 535 only required to be stored on a single machine (striping across 495 computers is not a requirement). 496 497 \item The IPP Image Server must be capable of honoring requests to498 s tore an image on a specific machine.536 computers is not a requirement).\VER{TEST}{TLR:17, TLR:23} 537 538 \item The IPP Image Server shall attempt to store an image on a 539 specific machine if requested by the user.\VER{TEST}{TLR:17, TLR:23} 499 540 500 541 \item If such a request cannot be honored (ie, the machine is down), 501 the IPP Image Server mustselect an appropriate machine and notify502 the requesting agent of the new location. 503 504 \item The IPP Image Server muststore multiple copies of each image542 the IPP Image Server shall select an appropriate machine and notify 543 the requesting agent of the new location.\VER{TEST}{TLR:17, TLR:23} 544 545 \item The IPP Image Server shall store multiple copies of each image 505 546 upon request, the number of copies specified independently for each 506 file by the user. 507 508 \item The IPP Image Server must maintain a record of all image copies 509 currently available in the repository. This record must include the 510 image name, location (which machine), the image size, and the state 511 of the image (available, locked, deleted). 512 513 \item The IPP Image Server must lock images in the repository on 514 request. Both read (shared) and write (exclusive) locks must be 515 provided. A read lock must prevent write access to the file; a 516 write lock must prevent both read and write access. 517 518 \item The IPP Image Server must return the image location (the 519 computer on which it resides) upon request. 520 521 \item The IPP Image Server must provide a specified image upon request. 522 523 \item The IPP Image Server must delete images in the repository on request. 524 525 \item The IPP Image Server must accept images from the summit at the 526 maximum rate of 1 full-camera image every 30 seconds. The IPP Image 527 Server must therefore accept new images into the repository at a 528 rate of 64 raw OTAs in 30 seconds and a total input data volume rate 529 of 75 MB/sec. 530 \end{enumerate} 531 532 \tbd{archive lifetime} 533 534 \tbd{reliability} 535 536 \tbd{backups} 547 file by the user.\VER{TEST}{TLR:17, TLR:23} 548 549 \item The IPP Image Server shall maintain a record of all image copies 550 currently available in the repository. This record shall include at 551 least the image name, location (which machine), the image size, and 552 the state of the image (available, locked, deleted).\VER{INSPECT}{TLR:17, TLR:23} 553 554 \item The IPP Image Server shall lock images in the repository on 555 request. Both read (shared) and write (exclusive) locks shall be 556 provided. A read lock shall prevent write access to the file; a 557 write lock shall prevent both read and write access. \tbr{Access 558 prevention may be advisory rather than enforced.} \VER{TEST}{TLR:17, TLR:23} 559 560 \item The IPP Image Server shall return the image location (the 561 computer or computers on which it resides) upon request.\VER{TEST}{TLR:17, TLR:23} 562 563 \item The IPP Image Server shall provide a specified image upon request.\VER{TEST}{TLR:17, TLR:23} 564 565 \item The IPP Image Server shall delete images in the repository on request.\VER{TEST}{TLR:17, TLR:23} 566 567 \end{enumerate} 537 568 538 569 \subsubsection{AP Database} 570 571 The purpose of the AP Database is: 572 \begin{itemize} 573 \item to enable the photometric calibration of images 574 \item to enable the astrometric calibration of images 575 \item to enable the construction of flat-field correction frames 576 \item to enable the construction of a photometric calibration catalog 577 \item to enable the construction of an astrometric calibration catalog 578 \item to monitor the system photometry calibration parameters 579 \item to monitor the system astrometry calibration parameters 580 \item to perform the identification of single-occurance transients 581 \end{itemize} 539 582 540 583 \begin{table} … … 544 587 \hline 545 588 \hline 546 Object Parameter & P2 & P4 S & P4D& SS \\589 Object Parameter & P2 & P4$\Sigma$ & P4$\Delta$ & SS \\ 547 590 \hline 548 PSF x,y, M, $\sigma_{\rm M}$ & + & + & + & + \\ 549 $\sigma_x$, $\sigma_y$, covar. & + & + & + & + \\ 550 exp. spaced aps., Poisson noise, variance & - & - & - & + \\ 551 streak L, $\phi$, $\sigma_L$, $\sigma_\phi$ & - & - & + & + \\ 552 $x_g$, $y_g$, flag & + & + & - & + \\ 553 local sky data & + & + & + & + \\ 554 Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ 555 S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ 556 W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ 557 star/gal sep, star/streak sep. & - & + & + & + \\ 558 \hline 559 deVeucaleur R, M, AB, $\phi$ & - & + & - & + \\ 560 exponential R, M, AB, $\phi$ & - & + & - & + \\ 591 PSF x,y, covar, $\alpha,\delta$ & + & + & + & + \\ 592 PSF mag, $\sigma_{\rm mag}$ & + & + & + & + \\ 593 star/gal sep & + & + & + & + \\ 594 $\sigma_x$, $\sigma_y$, $\theta$ & + & + & + & + \\ 595 local sky data & + & + & + & + \\ 596 Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ 597 S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ 598 W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ 599 exp. spaced aps., Poisson noise, variance & - & - & - & + \\ 561 600 \hline 562 601 \end{tabular} … … 565 604 566 605 \begin{enumerate} 567 \item The AP Database mustaccept and store individual detections and606 \item The AP Database shall accept and store individual detections and 568 607 collections of detections along with information about the image 569 which provided the detections. 570 571 \item Detections must be saved as one of several detection classes 572 (P2, P4S, P4D, SS) and the AP Database must store the appropriate 573 parameters, listed in Table~\ref{APdetections}, for each class. 574 575 \item The AP Database must identify the image which provided the 608 which provided the detections.\VER{TEST}{TLR:2, TLR:3, TLR:22, TLR:24} 609 610 \item Detections shall be saved as one of several detection classes 611 (P2, P4$\Sigma$, P4$\Delta$, SS) and the AP Database shall store the 612 appropriate parameters, listed in Table~\ref{APdetections}, for each 613 class.\VER{TEST}{TLR:2, TLR:3, TLR:22, TLR:24} 614 615 \item The AP Database shall identify the image which provided the 576 616 detection, or in the case of external references, an identifier 577 specific to the reference source. 578 579 \item The AP Database mustgroup detections into objects and measure580 average parameters of those objects. 581 582 \item The AP Database muststore parallax and proper motion parameters583 for a subset of the average objects. 584 585 \item The AP Database muststore image and filter calibration617 specific to the reference source.\VER{TEST}{TLR:2, TLR:3} 618 619 \item The AP Database shall group detections into objects and measure 620 average parameters of those objects.\VER{ANALYSIS}{TLR:2, TLR:3, TLR:22} 621 622 \item The AP Database shall store parallax and proper motion parameters 623 for a subset of the average objects.\VER{TEST}{TLR:2, TLR:3, TLR:22} 624 625 \item The AP Database shall store image and filter calibration 586 626 information necessary to convert between instrumental magnitudes and 587 calibrated magnitudes in standard systems. 588 589 \item The AP Database mustperform at least the follow queries, with627 calibrated magnitudes in standard systems.\VER{INSPECT}{TLR:3} 628 629 \item The AP Database shall perform at least the follow queries, with 590 630 constraints on the output based on at least time ranges, magnitude 591 631 limits, error limits: … … 593 633 \begin{enumerate} 594 634 \item given $(RA,DEC)$ and a Radius, return all objects and/or 595 detections in the region. 635 detections in the region.\VER{TEST}{TLR:2, TLR:3} 596 636 597 637 \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all objects and/or 598 detections in the region. 599 600 \item given $(RA,DEC)$, return closest object. 601 602 \item given object ID, return all detections 603 604 \item given detection, return source image data. 605 606 \item given detection, return object. 607 608 \item given $(RA,DEC)$, return all images overlapping coordinate. 609 610 \item given $(RA,DEC)$ and a Radius, return all images overlapping region. 611 612 \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping 613 region. 638 detections in the region.\VER{TEST}{TLR:2, TLR:3} 639 640 \item given $(RA,DEC)$, return closest object.\VER{ANALYSIS}{TLR:2, TLR:3, TLR:22} 641 642 \item given object ID, return all detections\VER{TEST}{TLR:2, TLR:3} 643 644 \item given detection, return source image data.\VER{TEST}{TLR:2, TLR:3} 645 646 \item given detection, return object.\VER{TEST}{TLR:2, TLR:3, TLR:22} 647 648 \item given $(RA,DEC)$, return all images overlapping coordinate.\VER{ANALYSIS}{TLR:2, TLR:3} 649 650 \item given $(RA,DEC)$ and a Radius, return all images overlapping region.\VER{ANALYSIS}{TLR:2, TLR:3} 651 652 \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping region.\VER{ANALYSIS}{TLR:2, TLR:3} 614 653 615 654 \item given detection instrumental magnitude, return derived 616 magnitudes based on calibration information. 655 magnitudes based on calibration information.\VER{TEST}{TLR:2, TLR:3} 617 656 618 657 \item given a collection of detections in a filter, determine the 619 object average magnitude in that filter. 658 object average magnitude in that filter.\VER{ANALYSIS}{TLR:2, TLR:3} 620 659 621 660 \item given a collection of objects and detections, determine the 622 individual image zero-points. 661 individual image zero-points.\VER{ANALYSIS}{TLR:2, TLR:3} 623 662 624 663 \item given a region, return all possible combinations of the object 625 or detection magnitudes $(M_1 - M_2)$. 626 627 \item given a list of $(RA,DEC)$ entries, return all nearest objects. 664 or detection magnitudes $(M_1 - M_2)$.\VER{TEST}{TLR:2, TLR:3} 665 666 \item given a list of $(RA,DEC)$ entries, return all nearest objects.\VER{ANALYSIS}{TLR:2, TLR:3} 628 667 629 668 \item given a filter, telescope, or detector, return all calibration 630 terms and history. 669 terms and history.\VER{TEST}{TLR:2, TLR:3} 631 670 632 671 \item given a detection, return all non-detections from images which 633 overlapped the detection coordinates. 672 overlapped the detection coordinates.\VER{ANALYSIS}{TLR:2, TLR:3, TLR:22} 634 673 \end{enumerate} 635 674 636 \item The AP Database must accept detection IDs of moving objects and 637 label the detections with the identified object. 638 639 \item The AP Database must accept new detections at the rate generated 640 by the telescope from the Phase 2 and Phase 4 analysis. Except 641 within 10 degrees of the galactic plane, the AP Database must keep 642 up with the incoming rates. The expected rates are listed in 643 Table~\ref{APrates}, along with the total data volume required for 644 storage space over the PS-1 lifetime. 645 646 \end{enumerate} 647 648 \tbd{archive lifetime} 649 650 \tbd{reliability} 651 652 \tbd{backups} 675 \item The AP Database shall accept detection IDs of moving objects and 676 label the detections with the identified object.\VER{TEST}{TLR:2, TLR:3, TLR:22} 677 678 \item The AP Database shall accept new detections at the rate 679 generated by the telescope from the Phase 2 and Phase 4 analysis. 680 \tbr{Except within 10 degrees of the galactic plane, the AP Database 681 shall keep up with the incoming rates.} The expected rates are 682 listed in Table~\ref{APrates}, along with the total data volume 683 required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2, TLR:3, TLR:22} 684 685 \item The AP Database shall provide access to external Pan-STARRS 686 clients to the detected objects within \tbr{5 minute} after the 687 image is obtained.\VER{TEST}{TLR:22} 688 \label{IPP:DeReq:29c} 689 \end{enumerate} 653 690 654 691 \begin{table} 655 692 \begin{center} 656 \caption{AP Data Volume Requirements\label{APrates}}657 \begin{tabular}{lrrr r}693 \caption{AP Data Volume and Throughput Requirements\label{APrates}} 694 \begin{tabular}{lrrr} 658 695 \hline 659 696 \hline 660 Quantity & P2 & P4$\Sigma$ & P4$\Delta$ & SS\\697 Quantity & P2 & P4$\Sigma$ & P4$\Delta$ \\ 661 698 \hline 662 detection limit & $20 \sigma$ & $5 \sigma$ & $3 \sigma$ & \\ 663 depth (r') & 20.8 & 23.0 & ? & \\ 664 stars deg$^{-2}$ ($|b|>10$) & $1 \times 10^5$ & $4 \times 10^5$ & $2 \times 10^5$ & \\ 665 stars FPA$^{-1}$ ($|b|>10$) & $7 \times 10^5$ & $2.8 \times 10^6$ & $1.4 \times 10^6$ & \\ 666 stars sec$^{-1}$ ($|b|>10$) & $2.3 \times 10^4$ & $2.3 \times 10^4$ & $1.2 \times 10^4$ & \\ 667 bytes star$^{-1}$ & 64 & 100 & 64 & \\ 668 MB sec$^{-1}$ & 1.4 & 2.2 & 0.7 & \\ 669 PS-1 total TB & 8 & 12 & 4 & \\ 699 detection limit & $20 \sigma$ & $5 \sigma$ & $3 \sigma$ \\ 700 depth (r') & 21.8 & 24.0 & 24.5 \\ 701 bytes star$^{-1}$ & 64 & 100 & 64 \\ 702 stars deg$^{-2}$ ($|b|>10$) & $2.0 \times 10^5$ & $8.0 \times 10^5$ & $2.0 \times 10^5$ \\ 703 stars FPA$^{-1}$ ($|b|>10$) & $1.4 \times 10^6$ & $5.6 \times 10^6$ & $1.4 \times 10^6$ \\ 704 stars sec$^{-1}$ ($|b|>10$) & $3.5 \times 10^4$ & $3.5 \times 10^4$ & $8.8 \times 10^3$ \\ 705 MB sec$^{-1}$ & 2.3 & 3.5 & 0.6 \\ 706 AP total TB & 7.7 & - & - \\ 707 IVP total TB & 13 & 20 & 3 \\ 708 MOPS total TB & 4 & 6 & 1 \\ 709 PS-1 total TB & 25 & 26 & 4 \\ 670 710 \hline 671 711 \end{tabular} … … 679 719 \caption{Metadata Classes\label{metadata}} 680 720 \begin{tabular}{l} 681 \hline682 721 \hline 683 722 \hline … … 700 739 \end{table} 701 740 702 The IPP requires a Metadata Database to store and provide access to 703 metadata of various types and from various sources. Metadata in the 704 context of the IPP corresponds to all data which is not included in 705 the two data stores discussed above (Images and Detection/Objects). 706 Metadata is generated at the telescope and during the various analysis 707 stages 708 709 \begin{enumerate} 710 \item The Metadata Database must store and provide metadata for all 711 raw images, for processed images, for the calibration images (both 712 raw and master), for the extracted object lists. Metadata 713 describing the environmental conditions at the telescope must also 714 be stored and provided as needed. Table~\ref{metadata} lists the 715 classes of metadata which must be stored by the Metadata Database. 716 717 \item If analysis results are exchanged between analysis stages via 718 the Metadata Database, it must provide access to the queried data on 719 timescales of $<2$ seconds to avoid slowing down the analysis 720 systems. 721 722 \item The Metadata Database must store the metadata for the lifetime 723 of the project. 724 725 \item The Metadata Database must be capable of accepting a total data 726 volume after 2 years of operation of 128 GB. 727 728 \item The Metadata Database must respond to simple queries which 741 \begin{enumerate} 742 \item The IPP Metadata Database shall accept metadata from the summit 743 at a sustained rate of \tbr{1 MB per second}.\VER{TEST}{TLR:17, TLR:21, TLR:25} 744 745 \item The Metadata Database shall store the classes of data listed in 746 Table~\ref{metadata}. Thus, the Metadata Database shall store and 747 provide metadata for all raw images, for processed images, for the 748 calibration images (both raw and master), for the extracted object 749 lists. Metadata describing the environmental conditions at the 750 telescope shall also be stored and provided as needed. 751 Database.\VER{INSPECT}{TLR:21, TLR:25} 752 753 \item The Metadata Database queries shall have a latency of $< 0.1$ seconds.\VER{TEST}{TLR:17} 754 755 \item The Metadata Database shall be capable of at least 100 queries per second.\VER{TEST}{TLR:17} 756 757 \item The Metadata Database shall be capable of accepting a total data 758 volume after 2 years of operation of 280 GB. \VER{INSPECT}{TLR:25} 759 760 \item The Metadata Database shall respond to simple queries which 729 761 return the data in the categories listed in Table~\ref{metadata} 730 762 based on the primary data key and with basic constraints of time 731 ranges and other simple conditional constraints. 732 733 \item The Metadata muststore descriptive information about the raw763 ranges and other simple conditional constraints.\VER{TEST}{TLR:17} 764 765 \item The Metadata shall store descriptive information about the raw 734 766 images received from the summit and the current state of the data 735 processing. 736 737 \item The Metadata must also store descriptive information for each of 738 the static sky images currently available. 739 740 \item The IPP requires configuration information defining the 741 organization and configuration of the IPP itself. The Metadata 742 database must store the configuration information with restricted 743 access so that only specific people may change the information. 744 Examples of configuration data include the default parameters for 745 the various analysis programs, the description of the computing 746 environment, and the process status information, etc. 747 748 \item The Metadata Database must restrict access to the scientific 749 parameters to a different group from the software and hardware 750 configuration parameters. 751 752 \item In the discussion of the Analysis Stages below, various steps 753 specify that the values are user-configurable parameters. These 754 parameters must be stored in and extracted from the Metadata 755 Database. 767 processing.\TASK 768 769 \item The Metadata shall also store descriptive information for each of 770 the static sky images currently available.\TASK 771 772 \item Software configuration parameters shall be stored in and 773 extracted from the Metadata Database.\TASK 774 775 \item The Metadata database shall store the configuration information 776 with restricted access so that only specific people may change the 777 information.\VER{TEST}{allocated} 778 779 \item User-configurable software parameters shall be stored in and 780 extracted from the Metadata Database.\TASK 781 782 \item The Metadata Database shall restrict write access of the 783 scientific parameters to a different group from the software and 784 hardware configuration parameters.\VER{TEST}{allocated} 785 756 786 \end{enumerate} 757 787 … … 759 789 \begin{enumerate} 760 790 761 \item The IPP Controller mustmanage tasks on a cluster of up to 128762 computers. 763 764 \item On startup, the IPP Controller mustattempt to establish791 \item The IPP Controller shall manage tasks on a cluster of up to 128 792 computers.\VER{TEST}{TLR:17} 793 794 \item On startup, the IPP Controller shall attempt to establish 765 795 communication with all of its computers and set their state to be 766 {\tt alive} or {\tt dead} based on the success of the connection. 767 768 \item The IPP Controller mustdetect computers which crash or stop769 responding and set their state to {\tt dead}. 770 771 \item The IPP Controller mustattempt to re-establish communication772 with {\tt dead} computers. 773 774 \item The IPP Controller mustaccept tasks from external users and796 {\tt alive} or {\tt dead} based on the success of the connection.\VER{TEST}{TLR:17} 797 798 \item The IPP Controller shall detect computers which crash or stop 799 responding and set their state to {\tt dead}.\VER{TEST}{TLR:17} 800 801 \item The IPP Controller shall attempt to re-establish communication 802 with {\tt dead} computers.\VER{TEST}{TLR:17} 803 804 \item The IPP Controller shall accept tasks from external users and 775 805 systems, which may specify a desired CPU (node) and priority in 776 addition to the task command. 777 778 \item The IPP Controller mustattempt to run pending tasks on the779 desired node, if available (not {\tt dead} or {\tt off}). 780 781 \item If the node is unavailable, the IPP Controller mustattempt to782 run the task on another node. 783 784 \item If the node is available, the IPP Controller mustattempt to run806 addition to the task command.\VER{TEST}{TLR:17} 807 808 \item The IPP Controller shall attempt to run pending tasks on the 809 desired node, if available (not {\tt dead} or {\tt off}).\VER{TEST}{TLR:17} 810 811 \item If the node is unavailable, the IPP Controller shall attempt to 812 run the task on another node.\VER{TEST}{TLR:17} 813 814 \item If the node is available, the IPP Controller shall attempt to run 785 815 a given task only if no higher-priority tasks are available and no 786 task is currently being executed. 787 788 \item The IPP Controller mustmonitor the output from the task and789 write it to an associated log destination. 790 791 \item The IPP Controller mustmonitor the execution status of each816 task is currently being executed.\VER{TEST}{TLR:17} 817 818 \item The IPP Controller shall monitor the output from the task and 819 write it to an associated log destination.\VER{TEST}{TLR:17} 820 821 \item The IPP Controller shall monitor the execution status of each 792 822 task currently executing on a node and perform the following 793 823 actions: 794 824 795 825 \begin{enumerate} 796 \item identify the task as successful if it has a valid exit status. 797 \item identify the task as unsuccessful if it has an error exit 798 status. 799 \item identify the task as unattempted if the computer crashed. 826 \item identify the task as successful if it has a valid exit status.\VER{TEST}{TLR:17} 827 \item identify the task as unsuccessful if it has an error exit status.\VER{TEST}{TLR:17} 828 \item identify the task as unattempted if the computer crashed.\VER{TEST}{TLR:17} 800 829 \end{enumerate} 801 830 802 \item The IPP Controller mustaccept and perform the following831 \item The IPP Controller shall accept and perform the following 803 832 external commands: 804 833 \begin{enumerate} 805 \item add a task to the pending task list. 806 \item delete a specific task from the pending task list. 807 \item return the current status of a specific task. 808 \item return a list of all pending and non-pending tasks. 809 \item set a specified computer state to {\tt off} or {\tt dead}. 810 \item restrict a specified CPU to a class of tasks. 811 \item halt execution of a specified task. 812 \item set the IPP Controller state to {\tt finish}, {\tt abort}, or 813 {\tt stop}. 834 \item add a task to the pending task list.\VER{TEST}{TLR:17} 835 \item delete a specific task from the pending task list.\VER{TEST}{TLR:17} 836 \item return the current status of a specific task.\VER{TEST}{TLR:17} 837 \item return a list of all pending and non-pending tasks.\VER{TEST}{TLR:17} 838 \item set a specified computer state to {\tt off} or {\tt dead}.\VER{TEST}{TLR:17} 839 \item restrict a specified CPU to a class of tasks.\VER{TEST}{TLR:17} 840 \item halt execution of a specified task.\VER{TEST}{TLR:17} 841 \item set the IPP Controller state to {\tt finish}, {\tt abort}, or {\tt stop}.\VER{TEST}{TLR:17} 814 842 \end{enumerate} 843 844 \item The IPP Controller shall limit command latency to \tbr{$< 0.1$} seconds.\VER{TEST}{TLR:17} 845 846 \item The IPP Controller shall be capable of performing up to \tbr{10 tasks per second}.\VER{TEST}{TLR:17} 847 848 \item The IPP Controller shall be capable of buffering up to a total of \tbr{64 MB} of messages.\VER{TEST}{TLR:17} 849 850 \item The IPP Controller shall be capable of executing up to \tbr{6 million tasks per month}.\VER{TEST}{TLR:17} 851 852 \item The IPP Controller shall be capable of interacting with up to \tbr{256} client processes.\VER{TEST}{TLR:17} 853 854 \item The IPP Controller shall be capable of accepting up to 2 non-client (external) requests per second.\VER{TEST}{TLR:17} 815 855 \end{enumerate} 816 856 817 857 \subsubsection{Scheduler} 818 858 \begin{enumerate} 819 \item The IPP Scheduler mustsend the analysis tasks which it820 initiates to the IPP Controller. 821 822 \item All analysis tasks sent by the IPP Scheduler mustinclude a859 \item The IPP Scheduler shall send the analysis tasks which it 860 initiates to the IPP Controller.\VER{TEST}{TLR:17} 861 862 \item All analysis tasks sent by the IPP Scheduler shall include a 823 863 complete UNIX command with necessary arguments, the priority of the 824 task, and optionally the desired processing node. 825 826 \item The IPP Scheduler must refer to several input data sources to 827 decide what tasks to initiate. These data sources include the IPP 828 Metadata Database, the Summit Metadata Database, and User requests. 829 830 \item The IPP Scheduler must query the Databases on a regular basis to 831 check for new input information. These queries must take place at 832 least once every \tbr{5 seconds}. 833 834 \item The IPP Scheduler must accept new User input in real-time 835 (within 0.1 seconds of the request). 836 837 \item The IPP Scheduler must construct new tasks on the basis of the 838 inputs and a task dependency table. 864 task, and optionally the desired processing node.\VER{INSPECT}{TLR:17} 865 866 \item The IPP Scheduler shall query the Databases on a regular basis 867 to check for new input information. These queries shall take place 868 at least once every \tbr{1 seconds}.\VER{TEST}{TLR:17} 869 870 \item The IPP Scheduler shall accept new User input in real-time: 871 within 0.1 seconds of the request.\VER{TEST}{TLR:17} 839 872 840 873 \item When the IPP Scheduler is placed in the {\em paused state}, it 841 must only initiate User-requested tasks.874 shall only initiate User-requested tasks.\VER{TEST}{TLR:17} 842 875 843 876 \item When the IPP Scheduler is placed in the {\em interactive state}, 844 it mustinitiate User-requested tasks as well as data transfer845 tasks. 877 it shall initiate User-requested tasks as well as data transfer 878 tasks.\VER{TEST}{TLR:17} 846 879 847 880 \item When the IPP Scheduler is placed in the {\em automatic state}, 848 it must initiate the most appropriate task based on the inputs. 849 850 \item The IPP Scheduler must receive the exit status of tasks from the 851 IPP Controller. 852 853 \item The IPP Scheduler must send the exit status of the analysis 881 it shall initiate the most appropriate task based on the inputs and 882 dependency rules.\VER{TEST}{TLR:17} 883 884 \item The IPP Scheduler shall send the exit status of the analysis 854 885 tasks to the appropriate destination as defined by the task 855 dependency table. 886 dependency table.\VER{TEST}{TLR:17} 887 888 \item The IPP Scheduler shall publish the static sky images to the 889 Pan-STARRS PSPS on a time-scale of \tbr{6 month}.\VER{TEST}{TLR:19} 890 891 \item The IPP Scheduler shall publish the detected objects to the 892 Pan-STARRS PSPS on a time-scale of \tbr{1 month}.\VER{TEST}{TLR:20} 893 894 \item The IPP Scheduler shall publish the IPP and OTIS metadata to the 895 Pan-STARRS PSPS on a time-scale of \tbr{1 week}.\VER{TEST}{TLR:21} 896 897 \item The IPP Scheduler shall send the detected single-occurance 898 transient objects to the MOPS subsystem within 5 minutes of the 899 image exposure time.\VER{TEST}{TLR:22} 900 901 \item The IPP Scheduler shall send the metadata appropriate to the 902 images from which single-occurance transient objects were detected 903 to the MOPS subsystem within 5 minutes of the image exposure 904 time.\VER{TEST}{TLR:22} 905 856 906 \end{enumerate} 857 907 858 908 \subsection{Analysis Stages} 859 909 860 We now consider the requirements of the analysis tasks which mustbe910 We now consider the requirements of the analysis tasks which shall be 861 911 performed by the IPP. These tasks represent the core of the required 862 912 IPP functionality; the architectural components discussed above can be … … 868 918 The Science Image analysis stages together represent the basic data 869 919 analysis required by the IPP. There are several requirements which 870 mustbe met by the collection of science image analysis stages as a920 shall be met by the collection of science image analysis stages as a 871 921 group. 872 922 873 923 \begin{enumerate} 874 \item The science image analysis stages must perform their analysis 875 quickly enough to keep up with the incoming data stream. The 876 required processing time is derived from the rate at which science 877 images are obtained by PS-1. 878 879 \item At a minimum, the Science Image Analysis must keep up with the 880 average image rate over the course of 1 day. 881 882 \item In order to provide a sufficient buffer for variations in the 883 processing speed, the Science Image Analysis must be able to process 884 all images from a night within 12 hours. 924 \item The IPP Science Analysis shall pre-process the science images 925 with the master calibration images at a sustained rate of 1 exposure 926 (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17} 927 928 \item The IPP Science Analysis shall merge multiple pre-processed 929 science images into stacked images with corresponding signal-to-noise 930 maps at a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17} 931 932 \item The IPP Science Analysis shall excise pixels from the input 933 images which are outliers for the ensemble of corresponding pixels 934 with an efficiency of $> 99$\%.\VER{ANALYSIS}{TLR:18} 935 936 \item The IPP Science Analysis shall merge the cleaned images into the 937 static sky image, and update the corresponding exposure (S/N) maps, 938 at a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17} 939 940 \item The IPP Science Analysis shall detect and measure parameters of 941 objects on the pre-processed science images.\VER{TEST}{TLR:12} 942 943 \item The IPP Science Analysis shall detect and measure parameters of 944 objects on the stacked science images.\VER{TEST}{TLR:13} 945 946 \item The IPP Science Analysis shall detect and measure parameters of 947 objects on the static sky images.\VER{TEST}{TLR:14} 948 949 \item The IPP Science Analysis shall detect and measure parameters of 950 objects on the difference images.\VER{TEST}{TLR:15} 951 952 \item The IPP Science Analysis shall determine astrometry of the 953 detected objects relative to an external astrometric reference with 954 an accuracy of \tbr{750 mas} (for bright objects) in the 955 Commissioning phase of the telescope.\VER{TEST}{TLR:4, TLR:3} 956 957 \item The IPP Science Analysis shall determine astrometry of the 958 detected objects relative to an external astrometric reference with 959 an accuracy of \tbr{250 mas} (for bright objects) during the 960 construction of the Pan-STARRS Astrometric Reference Catalog.\VER{ANALYSIS}{TLR:4, TLR:3} 961 962 \item The IPP Science Analysis shall determine astrometry of the 963 detected objects relative to the Pan-STARRS Astrometric Reference 964 with an accuracy of \tbr{100 mas} (for bright objects) during normal 965 operations.\VER{ANALYSIS}{TLR:4, TLR:3} 966 967 \item The IPP Science Analysis shall determine photometry of the 968 detected objects within an internal photometric system with scatter 969 less than \tbr{25 millimags} (for bright objects) during the 970 Commissioning phase of the telescope in photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2} 971 972 \item The IPP Science Analysis shall determine photometry of the 973 detected objects within an internal photometric system with scatter 974 less than \tbr{10 millimags} (for bright objects) during the 975 construction of the Pan-STARRS Photometric Reference Catalog in 976 photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2} 977 978 \item The IPP Science Analysis shall determine photometry of the 979 detected objects within an internal photometric system with scatter 980 less than \tbr{5 millimags} (for bright objects) during normal 981 operations in photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2} 982 983 \item The IPP Science Analysis shall determine photometry of the 984 detected objects in an external photometric system with scatter less 985 than \tbr{10 millimags} (for bright objects) during normal operations 986 in photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2} 885 987 886 988 \item The maximum latency between the acquisition of an image and the 887 989 completion of the science image analysis is set by the science 888 990 requirements of the fast transient recovery programs. The science 889 image analysis must process images from these observing programs890 within \tbr{5 min} of their arrival time in the IPP Image Server.891 892 \item The science image analysis stages mustprocesses up to 1000893 science images per night. 991 image analysis shall process images to detection transients within 992 \tbr{5 min} of their acquisition.\VER{TEST}{TLR:22} 993 994 \item The science image analysis stages shall processes up to 1000 995 science images per night.\VER{TEST}{TLR:17} 894 996 895 997 \end{enumerate} … … 898 1000 899 1001 \begin{enumerate} 900 \item The Phase 1 analysis stage must determine the astrometric 1002 \item the Phase 1 analysis shall execute within 2 seconds for a 1003 complete FPA image.\VER{TEST}{TLR:17} 1004 1005 \item The Phase 1 analysis stage shall determine the astrometric 901 1006 solution of the complete camera (FPA image) with an accuracy of 902 \tbr{1 arcsec} peak-to-peak deviation. 903 904 \item The Phase 1 analysis stage must load the guide star pixel and 905 celestial coordinates from the \tbd{IPP Metadata 906 Database}\comment{or from the image header?}. 1007 \tbr{1 arcsec} peak-to-peak deviation.\VER{TEST}{TLR:3} 1008 1009 \item The Phase 1 analysis stage shall load the guide star pixel and 1010 celestial coordinates.\TASK 907 1011 908 1012 \item If guide stars are not available, the Phase 1 analysis stage 909 must extract bright stars from the image.910 911 \item This extraction must be done in less than \tbr{1 second}.1013 shall extract bright stars from the image.\TASK 1014 1015 \item This extraction shall be done in less than \tbr{1 second}.\VER{TEST}{TLR:17} 912 1016 913 1017 \item The total number of stars and size of the bright-star 914 acquisition box must be a user-configurable parameter. 1018 acquisition box shall be a user-configurable parameter in the range 1019 20 - 250.\TASK 915 1020 916 1021 \item In order for blind astrometry of an image to succeed, it is 917 1022 necessary that approximate image coordinates be known. The Phase 1 918 analysis mustbe able to succeed despite initial coordinate errors919 as large as \tbr{20\arcsec}. 920 921 \item The Phase 1 analysis stage mustconstruct a table of the1023 analysis shall be able to succeed despite initial coordinate errors 1024 as large as \tbr{20\arcsec}.\VER{TEST}{TLR:3} 1025 1026 \item The Phase 1 analysis stage shall construct a table of the 922 1027 overlaps between the science image to be processed and the static 923 sky images. 924 925 \item The overlaps mustbe overestimated by a small amount so that1028 sky images.\TASK 1029 1030 \item The overlaps shall be overestimated by a small amount so that 926 1031 errors in astrometry at Phase 1 will not cause any valid static sky 927 / science image pairs to be missed. 928 929 \item The amount of overlap must be a user-configurable parameter.1032 / science image pairs to be missed.\TASK 1033 1034 \item The amount of overlap shall be a user-configurable parameter.\VER{TEST}{TLR:6, TLR:11} 930 1035 931 1036 \item Sky cells which do not have sufficient science image overlap 932 \tb d{$< 5\%$} must be excluded from the overlap table.1037 \tbr{$< 5\%$} shall be excluded from the overlap table.\VER{TEST}{TLR:6, TLR:11} 933 1038 934 1039 \item It is not unusual for an image to be obtained with invalid … … 936 1041 control system may make an error and report the wrong time or 937 1042 coordinates. Or, the image may be obtained in exceptionally poor 938 conditions with no detected stars. Phase 1 mustreturn a939 descriptive error message in these conditions. 1043 conditions with no detected stars. Phase 1 shall return a 1044 descriptive error message in these conditions.\TASK 940 1045 \end{enumerate} 941 1046 … … 945 1050 the detector are processed to remove instrumental signatures. 946 1051 1052 \paragraph{Timing} 1053 The complete Phase~2 analysis shall be performed in $< 38$ seconds for 1054 up to 4 complete FPA images at one time. \VER{TEST}{TLR:17} 1055 947 1056 \paragraph{Processing Recipe} 948 1057 \begin{enumerate} 949 \item The Phase 2 analysis stage must consult the processing recipe to 950 define the necessary analysis steps performed by the Phase 2 stage. 951 952 \item Phase 2 must perform the analysis steps only if required by the 953 processing recipe. 954 955 \item The processing recipe must define the stages to be executed with 1058 \item The Phase 2 analysis stage shall consult the processing recipe 1059 to define the necessary analysis steps performed by the Phase 2 1060 stage.\TASK 1061 1062 \item Phase 2 shall perform the analysis steps only if required by the 1063 processing recipe.\TASK 1064 1065 \item The processing recipe shall define the stages to be executed with 956 1066 optional exposure time and background flux limits to require or 957 exclude select certain stages. 1067 exclude select certain stages.\TASK 958 1068 \end{enumerate} 959 1069 … … 961 1071 \begin{enumerate} 962 1072 963 \item The Phase 2 analysis stage must determine the OT kernel from the964 IPP Metadata Database\comment{or image header}.965 966 \item The Phase 2 analysis stage must convolve the flat-field and967 high-spatial-frequency fringe images with the OT kernel.968 969 \item If no OT kernel exists, this step must be silently skipped.1073 \item The Phase 2 analysis stage shall convolve the flat-field and 1074 high-spatial-frequency fringe images with the OT kernel.\VER{TEST}{TLR:1} 1075 1076 \item The Phase 2 analysis stage shall determine the OT kernel from the 1077 IPP Metadata Database.\TASK 1078 1079 \item If no OT kernel exists, this step shall be silently skipped.\TASK 970 1080 \end{enumerate} 971 1081 … … 973 1083 \begin{enumerate} 974 1084 975 \item The Phase 2 analysis must load the basic bad pixel map appropriate to 976 the detector of interest. 977 978 \item The Phase 2 analysis must use the OT kernel to grow the traps in the 979 raw bad pixel map. 980 981 \item The Phase 2 analysis must mask saturated pixels and a user-specified 982 number of surrounding pixels. 983 984 \item Different bits must be set to identify different reasons for masking 985 the pixels. 1085 \item The Phase 2 analysis shall load the basic bad pixel map appropriate to 1086 the detector of interest.\VER{TEST}{TLR:18} 1087 1088 \item The Phase 2 analysis shall use the OT kernel to grow the traps in the 1089 raw bad pixel map. \VER{TEST}{TLR:18} 1090 1091 \item The Phase 2 analysis shall mask saturated pixels and a user-specified 1092 number of surrounding pixels.\VER{TEST}{TLR:18} 1093 1094 \item The Phase 2 analysis shall mask ghosts of bright stars.\VER{TEST}{TLR:18} 1095 1096 \item Different bits shall be set to identify different reasons for masking 1097 the pixels.\VER{TEST}{TLR:21} 986 1098 \end{enumerate} 987 1099 … … 989 1101 \begin{enumerate} 990 1102 991 \item Phase 2 must perform bias subtraction on the image. 992 993 \item Phase 2 must choose the bias subtraction method and analysis statistic 994 based on the user-configured parameters. 995 996 \item The bias correction must be measured from the image overscan region. 997 998 \item The overscan region must be determined from the image 999 header\comment{or Metadata DB}. 1000 1001 \item The bias subtraction must apply one of the following bias corrections, 1002 depending on the user parameters: 1003 1004 \begin{enumerate} 1005 \item subtract a single constant from the image. 1006 1007 \item subtract a 1-D bias which varies along the overscan. The function to be used must include 1103 \item Phase 2 shall perform bias subtraction on the image.\VER{TEST}{TLR:1} 1104 1105 \item Phase 2 shall choose the bias subtraction method and analysis statistic 1106 based on the user-configured parameters.\TASK 1107 1108 \item The bias correction shall be measured from the image overscan region.\TASK 1109 1110 \item The overscan region shall be determined from the Metadata DB.\TASK 1111 1112 \item The bias subtraction shall be capable of using one of following 1113 bias corrections, depending on the user parameters: 1114 1115 \begin{enumerate} 1116 \item subtract a single constant from the image. \VER{TEST}{TLR:1} 1117 1118 \item subtract a 1-D bias which varies along the overscan. The function to be used shall include 1008 1119 a spline or a Chebychev polynomial derived from the data values along 1009 the overscan, as specified by the user parameters. 1120 the overscan, as specified by the user parameters. \VER{TEST}{TLR:1} 1010 1121 1011 1122 \item correct the overscan {\em and} subtract a 2-D bias image which 1012 has been overscan corrected using one of the two methods above. 1123 has been overscan corrected using one of the two methods above.\VER{TEST}{TLR:1} 1013 1124 \end{enumerate} 1014 1125 1015 1126 \item The statistic used to calculate the overscan constant or the 1016 inputs to the spline and polynomial fits mustbe derived from groups1127 inputs to the spline and polynomial fits shall be derived from groups 1017 1128 of pixels on the basis of one of several possible statistics, as 1018 specified by the user parameters. 1019 1020 \item The choice of statistics mustinclude the sample and robust1021 mean, median, and modes. 1129 specified by the user parameters.\VER{TEST}{TLR:1} 1130 1131 \item The choice of statistics shall include the sample and robust 1132 mean, median, and modes.\VER{TEST}{TLR:1} 1022 1133 1023 1134 \item In the case of a single constant, all of the overscan pixel 1024 values are used in the calculation of this statistic. 1135 values are used in the calculation of this statistic.\VER{TEST}{TLR:1} 1025 1136 1026 1137 \item In the case of the 1D functional representation, the input 1027 values to the fit mustrepresent the coordinate along the overscan,1138 values to the fit shall represent the coordinate along the overscan, 1028 1139 with the statistic derived from the pixels in the perpendicular 1029 direction at each location. 1030 1031 \item If specified in the user parameters, sigma-clipping mustbe1032 performed on the input data values. 1033 1034 The bias subtraction mustleave no residuals greater than \tbr{1 DN}1035 peak-to-peak. 1140 direction at each location.\VER{TEST}{TLR:1} 1141 1142 \item If specified in the user parameters, sigma-clipping shall be 1143 performed on the input data values.\VER{TEST}{TLR:1} 1144 1145 \item The bias subtraction shall leave no residuals greater than \tbr{1 DN} 1146 peak-to-peak.\VER{TEST}{TLR:1} 1036 1147 \end{enumerate} 1037 1148 … … 1039 1150 \begin{enumerate} 1040 1151 1041 \item The Phase 2 analysis musttrim the non-imaging pixels from the1042 image. 1043 1044 \item The definition of the imaging area mustbe determined from the1045 Metadata Database \comment{or image header?}.1046 1047 \item Phase 2 musttrim pixel near the edges that have been1048 compromised due to OT operation. 1152 \item The Phase 2 analysis shall trim the non-imaging pixels from the 1153 image.\TASK 1154 1155 \item The definition of the imaging area shall be determined from the 1156 Metadata Database.\TASK 1157 1158 \item Phase 2 shall trim pixel near the edges that have been 1159 compromised due to OT operation.\VER{TEST}{TLR:1} 1049 1160 \end{enumerate} 1050 1161 1051 1162 \paragraph{Correct for non-linearity} 1052 1163 1053 If required by the recipe, each chip mustbe independently corrected for the1054 effects of non-linearity. 1164 If required by the recipe, each chip shall be independently corrected for the 1165 effects of non-linearity.\VER{TEST}{TLR:1} 1055 1166 1056 1167 \paragraph{Flat-field correction} 1057 1168 \begin{enumerate} 1058 1169 1059 \item The Phase 2 analysis mustdivide the science image by the1060 provided flat-field image. 1061 1062 \item The division musthandle zero-valued pixels in the flat-field1170 \item The Phase 2 analysis shall divide the science image by the 1171 provided flat-field image.\VER{TEST}{TLR:1} 1172 1173 \item The division shall handle zero-valued pixels in the flat-field 1063 1174 image without raising floating point exceptions, setting the 1064 corresponding bit value in the mask. 1065 1066 \item The flat-field images mustbe appropriately normalized (see1067 section \ref{mkcal}). 1068 1069 \item The flat-fielded image musthave a consistent photometric1175 corresponding bit value in the mask.\VER{TEST}{TLR:1} 1176 1177 \item The flat-field images shall be appropriately normalized (see 1178 section \ref{mkcal}).\VER{TEST}{TLR:1} 1179 1180 \item The flat-fielded image shall have a consistent photometric 1070 1181 zero-point across the chip, and across the full FPA, to within 0.2\% 1071 with peak-to-peak deviations of \tbr{0.5\%}. 1072 \end{enumerate} 1073 1074 \tbd{color of stars in flat-field correction?} 1182 with peak-to-peak deviations of \tbr{0.5\%}.\VER{TEST}{TLR:1} 1183 \end{enumerate} 1075 1184 1076 1185 \paragraph{Sky \& Fringe subtraction} 1077 1186 \begin{enumerate} 1078 1187 1079 \item The Phase 2 analysis mustsubtract the sky (and fringe where1080 needed) contributions from the images. 1081 1082 \item The residual after the background subtraction musthave an1188 \item The Phase 2 analysis shall subtract the sky (and fringe where 1189 needed) contributions from the images.\VER{TEST}{TLR:1, TLR:5} 1190 1191 \item The residual after the background subtraction shall have an 1083 1192 average offset of 0 counts, excluding the signal from astronomical 1084 features. 1085 1086 \item The background residuals musthave peak-to-peak variations of1087 less than \tbr{1\%} of the input background amplitude. 1088 1089 \item The background residuals musthave a scatter of less than1193 features.\VER{TEST}{TLR:5} 1194 1195 \item The background residuals shall have peak-to-peak variations of 1196 less than \tbr{1\%} of the input background amplitude.\VER{TEST}{TLR:5} 1197 1198 \item The background residuals shall have a scatter of less than 1090 1199 \tbr{1\%} of the input background amplitude for apertures of less 1091 than \tbr{10~arcsec}.\comment{derived from the need for systematic 1092 errors of better than 0.5\% and known background ranges.} 1200 than \tbr{10~arcsec}.\VER{TEST}{TLR:1} 1093 1201 \end{enumerate} 1094 1202 … … 1096 1204 \begin{enumerate} 1097 1205 1098 \item The Phase 2 analysis must detect cosmic rays in single images 1099 which are brighter than a user-configurable threshold. 1100 1101 \item The Phase 2 analysis must mask detected cosmic rays with a 1102 unique bit value in the mask. 1103 1104 \item The Phase 2 analysis must extend the masked region by a 1105 user-configurable growth factor. 1106 1107 \item The Phase 2 analysis must perform the cosmic ray detection only 1108 if it is required by the analysis recipe. 1206 \item The Phase 2 analysis shall detect cosmic rays with flux $> 1207 5\sigma$ by morphology in single images with an efficiency of $> 95$\%. 1208 \VER{TEST}{TLR:18} 1209 1210 \item The Phase 2 analysis shall mask detected cosmic rays with a 1211 unique bit value in the mask.\TASK 1212 1213 \item The Phase 2 analysis shall extend the masked region by a 1214 user-configurable growth factor.\TASK 1215 1216 \item The Phase 2 analysis shall perform the cosmic ray detection only 1217 if it is required by the analysis recipe.\TASK 1109 1218 \end{enumerate} 1110 1219 … … 1112 1221 \begin{enumerate} 1113 1222 1114 \item The Phase 2 analysis mustperform object detection on the1115 processed images. 1116 1117 \item The object detection process mustdetect all objects above a1118 user-configured threshold. 1119 1120 \item The threshold must be a positive value; negative values must1121 invoke an error. 1122 1123 \item The detection threshold mustoptionally be a function of the1124 average background flux or the local noise level. 1125 1126 \item The object detection mustmeasure the following object1223 \item The Phase 2 analysis shall perform object detection on the 1224 processed images.\VER{TEST}{TLR:12} 1225 1226 \item The object detection process shall detect all objects above a 1227 user-configured threshold.\TASK 1228 1229 \item The threshold shall be a positive value; negative values shall 1230 invoke an error.\TASK 1231 1232 \item The detection threshold shall optionally be a function of the 1233 average background flux or the local noise level.\TASK 1234 1235 \item The object detection shall measure the following object 1127 1236 parameters: 1128 1237 \begin{enumerate} 1129 \item object centroid and position errors 1130 \item an extended object position ($x_g, y_g$) 1131 \item instrumental PSF magnitude and error 1132 \item local background level and error 1238 \item object centroid and position errors\VER{TEST}{TLR:12} 1239 \item an extended object position ($x_g, y_g$)\VER{TEST}{TLR:12} 1240 \item instrumental PSF magnitude and error\VER{TEST}{TLR:12} 1241 \item local background level and error\VER{TEST}{TLR:12} 1133 1242 \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) of the object 1134 and their covariance matrix 1243 and their covariance matrix\VER{TEST}{TLR:12} 1135 1244 \end{enumerate} 1136 1245 1137 \item Minimal object classification mustbe performed to distinguish1246 \item Minimal object classification shall be performed to distinguish 1138 1247 objects which are consistent with a single PSF, objects which are 1139 1248 inconsistently large, objects which are inconsistently small, and 1140 objects which are saturated. 1141 1142 \item The resulting collection of detected objects mustbe saved along1143 with the relevant image metadata (\ie filter, exposure time, etc). 1249 objects which are saturated.\VER{TEST}{TLR:12} 1250 1251 \item The resulting collection of detected objects shall be saved along 1252 with the relevant image metadata (\ie filter, exposure time, etc).\VER{TEST}{TLR:20} 1144 1253 \end{enumerate} 1145 1254 … … 1147 1256 \begin{enumerate} 1148 1257 1149 \item The Phase 2 analysis mustmatch the detected objects with known1150 astrometric reference objects. 1151 1152 \item The astrometric reference object coordinates mustbe adjusted1153 for proper motion. 1154 1155 \item The reference and detected object coordinates mustbe fit to1156 determine astrometric parameters for the individual OTAs. 1157 1158 \item The OTA astrometric parameters must include Chebychev1159 polynomials of the coordinates up to 3rd order. 1160 1161 \item The fitted number of polynomial orders mustbe a user-configured1162 parameter. 1163 1164 \item The Cell astrometric parameters mustnot be allowed to vary in1165 the fit. 1166 1167 \item The fit mustbe robust, rejecting outlier matches (either stars1168 with poorly determined proper motion or spurious matches). 1169 1170 \item The resulting astrometric solution mustbe consistent across the1171 OTA field to within \tbr{ 300 milli-arcsec}.1258 \item The Phase 2 analysis shall match the detected objects with known 1259 astrometric reference objects.\VER{TEST}{TLR:3} 1260 1261 \item The astrometric reference object coordinates shall be adjusted 1262 for proper motion.\VER{TEST}{TLR:3} 1263 1264 \item The reference and detected object coordinates shall be fit to 1265 determine astrometric parameters for the individual OTAs.\VER{TEST}{TLR:3} 1266 1267 \item The OTA astrometric parameters shall include polynomials of the 1268 coordinates up to 3rd order.\VER{TEST}{TLR:3} 1269 1270 \item The fitted number of polynomial orders shall be a user-configured 1271 parameter.\TASK 1272 1273 \item The Cell astrometric parameters shall not be allowed to vary in 1274 the fit.\VER{}{} 1275 1276 \item The fit shall be robust, rejecting outlier matches (either stars 1277 with poorly determined proper motion or spurious matches).\VER{TEST}{TLR:3} 1278 1279 \item The resulting astrometric solution shall be consistent across the 1280 OTA field to within \tbr{100 milli-arcsec}.\VER{TEST}{TLR:4} 1172 1281 \end{enumerate} 1173 1282 … … 1175 1284 \begin{enumerate} 1176 1285 1177 \item The Phase 2 analysis mustextract subrasters (`postage stamps')1286 \item The Phase 2 analysis shall extract subrasters (`postage stamps') 1178 1287 surrounding a user-specified list of coordinates from the flattened 1179 images. 1180 1181 \item The postage stamp images must be saved in the IPP Image Server.1288 images.\VER{TEST}{TLR:12} 1289 1290 \item The postage stamp images shall be saved in the IPP Image Server.\VER{TEST}{TLR:12} 1182 1291 \end{enumerate} 1183 1292 … … 1185 1294 \begin{enumerate} 1186 1295 1187 \item The Phase 3 analysis mustuse the objects detected in Phase 2,1296 \item The Phase 3 analysis shall use the objects detected in Phase 2, 1188 1297 matched with a user-specified reference photometry catalog, to 1189 1298 determine the image photometric zero point and zero-point variations 1190 across the field. 1191 1192 \item If zero-point variations are significant \tbd{level TBD}, the1193 zero-point variations must be modeled with a Chebychev polynomial1194 correction of order 3 or less.1195 1196 \item The photometric nature of the FPA image must be categorized1197 \tbd{numerical scale?} on the basis of the zero-point consistency,1198 the transparency compared with recent long-term measurements in the1199 filter, and the external indicators of photometricity.1200 1201 \item The Phase 3 analysis mustuse the objects detected in Phase 2,1299 across the field.\VER{TEST}{??} 1300 1301 \item If zero-point variations are significant (\tbr{$> 0.01$ mag 1302 peak-to-peak}), the zero-point variations shall be modeled with a 1303 polynomial correction of order 3 or less.\VER{TEST}{TLR:1} 1304 1305 \item The photometric nature of the FPA image shall be categorized on 1306 the basis of the zero-point consistency, the transparency compared 1307 with recent long-term measurements in the filter, and the external 1308 indicators of photometricity.\VER{TEST}{TLR:2} 1309 1310 \item The Phase 3 analysis shall use the objects detected in Phase 2, 1202 1311 matched with an appropriate astrometric reference catalog, to 1203 improve the distortion model used for the image. 1204 1205 \item The resulting astrometric accuracy must be limited by the 1206 astrometric reference catalog, ie, 250 mas for USNO-B1.0. 1312 improve the distortion model used for the image.\VER{TEST}{TLR:3} 1313 1314 \item The resulting astrometric accuracy shall be consistent across 1315 the field to 30 mas.\VER{TEST}{TLR:4} 1316 1317 \item The resulting astrometric accuracy shall be limited by the 1318 astrometric reference catalog, (eg, 100 - 250 mas for 1319 USNO-B1.0).\VER{TEST}{TLR:3} 1320 1321 \item The Phase 3 analysis shall modify the background correction of 1322 Phase 2 based on the full-field statistics to achieve an accuracy of 1\% 1323 of the background.\VER{TEST}{TLR:5} 1324 1325 \item The complete Phase~3 analysis shall be performed in $< 2$ 1326 seconds for up to 4 complete FPA images at one time. \VER{TEST}{TLR:17} 1327 1207 1328 \end{enumerate} 1208 1329 … … 1217 1338 \begin{enumerate} 1218 1339 1219 \item The Phase 4 analysis mustdetermine the corresponding set of1220 image pixels for a given sky cell. 1221 1222 \item The corresponding image pixels mustbe extracted from the input1340 \item The Phase 4 analysis shall determine the corresponding set of 1341 image pixels for a given sky cell.\TASK 1342 1343 \item The corresponding image pixels shall be extracted from the input 1223 1344 images, using the astrometric information for each OTA and Cell to 1224 determine the exact overlaps. 1225 1226 \item The Phase 4 analysis mustnot miss any pixels in this match, and1227 it mustread no more than 20\% more pixels than necessary from the1228 input images. 1229 1230 \item The Phase 4 analysis mustskip any sky cells with fewer than 5\%1231 of their pixels overlapping the input images. 1345 determine the exact overlaps.\TASK 1346 1347 \item The Phase 4 analysis shall not miss any pixels in this match, and 1348 it shall read no more than 20\% more pixels than necessary from the 1349 input images.\VER{TEST}{TLR:17} 1350 1351 \item The Phase 4 analysis shall skip any sky cells with fewer than 5\% 1352 of their pixels overlapping the input images.\VER{TEST}{TLR:17} 1232 1353 \end{enumerate} 1233 1354 … … 1235 1356 \begin{enumerate} 1236 1357 1237 \item Pixels which have been extracted from the input images mustbe1238 mapped to the corresponding pixels in the sky image. 1239 1240 \item The transformation mustbe based on the measured astrometric1358 \item Pixels which have been extracted from the input images shall be 1359 mapped to the corresponding pixels in the sky image.\TASK 1360 1361 \item The transformation shall be based on the measured astrometric 1241 1362 solution for the input images relative to the reference catalog used 1242 to generate the static sky image. 1243 1244 \item This warping must use a locally-linear astrometric solution.1363 to generate the static sky image.\VER{TEST}{TLR:3} 1364 1365 \item This warping shall use a locally-linear astrometric solution.\VER{TEST}{TLR:17} 1245 1366 1246 \item The output image must maintain photometric consistency with the 1247 input image to within 0.2\%. 1248 \end{enumerate} 1249 1250 \tbd{interpolation? does interpolation method choice risk losing flux?} 1367 \item The output image shall maintain photometric consistency with the 1368 input image to within 0.2\%.\VER{TEST}{TLR:1} 1369 \end{enumerate} 1251 1370 1252 1371 \paragraph{Flux matching} 1253 1372 1254 The Phase 4 analysis must determine appropriate photometry scaling 1255 factors needed to combine the images photometrically. 1256 1257 \tbd{is flux matched automatically by calibration?} 1373 The Phase 4 analysis shall determine appropriate photometry scaling 1374 factors needed to combine the images photometrically.\TASK 1258 1375 1259 1376 \paragraph{Image outlier pixel rejection} … … 1261 1378 1262 1379 \item When multiple images are combined, the group of input pixels 1263 which contribute to an output pixel must be examined and pixels from 1264 the group of images which are inconsistent with the ensemble 1265 \tbd{how much?} must be identified and flagged. 1266 1267 \item This outlier rejection must be performed optionally. 1268 1269 \tbd{for moving objects and images which are not simultaneous, do we 1270 identify the moving objects?} 1271 1272 \tbd{use the spatial information? fit a 2-D Nth order polynomial to 1273 the collection of pixels and then look for outliers} 1380 which contribute to an output pixel shall be examined and pixels from 1381 the group of images which are inconsistent with the ensemble (by an 1382 amount defined by the user-configurable parameters) shall be 1383 identified and flagged.\VER{TEST}{TLR:18} 1384 1385 \item This outlier rejection shall be performed optionally.\TASK 1386 1274 1387 \end{enumerate} 1275 1388 1276 1389 \paragraph{Initial cleaned image} 1277 1390 1278 The resulting collection of pixels mustbe used to construct a single1279 output image, cleaned of the outliers. 1391 The resulting collection of pixels shall be used to construct a single 1392 output image, cleaned of the outliers.\VER{TEST}{TLR:18} 1280 1393 1281 1394 \paragraph{PSF matching} 1282 1395 1283 The cleaned, combined image must be PSF matched with the static sky image.1396 The cleaned, combined image shall be PSF matched with the static sky image.\VER{TEST}{TLR:15} 1284 1397 1285 1398 \paragraph{Image Subtraction} 1286 1399 1287 The static sky image must be subtracted from the stacked, cleaned 1288 image. 1289 1290 \tbd{what about different stellar colors?} 1400 The static sky image shall be subtracted from the stacked, cleaned 1401 image. \VER{TEST}{TLR:15} 1291 1402 1292 1403 \paragraph{Find objects in the image} 1293 1404 \begin{enumerate} 1294 1405 1295 \item The Phase 4 analysis mustperform object detection on the1296 difference images. 1297 1298 \item All objects in the difference image mustbe detected and the1299 pixels belonging to variable sources flagged in the input image. 1300 1301 \item The object detection mustdetect all objects above a1302 user-configured threshold. 1303 1304 \item Both positive and negative objects mustbe detected: the1305 specified threshold mustdefine the absolute value of the detection1306 thresholds. 1307 1308 \item The detection threshold mustoptionally be a function of the1309 average background flux or the local noise level. 1310 1311 \item The object detection mustmeasure the following object parameters:1406 \item The Phase 4 analysis shall perform object detection on the 1407 difference images.\VER{TEST}{TLR:15} 1408 1409 \item All objects in the difference image shall be detected and the 1410 pixels belonging to variable sources flagged in the input image.\VER{TEST}{TLR:15} 1411 1412 \item The object detection shall detect all objects above a 1413 user-configured threshold.\VER{TEST}{TLR:15} 1414 1415 \item Both positive and negative objects shall be detected: the 1416 specified threshold shall define the absolute value of the detection 1417 thresholds.\VER{TEST}{TLR:15} 1418 1419 \item The detection threshold shall optionally be a function of the 1420 average background flux or the local noise level.\VER{TEST}{TLR:15} 1421 1422 \item The object detection shall measure the following object parameters: 1312 1423 \begin{enumerate} 1313 \item object centroid and position errors 1314 \item instrumental PSF magnitude and error 1315 \item local background level and error 1316 \item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$ 1317 \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix 1424 \item object centroid and position errors\VER{TEST}{TLR:15} 1425 \item instrumental PSF magnitude and error\VER{TEST}{TLR:15} 1426 \item local background level and error\VER{TEST}{TLR:15} 1427 \item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$\VER{TEST}{TLR:15} 1428 \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix\VER{TEST}{TLR:15} 1318 1429 \end{enumerate} 1319 1430 1320 \item Minimal object classification mustbe performed to distinguish1431 \item Minimal object classification shall be performed to distinguish 1321 1432 objects which are consistent with a single PSF, objects which are 1322 inconsistent, and objects which are saturated. 1323 1324 \item The resulting collection of detected objects mustbe saved along1325 with the relevant image metadata (\ie filter, exposure time, etc). 1433 inconsistent, and objects which are saturated.\VER{TEST}{TLR:15, TLR:18} 1434 1435 \item The resulting collection of detected objects shall be saved along 1436 with the relevant image metadata (\ie filter, exposure time, etc).\VER{TEST}{TLR:22} 1326 1437 \end{enumerate} 1327 1438 … … 1330 1441 1331 1442 \item The pixels flagged as being from the difference image sources 1332 must be masked in the input images. 1333 1334 \item A new, cleaned image must be constructed from the masked input 1335 images. 1336 1337 \end{enumerate} 1338 1339 \tbd{how to handle variable stars?} 1443 shall be masked in the input images.\VER{TEST}{TLR:6, TLR:11} 1444 1445 \item A new, cleaned image shall be constructed from the masked input 1446 images.\VER{TEST}{TLR:6, TLR:11} 1447 1448 \end{enumerate} 1340 1449 1341 1450 \paragraph{Find objects in the image} 1342 1451 \begin{enumerate} 1343 1452 1344 \item The Phase 4 analysis mustperform object detection on the1345 cleaned, summed image. 1346 1347 \item The object detection mustdetect all objects above a1348 user-configured threshold. 1349 1350 \item The threshold must be a positive value; negative values must1351 invoke an error. 1352 1353 \item The detection threshold optionally mustbe a function of the1354 average background flux or the local noise level. 1355 1356 \item The object detection mustmeasure the following object parameters:1453 \item The Phase 4 analysis shall perform object detection on the 1454 cleaned, summed image.\VER{TEST}{TLR:13} 1455 1456 \item The object detection shall detect all objects above a 1457 user-configured threshold.\VER{TEST}{TLR:13} 1458 1459 \item The threshold shall be a positive value; negative values shall 1460 invoke an error.\VER{TEST}{TLR:13} 1461 1462 \item The detection threshold optionally shall be a function of the 1463 average background flux or the local noise level.\VER{TEST}{TLR:13} 1464 1465 \item The object detection shall measure the following object parameters: 1357 1466 \begin{enumerate} 1358 \item object centroid and position errors 1359 \item an extended object position ($x_g, y_g$) 1360 \item instrumental PSF magnitude and error 1361 \item local background level and error 1467 \item object centroid and position errors\VER{TEST}{TLR:13} 1468 \item an extended object position ($x_g, y_g$)\VER{TEST}{TLR:13} 1469 \item instrumental PSF magnitude and error\VER{TEST}{TLR:13} 1470 \item local background level and error\VER{TEST}{TLR:13} 1362 1471 \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their 1363 covariance matrix 1364 \item the Petrosian radius, magnitude, axis ratio, and angle 1365 \item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$. 1472 covariance matrix\VER{TEST}{TLR:13} 1473 \item the Petrosian radius, magnitude, axis ratio, and angle\VER{TEST}{TLR:13} 1474 \item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$.\VER{TEST}{TLR:13} 1366 1475 \end{enumerate} 1367 1476 1368 \item Minimal object classification mustbe performed to distinguish1477 \item Minimal object classification shall be performed to distinguish 1369 1478 objects which are consistent with a single PSF, objects which are 1370 inconsistent, and objects which are saturated. 1371 1372 \item The resulting collection of detected objects mustbe saved along1373 with the relevant image metadata (\ie filter, exposure time, etc). 1479 inconsistent, and objects which are saturated.\VER{TEST}{TLR:13} 1480 1481 \item The resulting collection of detected objects shall be saved along 1482 with the relevant image metadata (\ie filter, exposure time, etc).\VER{TEST}{TLR:20} 1374 1483 \end{enumerate} 1375 1484 1376 1485 \paragraph{Image Processing Q/A} 1377 1486 1378 Before the image is added to the static sky, it must pass Q/A tests. 1379 1380 \tbd{how do we specify auotmatic Q/A tests? astrometry, photometry} 1487 Before the image is added to the static sky, it shall pass Q/A tests: 1488 1489 \begin{enumerate} 1490 \item the measured photometry scatter for the image shall be less than 1491 \tbr{1\%}.\VER{TEST}{TLR:1} 1492 1493 \item the measured astrometry scatter for the image shall be less than 1494 \tbr{30 mas}.\VER{TEST}{TLR:3} 1495 1496 \end{enumerate} 1381 1497 1382 1498 \paragraph{Update static sky} 1383 1499 1384 The final, cleaned input image must be added to the static sky so that 1385 an incrementally-deeper static sky image may be made. 1386 1387 \tbd{parameters, weight map} 1388 1389 \paragraph{Timing} 1390 1391 It is required that the {\em total} processing for each exposure by 1392 the Pan-STARRS system not take longer than the time between a complete 1393 set of exposures. For PS-1, the primary mode of operation will use 1394 four exposures to form a complete set (major frame), with 30 second 1395 exposures times and 2 second readout times. Thus, the complete Phase 1396 4 analysis must be performed on average within 120 seconds, assuming a 1397 separate collection of computers are dedicated to the Phase 2 1398 analysis. 1399 1400 \paragraph{Robustness} 1401 1402 It is essential that the static sky image (which may have been 1403 painstakingly accumulated over many months) not be corrupted by adding 1404 in transient sources, or data that is of suspect quality (due, e.g., 1405 to an error upstream in the processing). 1406 1407 \tbd{what are the corresponding requirements?} 1500 The final, cleaned input image shall be added to the static sky so that 1501 an incrementally-deeper static sky image may be made.\VER{TEST}{TLR:6, TLR:11} 1502 1503 \paragraph{Timing} 1504 The complete Phase~4 analysis shall be performed in $< 38$ seconds for 1505 up to 4 complete FPA images at one time. \VER{TEST}{TLR:17} 1408 1506 1409 1507 \subsubsection{Calibration Stages} 1410 1508 \label{mkcal} 1411 1509 1412 \tbd{Requirements on the speed of processing the calibration images.} 1413 1414 The Calibration analysis stages must construct the various types of 1415 calibration frames needed by the IPP. The requirements for each of 1416 these stages are discussed in detail below. 1510 The Calibration analysis stages construct the various types of 1511 calibration frames needed by the IPP. Requirements for the 1512 calibration processing include the following: 1513 1514 \begin{enumerate} 1515 \item The IPP Calibration Analysis shall produce master calibration images 1516 from the raw calibration images in less \tbr{2 hours}.\VER{TEST}{TLR:17, TLR:22} 1517 1518 \item Master calibration images shall not introduce systematic 1519 uncertainties in the photometry greater than \tbr{0.2\%}.\VER{TEST}{TLR:1} 1520 1521 \end{enumerate} 1522 1523 Requirements for each of the individual calibration analysis stages 1524 are discussed in detail below. 1417 1525 1418 1526 \paragraph{bias images} 1419 1527 \begin{enumerate} 1420 1528 1421 \item The \code{bias} calibration stage mustconstruct a master bias1422 image from a collection of raw bias images. 1423 1424 \item The \code{bias} calibration stage mustcorrect the input images1425 based on the overscan region. 1426 1427 \item The \code{bias} calibration stage mustcombine the input images1529 \item The \code{bias} calibration stage shall construct a master bias 1530 image from a collection of raw bias images.\TASK 1531 1532 \item The \code{bias} calibration stage shall correct the input images 1533 based on the overscan region.\TASK 1534 1535 \item The \code{bias} calibration stage shall combine the input images 1428 1536 using the statistic specified by the user, selected from one of the 1429 1537 following: sample mean, median, and mode, robust mean, median, and 1430 mode, and the clipped mean and median. 1431 1432 \item The \code{bias} calibration stage mustconstruct residual1433 images, in which the master bias is applied to the input images. 1538 mode, and the clipped mean and median.\TASK 1539 1540 \item The \code{bias} calibration stage shall construct residual 1541 images, in which the master bias is applied to the input images.\TASK 1434 1542 1435 1543 \item Outlier residual images, those for which the residual bias and 1436 variance in the bias image are excessive ($> 1DN$), mustbe excluded1437 from the input image stack the the bias image reconstructed. 1544 variance in the bias image are excessive ($> 1DN$), shall be excluded 1545 from the input image stack the the bias image reconstructed.\VER{TEST}{TLR:1} 1438 1546 \end{enumerate} 1439 1547 … … 1441 1549 \begin{enumerate} 1442 1550 1443 \item The \code{dark} calibration stage mustconstruct a master dark1444 image from a collection of raw dark images. 1445 1446 \item The \code{dark} calibration stage mustraise an error if the1551 \item The \code{dark} calibration stage shall construct a master dark 1552 image from a collection of raw dark images.\TASK 1553 1554 \item The \code{dark} calibration stage shall raise an error if the 1447 1555 input images have exposure time which deviate by more than 1448 \tbr{2\%}. 1449 1450 \item The \code{dark} calibration stage mustcorrect the input dark1451 images for the bias. 1452 1453 \item The \code{dark} calibration stage mustcombine the input images1556 \tbr{2\%}.\VER{TEST}{TLR:1} 1557 1558 \item The \code{dark} calibration stage shall correct the input dark 1559 images for the bias.\TASK 1560 1561 \item The \code{dark} calibration stage shall combine the input images 1454 1562 using the statistic specified by the user, selected from one of the 1455 1563 following: sample mean, median, and mode, robust mean, median, and 1456 mode, and the clipped mean and median. 1457 1458 \item The \code{dark} calibration stage mustconstruct residual1459 images, in which the master dark is applied to the input images. 1564 mode, and the clipped mean and median.\VER{TEST}{TLR:1} 1565 1566 \item The \code{dark} calibration stage shall construct residual 1567 images, in which the master dark is applied to the input images.\TASK 1460 1568 1461 1569 \item Outlier residual images, those for which the residual level and 1462 variance are excessive ($> 1DN$), mustbe excluded from the input1463 image stack the the dark image reconstructed. 1570 variance are excessive ($> 1DN$), shall be excluded from the input 1571 image stack the the dark image reconstructed.\VER{TEST}{TLR:1} 1464 1572 \end{enumerate} 1465 1573 … … 1467 1575 \begin{enumerate} 1468 1576 1469 \item The \code{flat-field} calibration stage mustconstruct a master1470 flat-field image from a collection of raw flat-field images. 1471 1472 \item The \code{flat-field} calibration stage mustaccept a group of1577 \item The \code{flat-field} calibration stage shall construct a master 1578 flat-field image from a collection of raw flat-field images.\VER{TEST}{TLR:1} 1579 1580 \item The \code{flat-field} calibration stage shall accept a group of 1473 1581 images from one of the following flat-field sources: dome, twilight, 1474 night-sky. 1475 1476 \item The \code{flat-field} calibration stage mustraise an error if1582 night-sky.\VER{TEST}{TLR:1} 1583 1584 \item The \code{flat-field} calibration stage shall raise an error if 1477 1585 the input images in a single stack used more than one of the above 1478 flat-field sources or multiple filters. 1479 1480 \item The \code{flat-field} calibration stage mustcorrect the input1481 flat-field images for the bias and dark. 1482 1483 \item The \code{flat-field} calibration stage mustcombine the input1586 flat-field sources or multiple filters.\TASK 1587 1588 \item The \code{flat-field} calibration stage shall correct the input 1589 flat-field images for the bias and dark.\TASK 1590 1591 \item The \code{flat-field} calibration stage shall combine the input 1484 1592 images using the statistic specified by the user, selected from one 1485 1593 of the following: sample mean, median, and mode, robust mean, 1486 median, and mode, and the clipped mean and median. 1487 1488 \item The \code{flat-field} calibration stage mustconstruct residual1594 median, and mode, and the clipped mean and median.\VER{TEST}{TLR:1} 1595 1596 \item The \code{flat-field} calibration stage shall construct residual 1489 1597 images, in which the master flat-field is applied to the input 1490 images. 1598 images.\TASK 1491 1599 1492 1600 \item Outlier residual images, those for which the residual level and 1493 1601 variance are excessive ($> 0.1$\%, or 1.02 times the Poisson limit 1494 of the flat-field image), mustbe excluded from the input image1495 stack the the flat-field image reconstructed. 1602 of the flat-field image), shall be excluded from the input image 1603 stack the the flat-field image reconstructed.\VER{TEST}{TLR:1} 1496 1604 \end{enumerate} 1497 1605 … … 1499 1607 \begin{enumerate} 1500 1608 1501 \item The \code{mask} calibration stage mustconstruct a bad-pixel1609 \item The \code{mask} calibration stage shall construct a bad-pixel 1502 1610 mask from a stack of raw flat-field images and a master flat-field 1503 image. 1504 1505 \item The \code{mask} calibration stage mustmask the pixels which are1611 image.\VER{TEST}{TLR:1} 1612 1613 \item The \code{mask} calibration stage shall mask the pixels which are 1506 1614 inconsistent in the input flats by more than \tbr{1\%}, given 1507 sufficient signal-to-noise in the input flats. 1508 1509 \item The \code{mask} calibration stage mustmask the pixels which are1615 sufficient signal-to-noise in the input flats.\VER{TEST}{TLR:1} 1616 1617 \item The \code{mask} calibration stage shall mask the pixels which are 1510 1618 consistently low or high in the input flats by more than a factor of 1511 \tbr{3} beyond the typical pixel. 1512 1513 \item The \code{mask} calibration stage mustmask the pixels1619 \tbr{3} beyond the typical pixel.\VER{TEST}{TLR:1} 1620 1621 \item The \code{mask} calibration stage shall mask the pixels 1514 1622 identified in a table of bad pixels generated externally to the 1515 calibration stage. 1516 1517 \item The \code{mask} calibration stage mustuse multiple bit values1518 to identify the different types of masked pixels. 1519 1520 \item The \code{mask} calibration stage mustraise an error if the1521 input images generate too many bad pixels in the mask. 1623 calibration stage.\TASK 1624 1625 \item The \code{mask} calibration stage shall use multiple bit values 1626 to identify the different types of masked pixels.\TASK 1627 1628 \item The \code{mask} calibration stage shall raise an error if the 1629 input images generate too many bad pixels in the mask.\TASK 1522 1630 \end{enumerate} 1523 1631 … … 1525 1633 \begin{enumerate} 1526 1634 1527 \item The \code{fringe} calibration stage mustconstruct a master fringe1635 \item The \code{fringe} calibration stage shall construct a master fringe 1528 1636 frame from a stack of raw night-time sky images or from a stack of 1529 dome fringe frames. 1530 1531 \item The \code{fringe} calibration stage mustraise an error if the input1637 dome fringe frames.\VER{TEST}{TLR:1, TLR:5} 1638 1639 \item The \code{fringe} calibration stage shall raise an error if the input 1532 1640 stack consists is images generated with more than one type of fringe 1533 source or with multiple filters. 1534 1535 \item The \code{fringe} calibration stage mustflatten the input images1536 to remove the pixel-to-pixel sensitivity variations of the detector. 1537 1538 \item The \code{fringe} calibration stage mustmeasure the fringe amplitude1539 on the input fringe images. 1540 1541 \item The \code{fringe} calibration stage mustscale the input fringe images1542 based on the fringe amplitude. 1543 1544 \item The \code{fringe} calibration stage mustcombine the scaled input1641 source or with multiple filters.\TASK 1642 1643 \item The \code{fringe} calibration stage shall flatten the input images 1644 to remove the pixel-to-pixel sensitivity variations of the detector.\VER{TEST}{TLR:1} 1645 1646 \item The \code{fringe} calibration stage shall measure the fringe amplitude 1647 on the input fringe images.\TASK 1648 1649 \item The \code{fringe} calibration stage shall scale the input fringe images 1650 based on the fringe amplitude.\TASK 1651 1652 \item The \code{fringe} calibration stage shall combine the scaled input 1545 1653 images using the statistic specified by the user, selected from one of 1546 1654 the following: sample mean, median, and mode, robust mean, median, and 1547 mode, and the clipped mean and median. 1548 1549 \item The \code{fringe} calibration stage mustconstruct residual images, in1655 mode, and the clipped mean and median.\VER{TEST}{TLR:5} 1656 1657 \item The \code{fringe} calibration stage shall construct residual images, in 1550 1658 which the master fringe image is applied to the input images, along 1551 with all necessary preceding calibration images. 1552 1553 \item The \code{fringe} calibration stage mustmeasure the residual fringe1554 amplitude on the residual images. 1659 with all necessary preceding calibration images.\TASK 1660 1661 \item The \code{fringe} calibration stage shall measure the residual fringe 1662 amplitude on the residual images.\TASK 1555 1663 \end{enumerate} 1556 1664 1557 1665 \paragraph{low-spatial-frequency sky models} 1558 1666 1559 The \code{sky model} calibration stage must construct a sky model 1560 image from a stack of raw night-time sky images. 1561 1562 \tbd{details of the image construction to be specified} 1667 The \code{sky model} calibration stage shall construct a sky model 1668 image from a stack of raw night-time sky images.\VER{TEST}{TLR:5} 1563 1669 1564 1670 \paragraph{Flat-field correction frame} 1565 1671 \begin{enumerate} 1566 1672 1567 \item The \code{flat-field correction} calibration stage mustconstruct a1673 \item The \code{flat-field correction} calibration stage shall construct a 1568 1674 flat-field correction image from dithered observations of a stellar 1569 field. 1570 1571 \item The \code{flat-field correction} calibration stage mustconstruct a1572 flat-field correction image for each filter and camera independently. 1573 1574 \item The \code{flat-field correction} calibration stage mustconstruct a1675 field.\VER{TEST}{TLR:1} 1676 1677 \item The \code{flat-field correction} calibration stage shall construct a 1678 flat-field correction image for each filter and camera independently.\TASK 1679 1680 \item The \code{flat-field correction} calibration stage shall construct a 1575 1681 correction image which makes the photometry of multiple observations 1576 1682 of the same stellar source consistent at different locations in the 1577 focal plane. 1578 1579 \item The \code{flat-field correction} calibration stage mustconstruct1580 corrected flat-field images using the measured correction. 1581 1582 \item The \code{flat-field correction} calibration stage mustdetermine the1683 focal plane.\VER{TEST}{TLR:1} 1684 1685 \item The \code{flat-field correction} calibration stage shall construct 1686 corrected flat-field images using the measured correction.\VER{TEST}{TLR:1} 1687 1688 \item The \code{flat-field correction} calibration stage shall determine the 1583 1689 consistency of the corrected flat-field images using the dithered 1584 1690 stellar field observations flattened with the corrected flat-field 1585 image. 1691 image.\TASK 1586 1692 \end{enumerate} 1587 1693 … … 1589 1695 \begin{enumerate} 1590 1696 1591 \item The \code{non-linear correction} calibration stage mustconstruct a1697 \item The \code{non-linear correction} calibration stage shall construct a 1592 1698 non-linear correction from a collection of images of a constant source 1593 with varying exposure times. 1594 1595 \item The \code{non-linear correction} calibration stage mustconstruct a1596 non-linear correction which linearizes the detector fluxes $<0.5\%$. 1597 1598 \item The \code{non-linear correction} calibration stage mustdetermine the1699 with varying exposure times.\VER{TEST}{TLR:1} 1700 1701 \item The \code{non-linear correction} calibration stage shall construct a 1702 non-linear correction which linearizes the detector fluxes $<0.5\%$.\VER{TEST}{TLR:1} 1703 1704 \item The \code{non-linear correction} calibration stage shall determine the 1599 1705 saturation regime, in which the non-linear correction is no longer 1600 consistent to $<0.5\%$. 1601 \end{enumerate} 1706 consistent to $<0.5\%$.\VER{TEST}{TLR:1} 1707 \end{enumerate} 1708 1709 \paragraph{Telescope Astrometry Parameters} 1710 1711 \begin{enumerate} 1712 \item The IPP Calibration system shall construct static models of the 1713 telescope astrometry parameters (e.g., distortion, detector warps) 1714 once per week.\VER{INSPECT}{TLR:4} 1715 1716 \item The IPP Calibration system shall construct static models of the 1717 telescope astrometry parameters (e.g., distortion, detector warps) 1718 with an accuracy to produce astrometry consistent to 30 1719 milliarcsec.\VER{TEST}{TLR:4} 1720 1721 \item The IPP Calibration system shall monitor changes in the 1722 telescope astrometry parameters and issue a warning if the 1723 parameters change by more than \tbr{2\%}.\VER{INSPECT}{TLR:4} 1724 \end{enumerate} 1725 1726 \paragraph{Zero-Point Monitoring} 1727 1728 The IPP Calibration system shall determine telescope filter and camera 1729 zero-points on a \tbd{timescale} with an accuracy sufficient to 1730 determine photometry in the native filter systems to 5 millimags. 1602 1731 1603 1732 \subsubsection{Reference Catalog Creation} … … 1607 1736 future Pan-STARRS calibration. The generation of these catalogs is 1608 1737 inherently a research project, and will require human control and 1609 intervention. The IPP mustprovide the data access, manipulation and1738 intervention. The IPP shall provide the data access, manipulation and 1610 1739 visualization tools needed to construct these reference catalogs and 1611 1740 to assess their quality. In this section, we list the requirements of … … 1635 1764 1636 1765 The IPP will generate an astrometric reference on the basis of the 1637 observations obtained by the AP survey. The IPP mustprovide the1766 observations obtained by the AP survey. The IPP shall provide the 1638 1767 analysis tools needed to generate the master astrometric reference 1639 1768 catalog. Much of the required functionality is covered by the AP … … 1642 1771 1643 1772 \begin{enumerate} 1644 \item The Astrometry Reference creation tools must return the match between 1773 \item The IPP Reference Creation System shall produce an astrometric 1774 reference catalog from the extracted objects within 6 months of the 1775 end of the AP Survey.\VER{TEST}{TLR:3, TLR:4} 1776 1777 \item The IPP Reference Creation System shall produce an astrometric 1778 reference catalog with an absolute accuracy of \tbr{100 mas} and a 1779 local relative accuracy of \tbr{30 mas} for bright objects.\VER{TEST}{TLR:3} 1780 1781 \item The IPP Reference Creation System shall produce an astrometric 1782 reference catalog with proper motions measurements for 1783 non-solar-system objects with an accuracy of \tbr{20 mas / year} for 1784 unsaturated, bright stars.\VER{TEST}{TLR:3} 1785 1786 \item The Astrometry Reference creation tools shall return the match between 1645 1787 stars observed with Pan-STARRS and any of several astrometric 1646 reference catalogs listed in Table~\ref{AstroRefs}. 1647 1648 \item The tools mustconvert the reference catalog object coordinates to all1788 reference catalogs listed in Table~\ref{AstroRefs}.\TASK 1789 1790 \item The tools shall convert the reference catalog object coordinates to all 1649 1791 of the coordinate frames of relevance in the telescope and camera 1650 1792 system: 1651 1793 \begin{enumerate} 1652 \item Catalog to mean positions 1653 \item Mean to apparent positions 1654 \item Apparent positions + pointing to tangent plane coordinates 1655 \item Apparent positions + pointing to focal plane coordinates 1656 \item focal plane to specific detector (OTA) 1657 \item specific detector to detector cell 1658 \end{enumerate} 1659 1660 \item The tools mustprovide the necessary calibration data: the telescope1794 \item Catalog to mean positions\VER{TEST}{TLR:3} 1795 \item Mean to apparent positions\VER{TEST}{TLR:3} 1796 \item Apparent positions + pointing to tangent plane coordinates\VER{TEST}{TLR:3} 1797 \item Apparent positions + pointing to focal plane coordinates\VER{TEST}{TLR:3} 1798 \item focal plane to specific detector (OTA)\VER{TEST}{TLR:3} 1799 \item specific detector to detector cell\VER{TEST}{TLR:3} 1800 \end{enumerate} 1801 1802 \item The tools shall provide the necessary calibration data: the telescope 1661 1803 and camera optical distortion models and the variation of the image 1662 PSF across the camera field, as a function of color. 1663 1664 \item The tools mustfit the observed stellar coordinates to the astrometric1804 PSF across the camera field, as a function of color.\TASK 1805 1806 \item The tools shall fit the observed stellar coordinates to the astrometric 1665 1807 reference catalog coordinates to determine improved astrometric 1666 solutions for both the stars and the detectors. 1667 1668 \item The tools must determine improved telescope optical distortion models 1669 based on the astrometric solutions. 1670 1671 \item The tools must optionally determine the fit coefficients as a function 1672 of position along, or with combinations of magnitude or color. 1673 1674 \item The fitting method must include robust outlier rejection. 1675 1676 \item The tools must identify objects which are detected in the catalog, but 1677 not the science image or vice-versa. 1678 1679 \item The tools must determine average centroiding errors for each object. 1680 1681 \item The tools must plot the fit residuals against a wide variety of 1682 parameters: the object positions, magnitudes, colors, etc. 1683 1684 \item The tools must allow the fit to exclude subsets of objects from the 1685 fits on the basis of these parameters. . 1686 1687 \item The tools must provide coordinates of the guide stars in the same frame 1688 of reference as the normal image data. 1689 1690 \item The tools must perform the various fitting steps for the guide stars 1691 rather than for the normal image data. 1808 solutions for both the stars and the detectors. \TASK 1809 1810 \item The tools shall determine improved telescope optical distortion models 1811 based on the astrometric solutions. \VER{TEST}{TLR:3} 1812 1813 \item The tools shall optionally determine the fit coefficients as a function 1814 of position along, or with combinations of magnitude or color. \VER{TEST}{TLR:3} 1815 1816 \item The fitting method shall include robust outlier rejection. \VER{TEST}{TLR:3} 1817 1818 \item The tools shall identify objects which are detected in the catalog, but 1819 not the science image or vice-versa.\TASK 1820 1821 \item The tools shall determine average centroiding errors for each object.\TASK 1822 1823 \item The tools shall plot the fit residuals against a wide variety of 1824 parameters: the object positions, magnitudes, colors, etc.\TASK 1825 1826 \item The tools shall allow the fit to exclude subsets of objects from the 1827 fits on the basis of these parameters.\TASK 1828 1829 \item The tools shall provide coordinates of the guide stars in the 1830 same frame of reference as the normal image data to within 30 1831 mas.\VER{TEST}{TLR:3} 1832 1833 \item The tools shall perform the various fitting steps for the guide stars 1834 rather than for the normal image data.\TASK 1692 1835 \end{enumerate} 1693 1836 … … 1703 1846 & mmag & mag & \\ 1704 1847 \hline 1705 SDSS & & &\\1706 CFHT-LS & & &\\1707 Landolt & & &\\1848 SDSS & 15? & 16? & {\em u,g,r,i,z} \\ 1849 CFHT-LS & 10? & 18 & {\em u,g,r,i,z} \\ 1850 Landolt & 10-20 & 15 & {\em U,B,V,R,I} \\ 1708 1851 \hline 1709 1852 \end{tabular} … … 1712 1855 1713 1856 The IPP will generate a photometric reference catalog on the basis of 1714 the observations obtained by the AP survey. The IPP mustprovide the1857 the observations obtained by the AP survey. The IPP shall provide the 1715 1858 analysis tools needed to generate a master photometric reference 1716 1859 catalog. Much of the required functionality is covered by the AP … … 1719 1862 1720 1863 \begin{enumerate} 1721 \item The Photometry Reference creation tools must return the match between 1864 \item The IPP Reference Creation System shall produce a photometric 1865 reference catalog from the extracted point-source objects within 6 1866 months of the end of the AP Survey.\VER{TEST}{TLR:1} 1867 1868 \item The IPP Reference Creation System shall produce a photometric 1869 reference catalog with a consistency across the sky of \tbr{5 1870 millimag}.\VER{TEST}{TLR:1} 1871 1872 \item The IPP Reference Creation System shall produce a photometric 1873 reference catalog with an absolute calibration to the external 1874 system (defined by \tbr{SDSS} and the CFHT Legacy Survey Standards) 1875 with an accuracy of \tbr{10 millimag} (for bright objects).\VER{TEST}{TLR:1} 1876 1877 \item The Photometry Reference creation tools shall return the match between 1722 1878 stars observed with Pan-STARRS and any of several photometric 1723 reference catalogs listed in Table~\ref{PhotoRefs}. 1724 1725 \item The tools mustconvert between different photometric systems, including:1726 \begin{enumerate} 1727 \item instrumental to nominal detector magnitude 1728 \item nominal detector magnitude to average filter system 1729 \item average filter system to reference photometry system 1730 \end{enumerate} 1731 1732 \item These transformations must account for color and airmass terms.1733 1734 \item The tools mustmeasure and apply relative photometry corrections1735 between images. 1736 1737 \item The tools mustdetermine photometric transformation fit coefficients1879 reference catalogs listed in Table~\ref{PhotoRefs}.\TASK 1880 1881 \item The tools shall convert between different photometric systems, including: 1882 \begin{enumerate} 1883 \item instrumental to nominal detector magnitude\VER{TEST}{TLR:1} 1884 \item nominal detector magnitude to average filter system\VER{TEST}{TLR:1} 1885 \item average filter system to reference photometry system\VER{TEST}{TLR:1} 1886 \end{enumerate} 1887 1888 \item These transformations shall account for color and airmass terms. \VER{TEST}{TLR:1} 1889 1890 \item The tools shall measure and apply relative photometry corrections 1891 between images.\VER{TEST}{TLR:1} 1892 1893 \item The tools shall determine photometric transformation fit coefficients 1738 1894 as a function of airmass, magnitude, color and detector coordinates, 1739 or with combinations of the above. 1740 1741 \item The fitting method must include robust outlier rejection.1742 1743 \item The tools mustreject specific objects from the fit on the basis of1895 or with combinations of the above.\TASK 1896 1897 \item The fitting method shall include robust outlier rejection.\VER{TEST}{TLR:1} 1898 1899 \item The tools shall reject specific objects from the fit on the basis of 1744 1900 object locations, instrumental magnitudes, observed and reference 1745 errors, and in particular time of the observations. 1746 1747 \item The tools mustplot the fit residuals against a wide variety of1748 parameters, including the object positions, magnitudes, colors, etc. 1749 1750 \item The tools mustprovide photometry from the guide stars in the same1751 system as observations of stars from the normal imaging data. 1752 1753 \item The tools mustperform the above fitting steps for the guide stars1754 rather than for the normal image data. 1901 errors, and in particular time of the observations. \TASK 1902 1903 \item The tools shall plot the fit residuals against a wide variety of 1904 parameters, including the object positions, magnitudes, colors, etc.\TASK 1905 1906 \item The tools shall provide photometry from the guide stars in the same 1907 system as observations of stars from the normal imaging data.\VER{TEST}{TLR:1} 1908 1909 \item The tools shall perform the above fitting steps for the guide stars 1910 rather than for the normal image data.\TASK 1755 1911 \end{enumerate} 1756 1912 1757 1913 \subsection{Modules} 1758 1914 1759 In order to encapsulat ionfunctionality, the analysis stages are1915 In order to encapsulate functionality, the analysis stages are 1760 1916 constructed of a sequence of steps. The analysis stages consist of a 1761 \tbd{python} script which executes a sequence of C-level functions. 1762 The C-level functions called by the \tbd{python} script are called1763 {\em modules} and represent basic data analysis operations. 1917 high-level script which executes a sequence of C-level functions. The 1918 C-level functions executed by the script are called {\em modules} and 1919 represent basic data analysis operations. 1764 1920 1765 1921 The required set of Pan-STARRS modules and their functionality is … … 1794 1950 1795 1951 \begin{enumerate} 1796 \item Certain IPP programs mustbe able to read and write standard1797 FITS images. 1798 1799 \item Certain IPP programs mustbe able to read and write files in1952 \item Certain IPP programs shall be able to read and write standard 1953 FITS images.\VER{TEST}{allocated} 1954 1955 \item Certain IPP programs shall be able to read and write files in 1800 1956 modified FITS format with Pan-STARRS definitions for non-square 1801 pixel arrays. 1957 pixel arrays.\VER{TEST}{allocated} 1802 1958 \end{enumerate} 1803 1959 1804 1960 \subsubsection{Table Formats} 1805 1961 1806 Certain IPP programs must be able to read and write FITS tables.1962 Certain IPP programs shall be able to read and write FITS tables.\VER{TEST}{allocated} 1807 1963 1808 1964 \subsubsection{Other Data Formats} 1809 1965 1810 Certain IPP program must be able to read and write XML files.1966 Certain IPP program shall be able to read and write XML files.\VER{TEST}{allocated} 1811 1967 1812 1968 \subsubsection{External Catalogs} 1813 1969 1814 The IPP AP Database must be able to interact with the following 1815 externally provided reference catalogs: 1816 1817 \begin{enumerate} 1818 \item Hipparcos 1819 \item Tycho2 1820 \item HST-GSC 1821 \item USNO-A 1822 \item UCAC 1823 \item 2Mass 1824 \item USNO-Bx 1825 \item YBx 1826 \end{enumerate} 1970 The IPP AP Database shall be able to interact with the following 1971 externally provided reference catalogs listed in Table~\ref{AstroRefs} 1972 and Table~\ref{PhotoRefs}.\VER{TEST}{TLR:1, TLR:3} 1827 1973 1828 1974 \subsubsection{Analysis Reference Data} 1829 1975 1830 The IPP must store reference data describing the following entities: 1831 1832 \begin{enumerate} 1833 \item Telescopes 1834 \item Cameras 1835 \item Detectors 1836 \item Filters 1837 \item software basic parameters 1838 \item computer configuration 1839 \end{enumerate} 1976 The IPP shall store reference data describing the relevant Pan-STARRS 1977 and IPP components, including the telescope, camera, detectors, 1978 filters, clustered computers, and IPP software parameters. 1979 1980 \subsubsection{Static Sky Pixel Size} 1981 1982 The IPP static sky shall have a pixel scale of \tbr{0.2\arcsec}. 1840 1983 1841 1984 \subsection{External Interfaces} 1842 1985 1843 Images from OATS / Camera data store. 1844 1845 Summit Metadata from OATS. 1846 1847 Image Q/A assessment to OATS 1848 1849 3$\sigma$ transient non-orphaned detections to MOPS 1850 1851 3$\sigma$ transient detections to Transient Science Client 1852 1853 Published static sky images to Science Database 1854 1855 Published objects (P2, P4S, P4D, SS) to Science Database 1986 The IPP shall interact with several Pan-STARRS systems and with the 1987 Client Science Pipelines, but it has no requirements to interact with 1988 external systems which are not associated with the Pan-STARRS project. 1989 1990 \begin{enumerate} 1991 1992 \item The IPP shall receive Metadata data from OTIS.\TASK 1993 1994 \item The IPP shall send image quality assessments to OTIS.\TASK 1995 1996 \item The IPP shall receive raw images from the Camera System.\TASK 1997 1998 \item The IPP shall send 3$\sigma$ transient non-orphaned detections 1999 to MOPS.\TASK 2000 2001 \item The IPP shall send all 3$\sigma$ transient detections to 2002 Transient Science Client.\TASK 2003 2004 \item The IPP shall send static sky images to Science Database after 2005 they are released for publication.\TASK 2006 2007 \item The IPP shall send object detections from Phase 2, Phase 4 (Sum 2008 and Difference detections) and the static sky images to the 2009 Published Science Products System after they are release for 2010 publication.\TASK 2011 \end{enumerate} 1856 2012 1857 2013 \subsection{Internal Interfaces} 1858 2014 2015 The IPP has internal interfaces between several of the architectural 2016 components and between the architectural components and the analysis 2017 stages. 2018 2019 \begin{enumerate} 2020 2021 \item IPP Scheduler - IPP Controller. The IPP Scheduler shall send to 2022 the IPP Controller information about the tasks to be performed and 2023 shall receive from the IPP Controller descriptions of the success or 2024 failure of these tasks.\TASK 2025 2026 \item IPP Scheduler - Metadata DB. The IPP Scheduler shall query the 2027 Metadata DB to determine an appropriate course of action. The IPP 2028 Scheduler shall send result and status information to the Metadata 2029 DB.\TASK 2030 2031 \item IPP Controller - Analysis Tasks. The IPP Controller shall 2032 initiate the Analysis Tasks and monitor their output and exit 2033 status.\TASK 2034 2035 \item Analysis Tasks - Metadata DB. The Analysis Tasks shall be able 2036 to query the Metadata DB as needed to extract metadata needed for a 2037 given task. The Analysis Tasks shall be able to send results and 2038 updates to the Metadata DB.\TASK 2039 2040 \item Analysis Tasks - Image Server. The Analysis Tasks shall be able 2041 to extract relevant images from the Image Server. The Analysis Tasks 2042 shall be able to send output images to the Image Server.\TASK 2043 2044 \item Analysis Tasks - AP DB. The Analysis Tasks shall be able to 2045 extract information related to specific objects from the Astrometric 2046 and Photometric Database. The Analysis Tasks shall be able to send 2047 result detections to the AP Database.\TASK 2048 \end{enumerate} 2049 1859 2050 \subsection{Internal Data Requirements} 2051 2052 The internal data requirements of the IPP are left as detailed design 2053 decisions, and are specified within the IPP Supplementary Design 2054 Requirements Documents for the IPP Modules and Library. 1860 2055 1861 2056 \subsection{Computer Hardware} … … 1866 2061 hardware requirements addressed in this section consist of: 1867 2062 1868 \begin{ enumerate}2063 \begin{itemize} 1869 2064 \item Total Disk Volume 1870 2065 \item Total Processing Power … … 1872 2067 \item Sustained Node Network I/O 1873 2068 \item Sustained Disk I/O 1874 \end{enumerate} 2069 \item Availabilty 2070 \end{itemize} 1875 2071 1876 2072 The report, `The Pan-STARRS Image Processing Pipeline Computational … … 1879 2075 configuration and the PS-4 configuration, under multiple assumptions 1880 2076 regarding the data volume. The requirements in this section are 1881 derived from that report, and follow the minimal data volum ne2077 derived from that report, and follow the minimal data volume 1882 2078 assumptions for PS-1. 1883 2079 … … 1889 2085 \hline 1890 2086 Raw data & 200 TB \\ 1891 static sky & 235TB \\1892 calibration frames & 1.8 TB \\1893 metadata db & 0. 2TB \\1894 AP db & 24TB \\2087 static sky & 350 TB \\ 2088 calibration frames & 2.8 TB \\ 2089 metadata db & 0.3 TB \\ 2090 AP db & 55 TB \\ 1895 2091 \hline 1896 total & 461TB \\2092 total & 610 TB \\ 1897 2093 \hline 1898 2094 \end{tabular} … … 1909 2105 1910 2106 \begin{enumerate} 1911 \item The IPP muststore all raw images from the first year from the2107 \item The IPP shall store all raw images from the first year from the 1912 2108 AP and IVP surveys. This corresponds to 175,000 images, or 175 TB, 1913 assuming 1 GB per image and compression. The IPP will require space 1914 for 200 TB of raw imagery to store the data from these two survey 1915 components along with raw calibration, test, and other raw images 1916 not in the AP and IVP surveys. 1917 1918 \item The IPP must store a single copy of the complete static sky in 1919 all four filters. With the assumed image sampling of 0.2 arcsec per 1920 pixel, this corresponds to 9.7 Tpix per filter, or a total of 235 TB 1921 for the 6 filters, with 2 bytes for the noise map and 2 bytes for 1922 the image map. 1923 1924 \item The IPP must also store other, smaller collections of data. The 1925 other components contribute only a small fraction of the data 1926 storage requirement. The metadata is a fraction of a terabyte, 1927 while the calibration frames (all master detrend frames) represent 1928 at most a few terabytes. The AP object and detection data make up a 1929 total of 24 terabytes (see Table~\ref{APrates}). 1930 1931 \item The IPP must have storage capacity for a total of 461 TB of data. 2109 assuming 1 GB per image with compression. The IPP will require 2110 space for 200 TB of raw imagery to store the data from these two 2111 survey components along with raw calibration, test, and short-term 2112 storage of other raw images not in the AP and IVP 2113 surveys.\VER{INSPECT}{TLR:23} 2114 2115 \item The IPP shall store a single copy of the complete static sky in 2116 all 6 filters. With the assumed image sampling of 0.2 arcsec per 2117 pixel, this corresponds to 9.7 Tpix per filter, or a total of 350 TB 2118 for the 6 filters, with 4 bytes for the image pixels and 2 bytes for 2119 the noise map pixesl.\VER{INSPECT}{TLR:6, TLR:11} 2120 2121 \item The IPP shall store all detections from the AP, IVP, and MVP 2122 Surveys. These detections make up a total of 55 terabytes (see 2123 Table~\ref{APrates}). \VER{INSPECT}{TLR:24} 2124 2125 \item The IPP shall store all metadata and master calibration images 2126 from two years of PS-1 operation. The metadata is a fraction of a 2127 terabyte, while the calibration frames (all master detrend frames) 2128 represent at most 2 terabytes. \VER{INSPECT}{TLR:25} 2129 2130 \item The IPP shall have storage capacity for a total of 610 TB of data. 1932 2131 \end{enumerate} 1933 2132 1934 2133 \subsubsection{CPU Requirements} 1935 2134 1936 The IPP must provide sufficient computing resources to keep up with 1937 the data analysis tasks. The minimal processing requirement is that 1938 the analysis of a typical night's worth of data be completed within 12 1939 hours of the start of the night. With a typical night length of 8 1940 hours, and a maximum read rate of 1 image every 30 seconds, this 1941 implies an average of 45 seconds per image. 1942 1943 The science image analysis dominates the processing requirements. 1944 Within the science image analysis, Phase 2 and Phase 4 dominate the 1945 processing requirements. These two phases are performed in sequence 1946 with separate computers performing the analyses. They may therefore 1947 be addressed independently. 1948 1949 \begin{enumerate} 1950 \item The IPP must perform the Phase 2 analysis within an average time of 45 1951 seconds per single Gigapixel camera image. The Phase 2 analysis has 1952 been measured to require 3200 GHz-sec on a x86/32 bit machine, 1953 implying a requirement of NN GHz for the Phase 2 analysis, if NN sec 1954 are devoted to I/O. 1955 1956 \item The IPP must perform the Phase 4 analysis on a set of 4 input frames 1957 within an average time of 180 seconds. The Phase 4 analysis has been 1958 measured to require a total of 7800 GHz-sec on an x86/32 bit machine 1959 for a major frame of 4 input Gigapixel camera images. 2135 \begin{enumerate} 2136 \item The IPP shall provide sufficient computing resources to process 2137 images obtained at a cadence of 1 image per 40 seconds.\VER{TEST}{TLR:17} 2138 2139 \item The IPP shall perform the Phase 2 analysis within an average 2140 time of 40 seconds per single Gigapixel camera image. The Phase 2 2141 analysis has been measured to require 3200 GHz-sec on a Pentium-4 2142 machine.\VER{TEST}{TLR:17} 2143 2144 \item The IPP shall perform the Phase 4 analysis on a set of 4 input 2145 frames within an average time of 180 seconds. The Phase 4 analysis 2146 has been measured to require a total of 7800 GHz-sec on a Pentium-4 2147 machine for a major frame of 4 input Gigapixel camera 2148 images.\VER{TEST}{TLR:17} 1960 2149 \end{enumerate} 1961 2150 … … 1967 2156 raw images and the corresponding detrend images, and that all Phase 4 1968 2157 processing requires complete network distribution of both the initial 1969 and updated static sky images, the total I/O for a 1 80 second2158 and updated static sky images, the total I/O for a 160 second 1970 2159 major-frame period is: 1971 \begin{ enumerate}2160 \begin{itemize} 1972 2161 \item 8 GB from summit to Phase 2 (4 images @ 2 GB each) 1973 2162 \item 18 GB from Phase 2 to Phase 4 (3 bytes per pixel for image + … … 1976 2165 input image pixel, 4 bytes per pixel). 1977 2166 \item 9 GB from Phase 4 to Static Sky 1978 \end{enumerate} 1979 for a grand total of 44 GB over 180 seconds, or 244 MB/second, of 1980 which 26 GB are processed by the Phase 2 nodes and 36 are processed by 1981 the Phase 4 nodes. The IPP must be capable of sustaining this network 1982 load. 1983 1984 \paragraph{Disk I/O Requirements} 1985 1986 The disk I/O requirements are determined by the total number of bytes 1987 read from and written to disk. For each major frame processed, the 1988 total I/O to and from disk for Phase 2 is: 1989 \begin{enumerate} 2167 \end{itemize} 2168 for a total of 44 GB, of which 26 GB are used by the Phase 2 nodes and 2169 36 are used by the Phase 4 nodes. The IPP shall be capable of 2170 sustaining this network load.\VER{TEST}{TLR:17} 2171 2172 \paragraph{Phase 2 Disk I/O Requirements} 2173 2174 For each major frame processed, the total I/O to and from disk for 2175 Phase 2 is: 2176 \begin{itemize} 1990 2177 \item 8 GB raw image from summit to Phase 2 nodes (4 images @ 2 GB each) 1991 2178 \item 8 GB raw image from Phase 2 disk to memory … … 1994 2181 + 1 byte mask). 1995 2182 \item 18 GB processed image from Phase 2 disk to Phase 4 1996 \end{enumerate} 1997 for a grand total of 86 GB I/O for Phase 2. Equivalently, for each 1998 major frame processed, the total I/O to and from disk for Phase 4 is: 1999 \begin{enumerate} 2000 \item 18 GB processed image from Phase 2 disk to Phase 4 2183 \end{itemize} 2184 for a total of 86 GB Disk I/O for Phase 2 for a complete major frame.\VER{TEST}{TLR:17} 2185 2186 \paragraph{Phase 4 Disk I/O Requirements} 2187 For each major frame processed, the total I/O to and from disk for 2188 Phase 4 is: 2189 \begin{itemize} 2001 2190 \item 9 GB static image from Phase 4 disk to memory 2002 2191 \item 9 GB static image from memory to Phase 4 disk 2003 \end{enumerate} 2004 for a total of 36 GB I/O for Phase 4. 2005 2006 \subsubsection{Per-Node I/O Requirements} 2007 2008 Data I/O per node is defined as the number of bytes per second passed 2009 through the node's network adapter. The data I/O per node is tied to 2010 the total processing power and the total number of nodes. A useful 2011 way to examine the per-node I/O requirements is to compare the I/O and 2012 CPU requirements to determine the required number of processing nodes. 2013 The assumption is made that each CPU is associated with a single disk 2014 RAID which may deliver data at a rate of 100 MB/sec and a GigE 2015 ethernet controller which may deliver data at a sustained rate of 50 2016 MB/sec, and that each CPU is equivalent to 4 GHz. The IPP must 2017 therefore have a total of 26 Phase 2 nodes and 16 Phase 4 nodes. 2192 \end{itemize} 2193 for a total of 18 GB I/O for Phase 4 for a complete major frame.\VER{TEST}{TLR:17} 2194 2195 \subsubsection{Total Node Requirements} 2196 2197 The I/O and CPU requirements above may be confronted with reasonable 2198 assumptions of bandwidth and CPU speeds to estimate the number of 2199 nodes required for the IPP. Each CPU is matched with one network 2200 adapter and one disk array. : 2201 \begin{enumerate} 2202 \item The IPP requires at least 40 Phase 2 Nodes (OTA Nodes)\VER{TEST}{TLR:17} 2203 \item The IPP requires at least 5 TB for each Phase 2 node\VER{TEST}{TLR:17} 2204 \item The IPP requires at least 25 Phase 4 Nodes (Static Sky Nodes)\VER{TEST}{TLR:17} 2205 \item The IPP requires at least 14 TB for each Phase 4 node\VER{TEST}{TLR:17} 2206 \item The IPP requires at least 10 AP DB Nodes\VER{TEST}{TLR:17} 2207 \end{enumerate} 2208 2209 \subsubsection{Availability} 2210 2211 The IPP Image Server nodes shall not be offline for more than 12 hours 2212 consecutively or 36 hours per year.\VER{ANALYSIS}{TLR:17} 2018 2213 2019 2214 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% … … 2021 2216 \section{Test Verification} 2022 2217 2023 A testing regime mustbe implemented to demonstrate the working state2024 of the provided software. Certain tests as specified mustbe2218 A testing regime shall be implemented to demonstrate the working state 2219 of the provided software. Certain tests as specified shall be 2025 2220 performed by MHPCC, with code release contingent on success. Other 2026 2221 specified tests will be performed by IfA to verify the validity of the … … 2031 2226 \subsection{Software Configuration Tests} 2032 2227 2033 MHPCC musttest the validity of the software configuration,2228 MHPCC shall test the validity of the software configuration, 2034 2229 specifically to check that the code can be compiled on the specified 2035 2230 platforms and that the compilation produces no errors or warnings, … … 2039 2234 \begin{enumerate} 2040 2235 2041 \item MHPCC musttest that the code does not produce memory leaks.2042 2043 \item MHPCC musttest that the code does not produce segmentation faults.2236 \item MHPCC shall test that the code does not produce memory leaks. 2237 2238 \item MHPCC shall test that the code does not produce segmentation faults. 2044 2239 \end{enumerate} 2045 2240 2046 2241 \subsection{Basic Unit Tests} 2047 2242 2048 MHPCC mustperform basic unit tests with sample input data and known2243 MHPCC shall perform basic unit tests with sample input data and known 2049 2244 output results, including invalid input data to test error handling. 2050 These tests mustexercise the complete range of module options.2245 These tests shall exercise the complete range of module options. 2051 2246 2052 2247 \subsection{Detailed Functional Analysis} 2053 2248 2054 IfA mustperform detailed tests with a wide range of input data and2249 IfA shall perform detailed tests with a wide range of input data and 2055 2250 compare the results with existing software system. 2056 2251 2057 2252 \subsection{Test Verification Matrix} 2058 2253 2059 \subsubsection{Pan-STARRS IPP Library} 2060 2061 See Appendix A \& B of the IPP Library SDR (PSDC-430-007) for the test 2062 verification matrices for the Pan-STARRS IPP Library 2254 Test Verification Matrix information is supplied with each identified 2255 requirement in this document. 2256 2257 \subsection{Trace Matrix} 2258 \input{ippSRStrace.tex} 2063 2259 2064 2260 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Note:
See TracChangeset
for help on using the changeset viewer.
