IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 2114


Ignore:
Timestamp:
Oct 13, 2004, 7:06:32 PM (22 years ago)
Author:
eugene
Message:

design work towards PDR

Location:
trunk/doc/design
Files:
3 edited

Legend:

Unmodified
Added
Removed

  • trunk/doc/design/ippSDRS.tex

    r1399 r2114  
    1 %%% $Id: ippSDRS.tex,v 1.4 2004-08-06 19:06:01 eugene Exp $
     1%%% $Id: ippSDRS.tex,v 1.5 2004-10-14 05:06:31 eugene Exp $
    22\documentclass[panstarrs]{panstarrs}
    33
    … …  
    3636\pagenumbering{arabic}
    3737
    38 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    39 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    40 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    41 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    42 
    4338\section{Scope}
    44 
    45 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    46 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    47 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    4839
    4940\subsection{Identification}
    … …  
    5445Pan-STARRS 1 (PS-1), the initial demonstration telescope to be
    5546constructed on Haleakala by Jan 2006. 
    56 
    57 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    58 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    59 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    6047
    6148\subsection{System Overview}
    … …  
    7259roughly 2 years.
    7360
    74 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    75 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    76 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    77 
    7861\subsection{Document Overview}
    7962
    … …  
    8568Open Issues and TBDs in this document are marked \tbd{in bold, red
    8669type with surrounding square brackets}.
    87 
    88 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    89 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    90 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    91 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    9270
    9371\DocumentsInternalSection
    … …  
    10078\DocumentsEnd
    10179
    102 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    103 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    104 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    105 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    106 
    107 \section{System Design Decisions}
     80\section{Subsystem Overview}
     81
     82The Pan-STARRS Image Processing Pipeline (IPP) performs the image
     83processing and data analysis tasks needed to enable the scientific use
     84of the images obtained by the Pan-STARRS telescopes.  The primary
     85goals of the IPP are to process the science images from the Pan-STARRS
     86telescopes and make the results available to other systems within
     87Pan-STARRS.  It also is responsible for combining all of the science
     88images in a given filter into a single representation of the
     89non-variable component of the night sky called the ``Static Sky''.  To
     90achieve these goals, the IPP also performs other analysis functions to
     91generate the calibrations needed in the science image processing and
     92to occasionally use the derived data to generate improved astrometric
     93and photometric reference catalogs.  It also provides the
     94infrastructure needed to store the incoming data and the resulting
     95data products.
     96
     97The IPP inherits lessons learned, and in some cases code and prototype
     98code, from several other astronomy image analysis systems, including
     99Imcat (Kaiser), the Sloan Digital Sky Survey (REF), the Elixir system
     100(Magnier \& Cuillandre), and Vista (Tonry).  Imcat and Vista have a
     101large number of robust image processing functions.  SDSS has
     102demonstrated a working analysis pipeline and large-scale database
     103system for a dedicated project.  The Elixir system has demonstrated an
     104automatic image processing system and an object database system for
     105operational usage.
     106
     107The users of the IPP output are all systems internal to the Pan-STARRS
     108project.  They consist of the Transient Science Client, which will
     109receive the detections of transient objects on short time-scales; the
     110Moving Object Processing System (MOPS), which will receive the
     111detections of non-stationary transient objects on day-to-week
     112timescales; and the Published Science Products Subsystem (PSPS), which
     113will receive all data products of interest to the outside world, and
     114will act as the long-term archive and publishing clearinghouse.
     115
     116An important operational choice for the IPP is the decision not to
     117attempt to save all raw data.  Once the IPP is running in a standard
     118operational mode, data will be processed by the pipeline and deleted
     119when it is no longer needed.  Raw images will only be saved for a
     120short period to allow tests and quality assurance, and potentially to
     121allow systems which study transient phenomena to return to recent data
     122for closer inspection.  In general, during normal operations, raw
     123science images will be deleted after $\sim$1 month.
     124
     125The primary IPP hardware system on which the software operates will
     126not be located at the summit.  Instead, because of thermal, power, and
     127space constraints, the hardware will likely be located in a facility
     128off the mountain.  A subset of processing tasks may eventually be
     129assigned to machines at the summit if justified by the savings in data
     130transfer time and cost.
     131
     132\subsection{Analysis Tasks and Stages}
     133
     134Specific programs are required to perform the processing steps listed
     135above.  These can be divided into well-defined analysis stages, each
     136of which operates on a particular unit of data, such as a single OTA
     137image or a collection of astronomical objects.  Analysis tasks
     138representing the different analysis stages are performed on the IPP
     139computer cluster.  Note the distinction between the generic analysis
     140{\em stage} and a specific analysis {\em task}.  An analysis stage
     141represents a type of analysis which is performed, such as the basic
     142image calibration and object detection analysis.  An analysis task is
     143a particular realization of an analysis stage, e.g., the analysis of
     144OTA number 61 from exposure 654321 to produce a specific set of output
     145data products.  The analysis stages are discussed in detail in
     146Section~\ref{IPP:AnalysisStages}.
     147
     148Depending on the particular stage, it may process individual images,
     149collections of images, or on derived data products.  Because of the
     150nature of the image data, many of the analysis stages can be run in
     151parallel because, for example, the analysis of a chip in one image
     152does not depend on the results from another chip.
     153
     154\subsection{Architectural Components}
     155
     156In order to achieve the required functionality, the IPP provides an
     157infrastructure within which the Analysis Stages above are exectuted.
     158We have divided the IPP software infrastructure into a number of
     159clearly-defined architectural software units, listed as follows:
     160
     161\begin{itemize}
     162
     163\item {\bf Image Server:} This component is a large data store for all
     164  images used by the IPP, including the raw images from the telescope,
     165  the master calibration images, the reference static-sky images, and
     166  any temporary image data products produced by the IPP.  The Image
     167  Server accepts the incoming data and stores it until it is no longer
     168  needed by other portions of the IPP.  The Image Server is not
     169  restricted to imaging data: it is capable of storing any large data
     170  files which are not well-suited for inclusion in a more structured
     171  relational database and for which access needs to be widely
     172  available beyond the individual process which created the file.
     173
     174\item {\bf Astrometry \& Photometry Database (AP DB):} This component
     175  stores and manipulates astronomical objects detected in various
     176  images, as identified above, including individual measurements of
     177  objects on the images, the summary information about those objects,
     178  and reference object data.  It also provides mechanisms for users to
     179  query and manipulate the objects and detections.
     180
     181\item {\bf Metadata Database:} This component stores the data which is
     182  not directly related to images or astronomical objects, but which is
     183  needed to perform the IPP analyses.  The metadata may include the
     184  summary weather information for each night, or details about the
     185  filters, camera, telescopes, etc. 
     186
     187\item {\bf IPP Controller:} In order to perform the analysis stages
     188  required by the IPP, it is necessary to use distributed computing
     189  processes on a large number of computers.  The IPP Controller
     190  manages the collection of analysis tasks performed on these
     191  machines.
     192
     193\item {\bf IPP Scheduler:} This component is a decision-making
     194  mechanism which guides the operation of the IPP.  It evaluates the
     195  currently available collection of data, identifies the necessary
     196  analysis, and assigns the analysis tasks to the IPP Controller.
     197
     198\end{itemize}
     199
     200The relationship between these software units is shown in
     201Figure~\ref{overview}.  This figure also shows the interactions
     202between the IPP and other Pan-STARRS systems.  The architectural
     203components are discussed in detail in
     204Section~\ref{IPP:ArchComponents}.
     205
     206\begin{figure}
     207\begin{center}
     208\resizebox{6in}{!}{\includegraphics{pics/IPPoverview}}
     209\caption{ \label{overview} IPP System Overview}
     210\end{center}
     211\end{figure}
     212
     213\subsection{IPP Hardware Organization}
     214
     215\begin{figure}
     216\begin{center}
     217\resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}}
     218\caption{ \label{hardware} IPP Hardware Organization}
     219\end{center}
     220\end{figure}
     221
     222The IPP needs substantial computer resources, both in terms of
     223computational power and in terms of data storage and network
     224bandwidth.  The IPP requires relatively large amounts of data storage
     225space, primarily for the image data.  Image data is organized in two
     226categories.  First, there is the per-OTA data -- data associated with
     227specific OTAs, including the raw images, the calibration images, and
     228temporary processed images at various stages.  Second, there is the
     229data associated with the static sky imagery, which is in turn
     230organized into smaller sky-cell units.  In addition to image data,
     231there are the somewhat smaller data entities of the Metadata Database
     232and AP Database.
     233
     234The computer hardware is organized into nodes which provide both data
     235storage and computational resources.  The data storage nodes are
     236divided into three classes: those which deal with the per-OTA image
     237data, those that provide the storage for the static sky images, and
     238those that provide the storage for the other data systems, the
     239Metadata Database and the AP Database.  In addition, the computational
     240tasks related to Phase 2 take place on the per-OTA storage nodes and
     241the Phase 4 computation takes place on the static sky storage nodes.
     242
     243Figure~\ref{hardware} shows our basic concept for the hardware
     244organization for the IPP.  This diagram shows the two types of compute
     245nodes: OTA-level processing and storage nodes (dominated by Phase 2)
     246and static sky processing and storage nodes (mostly Phase 4).  Also
     247shown are two switches which divide the network into OTA and
     248Static-Sky portions.  In such an organization, the interswitch
     249communication must meet the throughput needs between these network
     250portions.  The additional data systems (Metadata Database and AP
     251Database) are also shown.
     252
     253%%% needs some work / move around elsewhere
     254\subsection{System Interfaces}
     255
     256\paragraph{MOPS and other Client Science Pipelines}
     257
     258The Client Science Programs (CSPs) and the Moving Object Processing
     259System (MOPS) are not a part of the IPP, but are external systems.  We
     260include them here to show the required interfaces.
     261
     262The CSPs and MOPS may query the Pixel DB, the Metadata DB and the
     263Object DB.  In addition, they may write certain fields to the object
     264DB (e.g., the external identifiers and class of object) as they
     265process objects, and they may retrieve pixel data from the Nodes.
     266
     267Since ``CSPs'' is a vague term, we now give some examples which may
     268help to illustrate the functionality.
     269
     270One example of a CSP is a web front-end to retrieve (published) images
     271and objects from the Pixel DB and Object DB.
     272
     273Another example would be a program interested in searching for
     274transiting extrasolar planets.  Such a program may periodically poll
     275the Metadata DB for new processed observations in its region of
     276interest (such as the Galactic Plane), retrieve the object photometry
     277of all high signal-to-noise stars in the processed observations, and
     278attempt to identify a planetary transit in progress.
     279
     280Yet another example would be a Stationary Transient Object Pipeline,
     281which would periodically poll the Metadata DB for new processed
     282observations, and query the Object DB for variable sources which were
     283identified twice (so that they are not moving objects).
     284
     285\subsection{System Design Decisions}
    108286
    109287Since \PS{} is a survey project, all data from the telescopes will be
    … …  
    128306System (MOPS), and potentially other client science pipelines.
    129307
    130 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    131 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    132 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    133 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    134 
    135 \subsection{System Overview}
    136 
    137 The \PS{} Image Processing Pipeline (IPP) consists of a collection of
    138 computer hardware and software organized to perform the tasks required
    139 to process images from the \PS{} telescopes.  The primary goal of the
    140 IPP is to process the science images from the \PS{} telescopes and
    141 make the results available to other systems within \PS{}.  To achieve
    142 this goal, the IPP must also perform other analysis functions to
    143 generate the calibrations needed in the science image processing and
    144 to occasionally use the derived data to generate improved astrometric
    145 and photometric reference catalogs.
    146 
    147 In order to meet these broad goals, the IPP must have the following
    148 capabilities:
     308\section{System Design : Architectural Components}
     309
     310\subsection{IPP Image Server}
     311
     312\begin{figure}
     313\psfig{file=pics/ImageServer,width=15cm,angle=0}
     314\caption{The components of the IPP Image Server.}
     315\label{fig:ImageServer}
     316\end{figure}
     317
     318The IPP Image Server is a repository for all images and other large
     319data files required by the IPP.  In addition, it provides tools for
     320managing the distribution of these large data files and for accessing
     321the files.  Data files stored by the IPP Image Server include the raw
     322images, the calibration images, intermediate processing stage images
     323as needed, final processed images, difference images, image
     324subsections, and any large non-imaging datafiles needed by the IPP.
     325The IPP Image Server must retain the files for as long as they are
     326needed by the IPP.
     327
     328The IPP Image Server is a parallel storage system.  It stores data
     329across a collection of computer nodes, each with their own data
     330storage resources.  Any single file is stored on only a single
     331computer and storage system.  In order to achieve the data throughput
     332requirements, the IPP Image Server may distribute the images across
     333the processor nodes in an organized fashion, i.e.\ associating
     334specific machines with specific detectors.  It is not the
     335responsibility of the IPP Image Server to determine which computer
     336should be associated with a specific data concept (Chip / region of
     337sky), but it must enable the association of a particular file with a
     338particular machine.
     339
     340There are three data concepts relevant to the IPP Image Server:
    149341\begin{itemize}
    150 \item Store a large amount of image data, and other derived data
    151 products (metadata and extracted objects);
    152 \item Provide access mechanisms to these data products (both to the
    153 subsystems of the IPP and in some cases to external users);
    154 \item Continuously accept new image data and metadata from the
    155 telescope system;
    156 \item Execute various analysis processes using these data products;
    157 and
    158 \item Provide the decision-making logic needed to guide the data
    159 processing, and to automatically launch the data processing tasks on
    160 an appropriate timescale.
     342\item {\bf storage object} This represents a single, unique data
     343  entity the Image Server.
     344
     345\item {\bf instance} A single copy of the storage object in the Image
     346  Server.  In general, given storage object may have several instances
     347  in the Image Server, normally on different computer nodes.
     348
     349\item {\bf file ID} This is the identifier of a particular storage
     350  object in the Image Server.  The file ID is simply a unique string,
     351  equivalent to the filename in a UNIX file system.
    161352\end{itemize}
    162 The IPP therefore includes subsystems which provide the data storage
    163 framework, the data analysis framework, and the scheduling of the
    164 analysis processes.  The data storage subsystems also provide
    165 interface mechanisms to the external \PS{} systems.
    166 
    167 The IPP architecture can be viewed in several possible ways.  We first
    168 consider the software architecture components needed by the IPP.
    169 These subsystems provide the infrastructure for the data storage and
    170 the data processing.  Next, we consider the analysis pipelines which
    171 make up the major processing tasks that must be performed by the IPP.
    172 Finally, we consider the hardware organization required to efficiently
    173 and cost-effectively achieve the necessary computing and storage
    174 requirements.
    175 
    176 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    177 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    178 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    179 
    180 \subsection{System Architecture}
    181 
    182 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    183 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    184 
    185 \subsubsection{Architectural Components}
    186 
    187 In Figure~\ref{fig:functionalities} we show the functionality of the
    188 IPP.
    189 
    190 The Observatory and Telescope System (\textbf{OATS}) system at the
    191 summit periodically produces metadata (e.g.\ weather measurements,
    192 observations completed) and pixel data (the image pixels from the
    193 cameras).  The \textbf{Pollster} regularly (e.g., twice per minute)
    194 polls OATS for the existence of new data.  If new data exists, the
    195 Pollster writes it to the \textbf{Metadata DB}, which maintains a
    196 table of observations that have been obtained and whether these
    197 observations are reduced, not reduced, or being reduced.  The
    198 \textbf{Scheduler} regularly (e.g., twice per minute) polls the
    199 Metadata DB for observations that match predefined criteria that are
    200 required to run reduction processes.  For example, the Phase 1
    201 processing requires that Phase 0 has been run on a focal plane
    202 metadata, and also requires that the observations are available and
    203 have not yet been processed.  If the criteria are met, the appropriate
    204 stage is passed to the \textbf{Localiser} which, checks the
    205 \textbf{Pixel DB} to determine if the stage should be performed on a
    206 particular node.  The Localiser passes the reduction stage to be
    207 processed, along with the preferred (or mandatory) node that should
    208 execute the reduction stage, to the \textbf{Controller}.  It is the
    209 Controller's responsibility to maintain the list of reduction stages
    210 to be processed and execute these stages on the \textbf{Nodes}.  The
    211 Nodes may retrieve the pixel data from OATS, they write to the Pixel
    212 DB the location of the products of the reduction and report their
    213 completion to the Controller.
    214 
    215 External systems, such as the Moving Object Processing System
    216 (\textbf{MOPS}) and other Client Science Pipelines (\textbf{CSPs})
    217 read the Metadata DB and the Object DB.  They may also write to the
    218 Object DB the classification of particular objects (e.g., identify an
    219 object as an asteroid).  Also, the MOPS and CSPs may also query the
    220 Pixel DB for the location of pixel data and copies data from the
    221 Nodes.
    222 
    223 \begin{figure}
    224 \psfig{file=pics/IPPfunctionalities,width=15cm,angle=0}
    225 \caption{The functionalities of the architectural design.  See the text
    226 for further explanation.}
    227 \label{fig:functionalities}
    228 \end{figure}
    229 
    230 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    231 
    232 \paragraph{OATS}
    233 
    234 The Observatory And Telescope System (OATS) is not a part of the IPP,
    235 but interfaces are required with it in order to allow the Pollster to
    236 get the list of observations not in the Metadata DB, and the nodes to
    237 retrieve pixel data.  Also, the Scheduler may report the need for new
    238 calibration data.
    239 
    240 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    241 
    242 \paragraph{Pollster}
    243 
    244 The Pollster is a program that polls OATS at regular intervals for the
    245 existence of observations not contained in the Metadata DB.  New
    246 weather and image metadata are written to the Metadata DB.
    247 
    248 There is no reason why this architectural component cannot be
    249 contained within another (such as the Scheduler), but it is shown here
    250 as separate for simplicity.
    251 
    252 A polling model is adopted so that OATS' interface may be kept as
    253 simple as possible --- OATS should not be concerned with whether the
    254 IPP has received notifications.  Under this polling model, it is
    255 specifically the responsibility of the IPP to retrieve from OATS the
    256 metadata that is not not already in the Metadata DB.
    257 
    258 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    259 
    260 \paragraph{Metadata DB}
    261 
    262 The Metadata DB stores and maintains the metadata\footnote{Note that
    263 metadata is any data which is not pixel data or object data.},
    264 including the list of images taken by the telescope system and whether
    265 these images have been processed.  The Metadata DB is regularly polled
    266 by the Scheduler to determine what images are ready to be processed.
    267 
    268 Both the Scheduler and the Pollster update the status of the Metadata
    269 DB --- the Pollster as new images become available at the Summit, and
    270 the Scheduler as images are processed.
    271 
    272 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    273 
    274 \paragraph{Scheduler}
    275 
    276 The Scheduler is responsible for determining the processing stages
    277 that are required to be run on any data.  Examples of these processing
    278 stages are ``Copy the pixels from the summit'' and ``Run Phase 2
    279 processing on chip 12 of exposure 123''.
    280 
    281 Processing stages to be executed are passed to the Localiser, which
    282 returns to the Scheduler the list of processing stages with node
    283 assignments to each of the stages.  This list of processing stages
    284 with node assignments is passed to the Controller for execution.
    285 
    286 Processing stages which have executed are reported by the Controller,
    287 which updates the Metadata DB appropriately.
    288 
    289 The Scheduler may also interact with OATS to inform it of the need
    290 for new calibration data.
    291 
    292 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    293 
    294 \paragraph{Localiser}
    295 \label{sec:localiser}
    296 
    297 It is the duty of the Localiser to assign processing stages to
    298 particular nodes.  This may be in order to optimise performance by
    299 distributing the stages across the nodes, or in the simplest possible
    300 case, it may make no recommendation upon the node which performs a
    301 particular stage.
    302 
    303 The Localiser may query the Pixel DB in order to identify the location
    304 of calibration data that may be needed for the processing stage to run
    305 (and in all likelihood, assign the processing stage to the same node as
    306 that which holds the calibration data).
    307 
    308 The Localiser may either demand or request that a stage is performed on
    309 a particular node, or make no recommendation, and passes the processing
    310 stage back to the Scheduler.
    311 
    312 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    313 
    314 \paragraph{Controller}
    315 
    316 The Controller's job is to control the execution of the processing
    317 stages on the nodes.  It is passed stages by the Localiser, and
    318 executes them on the appropriate nodes.  It must detect whether a node
    319 executing a processing stage has died, and re-execute the stage on an
    320 alternate node.
    321 
    322 The completed stages are reported back to the Scheduler.
    323 
    324 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    325 
    326 \paragraph{Pixel DB}
    327 \label{sec:pixeldb}
    328 
    329 The Pixel DB is responsible for storing and maintaining the location
    330 of pixel data in the IPP, including the raw images from the telescope,
    331 the master calibration images, the reference static-sky images, and
    332 any temporary image data products produced by the IPP.  It provides
    333 this information upon request to the Localiser. 
    334 
    335 Note that this design assumes that the pixel data will be stored on
    336 the same nodes that will be doing the processing.
    337 
    338 The Pixel DB will be periodically ``published'' as the quality of the
    339 data is assured.  The external world will only have access to the
    340 published version of the Pixel DB.
    341 
    342 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    343 
    344 \paragraph{Nodes}
    345 
    346 The Nodes perform the grunt work of executing the processing stages as
    347 directed by the Controller.  When the processing stage has completed,
    348 they report back to the Controller.
    349 
    350 They may retrieve pixel data from OATS (the Summit) and write it to
    351 local disk when directed to do so by the Controller.  They also may
    352 access the Metadata DB to read configurations, weather information
    353 etc, and to write summary statistics etc.  They may also access the
    354 Object DB to read objects of interest, and to write objects from the
    355 processing stage.
    356 
    357 As they write products, the Nodes register with the Pixel DB that they
    358 have written the requested output (so that the Pixel DB is aware that
    359 the data has been written and is not merely scheduled to be written).
    360 The Nodes do not need to read from the Pixel DB, since everything
    361 (where to read input pixels from, where to write output pixels to) is
    362 specified by the Localiser.
    363 
    364 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    365 
    366 \paragraph{Object DB}
    367 
    368 The Object DB is a facility to store all of the information about
    369 astronomical objects, including individual measurements of objects on
    370 the images, the summary information about those objects, and reference
    371 object data\footnote{Note that this is (possibly) a separate entity
    372 from the object database being developed by SAIC.}.
    373 
    374 The Nodes, CSPs and MOPS may read objects from the Object DB, and the
    375 Nodes may write objects (either new objects or updates), and the CSPs
    376 and MOPS may write certain fields of objects (e.g., the external
    377 identifiers and class of object).
    378 
    379 The Object DB will be periodically ``published'' as the quality of the
    380 data is assured.  The external world will only have access to the
    381 published version of the Object DB.  The published version of the
    382 Object DB will likely be the DB being developed by SAIC.
    383 
    384 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    385 
    386 \paragraph{CSPs and MOPS}
    387 
    388 The Client Science Programs (CSPs) and the Moving Object Processing
    389 System (MOPS) are not a part of the IPP, but are external systems.  We
    390 include them here to show the required interfaces.
    391 
    392 The CSPs and MOPS may query the Pixel DB, the Metadata DB and the
    393 Object DB.  In addition, they may write certain fields to the object
    394 DB (e.g., the external identifiers and class of object) as they
    395 process objects, and they may retrieve pixel data from the Nodes.
    396 
    397 Since ``CSPs'' is a vague term, we now give some examples which may
    398 help to illustrate the functionality.
    399 
    400 One example of a CSP is a web front-end to retrieve (published) images
    401 and objects from the Pixel DB and Object DB.
    402 
    403 Another example would be a program interested in searching for
    404 transiting extrasolar planets.  Such a program may periodically poll
    405 the Metadata DB for new processed observations in its region of
    406 interest (such as the Galactic Plane), retrieve the object photometry
    407 of all high signal-to-noise stars in the processed observations, and
    408 attempt to identify a planetary transit in progress.
    409 
    410 Yet another example would be a Stationary Transient Object Pipeline,
    411 which would periodically poll the Metadata DB for new processed
    412 observations, and query the Object DB for variable sources which were
    413 identified twice (so that they are not moving objects).
    414 
    415 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    416 
    417 \paragraph{Related/Connected components}
    418 
    419 The Pollster may be contained within the Scheduler (i.e., the
    420 Scheduler may initiate and/or schedule as a processing stage the
    421 Pollster), but this is not assumed to be so in this document; this
    422 decision is left to the implementation.
    423 
    424 The Localiser is strongly coupled to the Pixel DB, and throughout this
    425 document, these are both referred to as components of the ``IPP Pixel
    426 Server''.
    427 
    428 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    429 
    430 \paragraph{Responsibility}
    431 
    432 The IPP team will develop and have responsibility for maintaining
    433 these systems.
    434 
    435 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    436 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    437 
    438 \subsubsection{Processing Stages}
    439 \label{sec:processingStages}
    440 
    441 We now consider the collection of IPP processing stages which are
    442 executed by the Controller on the Nodes.  We define a ``stage'' to be
    443 the largest complete task which may be performed in serial without
    444 interation between parallel threads.
    445 
    446 Depending on the particular stage, it may process individual images,
    447 collections of images, or on derived data products.  Because of the
    448 nature of the image data, many of the analysis stages can be run in
    449 parallel because, for example, the analysis of a chip in one image
    450 does not depend on the results from another chip.
    451 
    452 The data analysis stages are divided into several categories as follows:
    453 
    454 \begin{enumerate}
    455 \item Retrieval Stage --- pixel data are retrieved from OATS (the
    456   Summit).
    457 \item Science Image Processing Stages
    458   \begin{enumerate}
    459   \item Phase 1: image processing preparation --- estimates
    460     first-order astrometric and photometric solutions required to
    461     process each major frame.
    462   \item Phase 2: image reduction --- produces calibrated chips from
    463     raw chips.
    464   \item Phase 3: exposure analysis --- processes an FPA to produce
    465     unified and consistent backgrounds, photometry and astrometry for
    466     the component chips.
    467   \item Phase 4: image combination --- processes sky cells overlapped
    468     by a major frame.
    469   \end{enumerate}
    470 \item Calibration Image Processing Stages
    471   \begin{enumerate}
    472   \item Cal 1: Basic master-detrend creation --- combination of simple
    473     detrend images (e.g., bias, dome flat etc).
    474   \item Cal 2: Sky-model/fringe-mode generation --- combination of
    475     more-complicated detrend images (e.g., fringe, scattered light
    476     etc).
    477   \item Cal 3: Flat-field correction image creation --- analysis of
    478     photometry from multiple dithered FPAs.
    479   \end{enumerate}
    480 \item Calibration Test Processing Stage
    481   \begin{enumerate}
    482     \item CalTest 1: Detrend frame testing --- tests whether new
    483       calibration frames are required.
    484     \item CalTest 2: Photometric float correction testing --- tests
    485       whether a new photometric flat correction is required.
    486   \end{enumerate}
    487 \item Reference Catalog Processing Stages
    488   \begin{enumerate}
    489   \item Astrometry reference catalog generation --- processing of the
    490     astrometric data to determine and apply a consistent global
    491     solution.
    492   \item Photometry reference catalog generation --- processing of the
    493     photometric data to determine and apply a consistent global
    494     solution.
    495   \end{enumerate}
    496 \end{enumerate}
    497 
    498 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    499 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    500 
    501 \subsubsection{Hardware Systems}
    502 
    503 The basic IPP hardware organization is shown in Figure~\ref{hardware}.
    504 The overall hardware organization, with a Detector subcluster and a
    505 Static Sky subcluster, is largely chosen to reduce the I/O load during
    506 the pre-reduction analysis of the raw science images.  In addition, we
    507 have specified distinct machines to maintain the object and metadata
    508 databases.  \tbd{This last aspect is largely theoretical until we have
    509 defined the details of these databases; it may be more appropriate
    510 depending on the eventual solutions to distribute these database
    511 elements across the Detector and Static Sky subclusters.}
    512 
    513 \begin{figure}
     353
     354The Image Server provides file pointers (in C), handles (in Perl), or
     355file names corresponding to the instances of the storage objects.
     356Image Server requires a file system which provides files in the local
     357file system.  This may be done over many machines with a network file
     358system such as NFS or GFS.  \tbd{select file system for IPP / test NFS
     359vs GFS vs Mogile, etc}.
     360
     361The IPP Image Server provides the storage and access mechanisms, but
     362it does not include any logic or information about the data.  The
     363Image Server does not, e.g., monitor the age of images and delete them
     364on some schedule.
     365
     366The IPP Image Server consists of the following components:
     367
     368\begin{itemize}
     369\item Image Server storage hardware
     370\item Image Server database
     371\item Image Server daemon
     372\item Image Server client APIs
     373\end{itemize}
     374
     375\paragraph{IPP Image Server Client APIs}
     376
     377Clients interact with the IPP Image Server with a small number of C
     378APIs (Bindings are also provided for Perl \tbr{and Python}).  The
     379client commands are:
     380
     381\begin{itemize}
     382\item {\tt new object}: create a new storage object in the Image
     383  Server.  This function takes as input the file ID and returns a
     384  C-style file pointer or a Perl file handle to the instance of the
     385  storage object.  The arguments to the function include an optional
     386  node name on which the new storage object must be located.  If this
     387  target is not given, the Image Server places the new storage object
     388  on an appropriate machine from the pool (least filled?  most data?
     389  randomized?  the details need to be decided).
     390
     391\item {\tt open object}: open an instance of an existing storage
     392  object, as identified by the file ID.  This function may also
     393  specify the node on which the object should be opened (if an
     394  instance of the object is not stored on that node, the function
     395  returns an error).  On success, the function returns a file pointer.
     396
     397\item {\tt find object}: return a list of filenames in the UNIX name
     398  space associated with the storage object identified by the given
     399  file ID.  Since there are in general multiple instances for a given
     400  storage object, this function returns the collection of all
     401  available instances.  These may be freely opened by the client
     402  server using the standard \code{fopen} functions.
     403
     404\item {\tt stat object}: returns status information about the
     405  specified storage object, including the number of instances of the object.
     406
     407\item {\tt increment object count}: adds a new instance of the given
     408  storage object.  The target node may be optionally specified,
     409  otherwise an appropriate node is selected.
     410
     411\item {\tt decrement object count}: removes one of the instances of
     412  the storage object.  The input parameters may optionally specify the
     413  target machine to delete.
     414
     415\item {\tt delete object}: deletes all instances of the storage object
     416  and sets the storage object status to {\tt deleted}. 
     417\end{itemize}
     418
     419\subsubsection{IPP Image Server Daemon}
     420
     421The Image Server client requests are mediated via the Image Server
     422daemon.  Communication between the clients and the server is via
     423\tbr{SOAP (or flat text commands)} implementing the commands above.
     424
     425\subsubsection{IPP Image Server Database}
     426
     427The IPP Image Server daemon uses a database to store the information
     428about the data storage objects, their instances, and the available
     429hardware resources.  A \tt{mysql} database engine is used to manage
     430the database.  The database tables defined for the Image Server are
     431listed in Table~\ref{ImageServerTables}, and their current contents
     432are listed in Appendix A.  This database engine need not the same one
     433as used for the IPP Metadata Database.
     434%
     435\begin{table}
    514436\begin{center}
    515 \resizebox{8cm}{!}{\includegraphics{pics/hardware}}
    516 \caption{ \label{hardware} IPP Hardware Organization}
     437\caption{Image Server Database Tables\label{ImageServerTables}}
     438\begin{tabular}{ll}
     439\hline
     440\hline
     441{\bf Table Name} & {\bf Description} \\
     442\hline
     443\code{storage_object}  & all storage objects known to Image Server \\
     444\code{instance}        & all instances of all storage objects \\
     445\code{volume}          & data storage devices known to Image Server \\
     446\hline
     447\end{tabular}
    517448\end{center}
    518 \end{figure}
    519 
    520 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    521 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    522 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    523 
    524 \subsection{Software Hierarchy}
    525 
    526 In order to facilitate testing and development, and to encourage
    527 flexibility, the IPP will be built in a layered fashion.  The lowest
    528 level functions will be written in C and collected together into a
    529 \PS{} library.  These library functions will be used to write more
    530 complex modules.  The modules will be written in C but will make use
    531 of the SWIG tool to make their functionality available within other
    532 frameworks.  In particular, the modules can be tied together with a
    533 simple framework (an `engine') or with detailed flow-control through
    534 the use of a high-level language such as Perl, Python, or Tcl
    535 employing the SWIG interfaces.  For the high-level functions in the
    536 operational system, the IPP will make use of \tbd{Python} as the
    537 scripting language to provide the required flow-control to tie the
    538 modules together.
    539 
    540 This approach satisfies the requirement that complicated low-level
    541 analysis steps run fast, while preserving flexibility for coding the
    542 high-level wrappers for which the speed requirements are not so
    543 stringent.
    544 
    545 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    546 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    547 
    548 \subsubsection{External Libraries}
    549 
    550 \PS{} will employ several external libraries to save duplicating
    551 functionality that is already available.  These external libraries
    552 will be wrapped by the \PS{} Library, insulating the project from the
    553 implementation details of the external libraries.  Examples of the
    554 external libraries are FFTW and SLALib.
    555 
    556 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    557 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    558 
    559 \subsubsection{\PS{} Library}
    560 
    561 The \PS{} Library will consist of C structures describing the basic
    562 data types needed by the IPP and C functions which perform the basic
    563 data manipulation operations.  Note that a subset of the library
    564 functions will be provided with SWIG interfaces as well to allow for
    565 their use in the creation of the processing stages.  Examples of the
    566 \PS{} Library are fourier transforms and transforming between pixel
    567 and celestial coordinates.
    568 
    569 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    570 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    571 
    572 \subsubsection{Modules}
    573 
    574 The IPP analysis stages are broken down into modules which represent
    575 specific functional operations.  The modules will be written in C
    576 using the \PS{} Library functions and will be grouped into a \PS{}
    577 Module Library.  The modules will be provided with SWIG interfaces to
    578 all public APIs for their use in processing stages.  Examples of
    579 modules are overscan subtraction and image combination.  Some modules
    580 (e.g.\ find objects on an image) will be used by multiple stages.
    581 
    582 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    583 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    584 
    585 \subsubsection{Stages}
    586 
    587 The major IPP processing tasks are organized into stages, which
    588 consist of multiple modules.  Each stage represents a collection of
    589 complex operations performed on a single data entity.  Each stage
    590 therefore represents the maximum amount of effort which can be
    591 performed in serial without interaction between parallel threads.  The
    592 stages will be written in \tbd{Python}, linking the modules together.
    593 Examples of stages are Phase 2 (detrend images) and Phase 4 (combine
    594 images from multiple telescopes and search for transients).
    595 
    596 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    597 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    598 
    599 \subsubsection{Orchestration}
    600 
    601 High-level components such as the Scheduler, the Controller and the
    602 Localiser are for process control.  As such, they shall be written in
    603 \tbd{Python} in order to maintain flexibility.
    604 
    605 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    606 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    607 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    608 
    609 \subsection{System Interfaces}
    610 
    611 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    612 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    613 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    614 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    615  
    616 \section{System Architectural Design}
    617 
    618 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    619 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    620 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    621 
    622 \subsection{Architectural Components}
    623 
    624 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    625 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    626 
    627 \subsubsection{Pollster}
    628 
    629 The Pollster simply polls OATS on a regular basis for metadata
    630 (including telescope exposures) which is not known by the IPP (i.e.,
    631 already written in the Metadata DB).  On the discovery of such metadata,
    632 it is written to the Metadata DB.
    633 
    634 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    635 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    636 
    637 \subsubsection{Pixel Server}
    638 
    639 The IPP Pixel Server (IPS) is a repository for all image pixel data
    640 required by the IPP, and fulfills the roles of the Pixel DB
    641 (\S\ref{sec:pixeldb}) and the Localiser (\S\ref{sec:localiser}).  In
    642 addition, it also provides components for managing the distribution of
    643 data, and accessing the data.
    644 
    645 Images may reside in the IPS for different periods depending on their
    646 use and type.  Data stored by the IPS include the raw images, the
    647 calibration images, intermediate processing stage images as needed,
    648 final processed images, difference images, and image subsections,
    649 \tbd{along with the associated metadata}.  The IPS must retain images
    650 as long as they are needed, up to the lifetime of the project.  In
    651 order to achieve the I/O requirements, the IPS may maintain the pixel
    652 data distributed across the processor nodes in an organized fashion,
    653 i.e.\ associating specific machines with specific detectors.  The IPS
    654 interacts with the IPP Metadata Database to allow other systems or
    655 subsystems to identify the available images meeting specified
    656 criteria.  IPS specifications are described in the IPS subsystem
    657 specification.
    658 
    659 In addition to storing the pixel data, the IPS is responsible for
    660 acquiring new image data and metadata from the Summit Pixel Server and
    661 making it available for processing by the IPP System.
    662 
    663 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    664 
    665 \paragraph{IPP Pixel Server Components}
    666 
    667 The IPP Pixel Server (IPS) fulfills the roles of the Pixel DB
    668 (\S\ref{sec:pixeldb}) and the Localiser (\S\ref{sec:localiser}), and
    669 consists of the following components:
    670 
    671 \begin{enumerate}
    672 \item IPP Pixel Server Data Locality Optimizer (IPSDLO)
    673 \item IPP Pixel Server Database (IPSD)
    674 \item IPP Pixel Server Maintainance (IPSM)
    675 \item IPP Pixel Server I/O Library (IPSIOL)
    676 \end{enumerate}
    677 
    678 This assumes that the pixel data will be stored on the nodes.  Each
    679 image shall have a unique Universal Resource Identifier (URI) which
    680 specifies the location of the pixel data.  As an example, consider a
    681 cluster with cross-mounted disks --- in this case, the filename
    682 incorporating the full path would serve as the URI.
    683 
    684 The components of the IPS and their relation to other components (both
    685 within the IPS and without) are showin in Figure~\ref{fig:ips}.
    686 
    687 \begin{figure}
    688 \psfig{file=pics/IPS,width=15cm,angle=0}
    689 \caption{The components of the IPS.  In addition to the IPSDLO, IPSD
    690 and IPSM, the IPSIOL is also a component of the IPS; use of the IPSIOL
    691 is shown as dotted arrows in the interactions.  Note that the nodes use
    692 the IPSIOL to pass pixel data between each other.}
    693 \label{fig:ips}
    694 \end{figure}
    695 
    696 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    697 
    698 \subparagraph{IPP Pixel Server Data Locality Optimizer (IPPDLO)}
    699 
    700 Processing stages generated by the Scheduler are passed through the
    701 IPSDLO which does the following:
    702 \begin{enumerate}
    703 \item assigns tasks to specific nodes;
    704 \item identifies the URI of the required input data; and
    705 \item identifies the URI the output data should be written to.
    706 \end{enumerate}
    707 
    708 This allows the choice of processing node to be optimized so that it
    709 resides on the node which will process it, as well as allowing the
    710 output to be written to the node which requires it for the next
    711 processing stage.
    712 
    713 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    714 
    715 \subparagraph{IPP Pixel Server Database (IPSD)}
    716 \label{sec:ipsd}
    717 
    718 The IPSD maintains a database of URIs for the pixel data on the nodes.
    719 It should be able to return the URI of the pixel data given one of:
    720 \begin{enumerate}
    721 \item an exposure identifier and a chip identifier (raw and processed
    722   pixel data from the telescope);
    723 \item a calibration identifier (detrend pixel data); and
    724 \item a sky cell identifier (summed static sky, reduced and difference
    725   pixel data).
    726 \end{enumerate}
    727 
    728 The IPSD will also contain a history of data management commands and
    729 actions.
    730 
    731 \tbd{Is there a reason why this is not a part of the Metadata DB?}
    732 
    733 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    734 
    735 \subparagraph{IPP Pixel Server Maintenance (IPSM)}
    736 
    737 The IPSM initiates the execution of bulk data management processing
    738 stages.  It may have an automated component which, e.g., monitors the
    739 disk space on each of the nodes and redistributes them if they become
    740 unbalanced.  However, the main intent is that it is used by a human
    741 operator to reorgainise the data, e.g., after a new data optimisation
    742 plan has been formulated, or to delete old data.
    743 
    744 The IPSM passes processing stages to the Controller which executes
    745 them on the specified nodes.
    746 
    747 The IPSM allows four types of operation:
    748 \begin{itemize}
    749 \item Retrieve external data --- to manually trigger the copying of
    750   external data (routine copying of the pixel data from OATS is
    751   handled by the Scheduler).  The IPSM generates {\em retrieve data}
    752   stages which are passed to the Controller for execution.
    753 \item Delete data --- to delete old data.  The IPSM looks up the
    754   location of the data in the IPSD and generates {\em delete data}
    755   stages which are passed to the Controller for execution.
    756 \item Replicate data --- to backup and rearrange data.  The IPSM
    757   generates {\em copy data} stages which are passed to the Controller
    758   for execution.  Note that this mode differs from the ``copy external
    759   data'' mode in that it copies data already within the IPS.
    760 \item Move data --- to reorganise storage.  The IPSM executes a
    761   replication followed by a deletion.
    762 \end{itemize}
    763 
    764 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    765 
    766 \subparagraph{IPP Pixel Server I/O Library (IPSIOL)}
    767 
    768 The IPSIOL provides a mechanism for reading and writing pixel data to
    769 the IPS.  The existence of the IPSIOL insulates the processing stages
    770 from the details of how the pixel data are stored (i.e., the
    771 processing stages need not worry whether the data is stored locally or
    772 remotely).  It will generally be used on the nodes and the IPSDLO,
    773 although other components will also make use of it.
    774 
    775 The IPSIOL will be able to:
    776 \begin{itemize}
    777 \item Open a file specified by a URI --- it may simply open the file
    778   if it exists on the particular node, or it may retrieve the file
    779   over the network.
    780 \item Write a file specified by a URI --- it may simply write the file
    781   if it exists on the particular node, or it may copy the file over
    782   the network.  It should also register with the IPSD that a file
    783   specified by a URI has been written.
    784 \item Delete a file specified by a URI --- it may simply delete the
    785   file if it exists on the particular node, or it may delete the file
    786   over the network.
    787 \item Interface with the IPSD to return a URI given one of the
    788   identifiers in \S\ref{sec:ipsd}.
    789 \end{itemize}
    790 
    791 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    792 
    793 \paragraph{Pixel Data Flow Examples}
    794 
    795 For examples of the operation of the IPS, below we sketch out the
    796 intended sequence of events for common operations.
    797 
    798 Reads during processing:
    799 \begin{enumerate}
    800 \item A processing stage has been passed (from the Scheduler) the URI
    801   for an image that it needs to load into memory.
    802 \item The processing stage uses the IPSIOL to open the image.
    803 \item The processing stage reads the image into local memory in the
    804   usual manner.
    805 \item The processing stage closes the image using the IPSIOL.
    806 \end{enumerate}
    807 
    808 Writes during processing:
    809 \begin{enumerate}
    810 \item A processing stage has been passed (from the Scheduler) the URI
    811   for an image that needs to be saved, e.g., a subtracted image.
    812 \item The processing stage uses the IPSIOL to open the image.
    813 \item The processing stage writes the image in the usual manner.
    814 \item The processing stage closes the image using the IPSIOL.
    815 \end{enumerate}
    816 
    817 Note how the IPSIOL has insulated the processing stage from the details
    818 of the reading and writing.
    819 
    820 Maintenance:
    821 \begin{enumerate}
    822 \item A human operator decides that all the pixel data for chip 12
    823   should be stored on node 3.
    824 \item Operator plugs this into the IPSM.
    825 \item The IPSM queries the IPSD using the IPSIOL.
    826 \item The IPSD returns the URIs for all the pixel data for chip 12.
    827 \item The IPSM generates processing tasks to be executed on the nodes
    828   that will copy the data from the old URIs to a new URI which
    829   specifies node 3.
    830 \item The IPSM generates processing tasks to be executed on the nodes
    831   that deletes the data pointed to by the old URIs.
    832 \item The IPSM reports success to the operator.
    833 \end{enumerate}
    834 
    835 Client Science Pipelines:
    836 \begin{enumerate}
    837 \item A CSP wants some pixel data.
    838 \item The CSP queries the IPSD using the IPSIOL (e.g., asking for a
    839   particular exposure or sky cell).
    840 \item The IPSD returns the URI for the pixel data.
    841 \item The CSP opens the image using the IPSIOL and the URI.
    842 \item The CSP reads the pixel data into memory in the usual manner.
    843 \item The CSP closes the image using the IPSIOL.
    844 \end{enumerate}
    845 
    846 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
     449\end{table}
     450
     451\subsubsection{IPP Image Server Storage Hardware}
     452
     453The IPP Image Server manages data across a collection of computers and
     454possibly on multiple storage devices on those computer nodes.  The
     455Image Server maintains a table of the available data volumes.  The
     456Image Server tracks information about each volume such as the total
     457capacity, the current capacity, the association between computer and
     458data volume.
     459
     460\paragraph{IPP Image Server Maintenance Tools}
     461
     462The IPP Image Server provides a collection of administration tools
     463which allow for maintainence.  These are operations which may be
     464automatically scheduled for the IPP or which may be initiated by a
     465human via a command-shell interface.  The maintainence functions
     466include migrating data between nodes to rebalance the available space
     467(this would only occur for instances which have not been placed on a
     468specific node by the client API).  Other functions include checking
     469for file corruption, which involves sweeping all files on a data
     470volume and comparing the calculated file checksum to the currently
     471recorded value. 
     472
    847473%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    848474
    … …  
    14771103manner given the capabilities of the science pipelines.
    14781104
     1105\paragraph{Pollster}
     1106
     1107The Pollster is a program that polls OATS at regular intervals for the
     1108existence of observations not contained in the Metadata DB.  New
     1109weather and image metadata are written to the Metadata DB.
     1110
     1111There is no reason why this architectural component cannot be
     1112contained within another (such as the Scheduler), but it is shown here
     1113as separate for simplicity.
     1114
     1115A polling model is adopted so that OATS' interface may be kept as
     1116simple as possible --- OATS should not be concerned with whether the
     1117IPP has received notifications.  Under this polling model, it is
     1118specifically the responsibility of the IPP to retrieve from OATS the
     1119metadata that is not not already in the Metadata DB.
     1120
     1121\subsubsection{Pollster}
     1122
     1123The Pollster simply polls OATS on a regular basis for metadata
     1124(including telescope exposures) which is not known by the IPP (i.e.,
     1125already written in the Metadata DB).  On the discovery of such metadata,
     1126it is written to the Metadata DB.
     1127
    14791128%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    14801129%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    … …  
    15251174%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    15261175
    1527 \subsection{Processing Stages}
     1176\subsection{Analysis Tasks and Stages}
    15281177
    15291178In this section, we review the processing stages which are executed on
    … …  
    23261975%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    23271976%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
     1977
     1978\subsection{Software Hierarchy}
     1979
     1980In order to facilitate testing and development, and to encourage
     1981flexibility, the IPP will be built in a layered fashion.  The lowest
     1982level functions will be written in C and collected together into a
     1983\PS{} library.  These library functions will be used to write more
     1984complex modules.  The modules will be written in C but will make use
     1985of the SWIG tool to make their functionality available within other
     1986frameworks.  In particular, the modules can be tied together with a
     1987simple framework (an `engine') or with detailed flow-control through
     1988the use of a high-level language such as Perl, Python, or Tcl
     1989employing the SWIG interfaces.  For the high-level functions in the
     1990operational system, the IPP will make use of \tbd{Python} as the
     1991scripting language to provide the required flow-control to tie the
     1992modules together.
     1993
     1994This approach satisfies the requirement that complicated low-level
     1995analysis steps run fast, while preserving flexibility for coding the
     1996high-level wrappers for which the speed requirements are not so
     1997stringent.
     1998
     1999\subsubsection{External Libraries}
     2000
     2001\PS{} will employ several external libraries to save duplicating
     2002functionality that is already available.  These external libraries
     2003will be wrapped by the \PS{} Library, insulating the project from the
     2004implementation details of the external libraries.  Examples of the
     2005external libraries are FFTW and SLALib.
     2006
     2007\subsubsection{\PS{} Library}
     2008
     2009The \PS{} Library will consist of C structures describing the basic
     2010data types needed by the IPP and C functions which perform the basic
     2011data manipulation operations.  Note that a subset of the library
     2012functions will be provided with SWIG interfaces as well to allow for
     2013their use in the creation of the processing stages.  Examples of the
     2014\PS{} Library are fourier transforms and transforming between pixel
     2015and celestial coordinates.
     2016
     2017\subsubsection{Modules}
     2018
     2019The IPP analysis stages are broken down into modules which represent
     2020specific functional operations.  The modules will be written in C
     2021using the \PS{} Library functions and will be grouped into a \PS{}
     2022Module Library.  The modules will be provided with SWIG interfaces to
     2023all public APIs for their use in processing stages.  Examples of
     2024modules are overscan subtraction and image combination.  Some modules
     2025(e.g.\ find objects on an image) will be used by multiple stages.
     2026
     2027\subsubsection{Stages}
     2028
     2029The major IPP processing tasks are organized into stages, which
     2030consist of multiple modules.  Each stage represents a collection of
     2031complex operations performed on a single data entity.  Each stage
     2032therefore represents the maximum amount of effort which can be
     2033performed in serial without interaction between parallel threads.  The
     2034stages will be written in \tbd{Python}, linking the modules together.
     2035Examples of stages are Phase 2 (detrend images) and Phase 4 (combine
     2036images from multiple telescopes and search for transients).
    23282037
    23292038\subsection{Modules}
    … …  
    32652974\section{Appendices}
    32662975
     2976\subsection{Image Server Database Tables}
     2977
     2978\begin{table}
     2979\begin{center}
     2980\caption{Storage Object Table Contents\label{ImageServerTables:SO}}
     2981\begin{tabular}{ll}
     2982\hline
     2983\hline
     2984{\bf Column Name} & {\bf Datatype} & {\bf Description} \\
     2985\hline
     2986\code{so_id}      & integer        & internal storage object identifier \\
     2987\code{ext_id}     & string         & external storage object identifier (file ID) \\
     2988\code{comment}    & string         & user description of object \\
     2989\code{epoch}      & time/date      & last date of access \\
     2990\hline
     2991\end{tabular}
     2992\end{center}
     2993\end{table}
     2994
     2995\begin{table}
     2996\begin{center}
     2997\caption{Instance Table Contents\label{ImageServerTables:INT}}
     2998\begin{tabular}{ll}
     2999\hline
     3000\hline
     3001{\bf Column Name} & {\bf Datatype} & {\bf Description} \\
     3002\hline
     3003\code{ins_id}     & integer        & internal instance identifier \\
     3004\code{so_id}      & integer        & key to storage object table \\
     3005\code{uri}        & string         & location in hardware collection \\
     3006\code{sha1sum}    & string         & checksum information \\
     3007\code{assigned_location} & boolean & is location user-specified? \\
     3008\code{epoch}      & time/date      & last date of access \\
     3009\code{atime}      & time/date      & last date of access \\
     3010\hline
     3011\end{tabular}
     3012\end{center}
     3013\end{table}
     3014
     3015\begin{table}
     3016\begin{center}
     3017\caption{Volume Table Contents\label{ImageServerTables:VOL}}
     3018\begin{tabular}{ll}
     3019\hline
     3020\hline
     3021{\bf Column Name} & {\bf Datatype} & {\bf Description} \\
     3022\hline
     3023\code{vol_id}     & integer        & internal volume identifier \\
     3024\code{uri}        & string         & node name? \\
     3025\hline
     3026\end{tabular}
     3027\end{center}
     3028\end{table}
    32673029
    32683030\bibliographystyle{plain}

  • trunk/doc/design/ippSRS.tex

    r1399 r2114  
    1  %%% $Id: ippSRS.tex,v 1.7 2004-08-06 19:06:01 eugene Exp $
     1 %%% $Id: ippSRS.tex,v 1.8 2004-10-14 05:06:32 eugene Exp $
    22\documentclass[panstarrs,spec]{panstarrs}
    33
    … …  
    154154\section{Requirements}
    155155
     156\begin{table}
     157\begin{center}
     158\caption{Valid Moon Conditions for the 6 PS filters\label{moonconditions}}
     159\begin{tabular}{lrrrr}
     160\hline
     161\hline
     162filter & phase (days) & min. distance (degrees) \\
     163\hline
     164g &  6 & 60 \\
     165r &  5 & 40 \\
     166i &  4 & 30 \\
     167z &  2 & 20 \\
     168y &  1 & 10 \\
     169w &  5 & 50 \\
     170\hline
     171\end{tabular}
     172\end{center}
     173\end{table}
     174
    156175\subsection{Top-Level Requirements}
    157176\label{req:system-capabilities}
    … …  
    162181
    163182\begin{enumerate}
    164 \item Produce reduced science images for each full camera exposure
    165   which are photometrically consistent across the field to within 1\%.\VER{ANALYSIS}{SCD:3.2.2.5}
     183\item For images obtained in photometric weather, produce reduced
     184  science images for each full camera exposure with photometric
     185  zero-point scatter less than 1\% across the full
     186  field. \VER{ANALYSIS}{SCD:3.2.2.5}
    166187  \label{TLR:1}
    167188
    168 \item Produce reduced science images for each full camera exposure
    169   which are photometrically calibrated to within 1\%.\VER{ANALYSIS}{SCD:3.2.2.5}
     189\item For images obtained in photometric weather, produce reduced
     190  science images for each full camera exposure which are
     191  photometrically calibrated with respect to the Pan-STARRS filter
     192  system with a 1$\sigma$ accuracy of 1\%.\VER{ANALYSIS}{SCD:3.2.2.5}
    170193  \label{TLR:2}
    171194
    172 \item Produce reduced science images for each full camera exposure
    173   which are astrometrically calibrated to 100 milliarcseconds to an
    174   absolute reference.\VER{ANALYSIS}{SCD:3.2.2.6}
     195\item For images obtained under normal seeing conditions and optical
     196  distortion, produce reduced science images for each full camera
     197  exposure with an astrometric calibration providing $< 30$
     198  milliarcsecond scatter (1$\sigma$) for sequential images of the same
     199  location.\VER{ANALYSIS}{SCD:3.2.2.7}
     200  \label{TLR:4}
     201
     202\item For images obtained under normal seeing conditions and optical
     203  distortion, produce reduced science images for each full camera
     204  exposure with an astrometric calibration providing $< 100$
     205  milliarcsecond scatter (1$\sigma$) relative to the ICRS reference
     206  system.\VER{ANALYSIS}{SCD:3.2.2.6}
    175207  \label{TLR:3}
    176208
    177 \item Produce reduced science images for each full camera exposure
    178   which are astrometrically consistent to 30
    179   milliarcseconds.\VER{ANALYSIS}{SCD:3.2.2.7}
    180   \label{TLR:4}
    181 
    182 \item Produce reduced science images for each full camera exposure
    183   which have foreground emission subtracted with no more than 1\%
    184   variation in the non-astronomical background.\VER{ANALYSIS}{SCD:3.5.12}
     209\item In photometric weather and under moon conditions listed in
     210  Table~\ref{moonconditions}, produce reduced science images for each
     211  full camera exposure which have background variations of less than
     212  1\% in regions free of large ($> 30$ pixels diameter) astronomical
     213  structures.\VER{ANALYSIS}{SCD:3.5.12}
    185214  \label{TLR:5}
    186215
    187 \item Merge all $g$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10}
     216\item In photometric weather, produce reduced science images for each
     217  full camera exposure which have background deviations from the
     218  static sky in the same filter of less than \tbd{1\%} for the median
     219  in large ($> 30$ pixels diameter) regions.\VER{ANALYSIS}{SCD:3.5.12}
     220  \label{TLR:5a}
     221
     222\item Merge all $g$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
    188223  \label{TLR:6}
    189224
    190 \item Merge all $r$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10}
     225\item Merge all $r$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
    191226  \label{TLR:7}
    192227
    193 \item Merge all $i$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10}
     228\item Merge all $i$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
    194229  \label{TLR:8}
    195230
    196 \item Merge all $z$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10}
     231\item Merge all $z$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
    197232  \label{TLR:9}
    198233
    199 \item Merge all $y$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10}
     234\item Merge all $y$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
    200235  \label{TLR:10}
    201236
    202 \item Merge all $w$ filter science images into a static sky image.\VER{TEST}{SCD:3.2.2.10}
     237\item Merge all $w$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
    203238  \label{TLR:11}
    204239
    205 \item Detect and classify objects on the individual processed science images.\VER{TEST}{SCD:3.2.2.16}
     240\item Detect and classify objects on the individual processed science
     241  images.\VER{TASK}{SCD:3.2.2.16}
    206242  \label{TLR:12}
    207243
    208 \item Detect and classify objects on the stacked groups of science images.\VER{TEST}{SCD:3.2.2.16}
     244\item Detect and classify objects on the stacked groups of science
     245  images.\VER{TASK}{SCD:3.2.2.16}
    209246  \label{TLR:13}
    210247
    211 \item Detect and classify objects on the static sky image.\VER{TEST}{SCD:3.2.2.16}
     248\item Detect and classify objects on the static sky image.\VER{TASK}{SCD:3.2.2.16}
    212249  \label{TLR:14}
    213250
    214 \item Detect all significant transients in the individual science
    215   images relative to the static sky image.\VER{TEST}{SCD:3.2.2.16}
     251\item Detect transients with significance $>3\sigma$ in the individual
     252  science images relative to the static sky
     253  image.\VER{ANALYSIS}{SCD:3.2.2.16}
    216254  \label{TLR:15}
    217255
    218 \item Degrade the stacked image by no more than \tbr{10 milliarcseconds}.\VER{ANALYSIS}{SCD:3.5.2}
     256\item Degrade the stacked image by no more than \tbr{10
     257  milliarcseconds} (FWHM added in quadrature) over the theoretical
     258  limit for the stack under infinite
     259  sampling.\VER{ANALYSIS}{SCD:3.5.2}
    219260  \label{TLR:16}
    220261
    221262\item Perform the processing of science images to the level of
    222263  transient detection and static sky inclusion at a rate such that
    223   exposures taken at a cadence of \tbr{40} seconds do not accumulate
    224   in the processing buffer.\VER{TEST}{SCD:3.2.2.3}
     264  exposures taken at an average cadence of \tbr{40} seconds do not
     265  accumulate in the processing buffer (average throughput
     266  requirement).\VER{TEST}{SCD:3.2.2.3}
    225267  \label{TLR:17}
    226268
    227269\item Limit the false alarm rate (FAR) to less than \tbr{5\%} for
    228  transient detections $> 5\sigma$ sent to the preferred client science
    229  pipelines.\footnote{note difference with PS-4: 1\%}
    230  \VER{ANALYSIS}{SCD:3.2.2.13}
     270  transient detections $> 5\sigma$ sent to the preferred client
     271  science pipelines.\footnote{note difference with PS-4: 1\%}
     272  \VER{ANALYSIS}{SCD:3.2.2.13}
    231273 \label{TLR:18}
     274
     275\item Perform transient detection to a completeness of \tbr{99\%} at
     276  the completeness for transient detections with a significant $>
     277  5\sigma$.\VER{ANALYSIS}{SCD:xxx}
    232278
    233279\item Publish the static sky images to the Pan-STARRS Published
    234280  Science Products Subsystem (PSPS) once per \tbr{6
    235   months}.\VER{TEST}{SCD:3.2.2.18}
     281  months}.\VER{TASK}{SCD:3.2.2.18}
    236282  \label{TLR:19}
    237283
    238284\item Publish the detected objects to the Pan-STARRS Published Science
    239   Products Subsystem (PSPS) once per month.\VER{TEST}{SCD:3.2.2.18}
     285  Products Subsystem (PSPS) once per month.\VER{TASK}{SCD:3.2.2.18}
    240286  \label{TLR:20}
    241287
    242288\item Send the IPP metadata and received OTIS metadata to the
    243   Pan-STARRS Published Science Products Subsystem (PSPS) weekly.\VER{TEST}{SCD:3.2.2.18}
     289  Pan-STARRS Published Science Products Subsystem (PSPS) weekly.\VER{TASK}{SCD:3.2.2.18}
    244290  \label{TLR:21}
    245291
    … …  
    441487
    442488Timing requirements specified in this document shall be achieved on the
    443 deployed Pan-STARRS analysis computers.\VER{TEST}{allocated}
     489deployed Pan-STARRS analysis computers.\VER{INSPECT}{allocated}
    444490
    445491\subsubsection{Software Configuration}
    … …  
    526572\subsubsection{Image Server}
    527573
     574%% IPP Image Server T & F
     575
     576Image Server tasks and functions:
     577
     578\begin{itemize}
     579
     580\item The IPP Image Server stores images on a distributed collection
     581  of computer disks.  Individual instances of a file are only required
     582  to be stored on a single machine (striping across computers is not a
     583  requirement).
     584
     585\item The IPP Image Server attempts to store an image on a specific
     586  machine if requested by the user.
     587
     588\item If such a request cannot be honored (ie, the machine is down),
     589  the IPP Image Server selects an appropriate machine and notifies the
     590  requesting agent of the new location.
     591
     592\item The IPP Image Server stores multiple copies of each image upon
     593  request, the number of copies specified independently for each file
     594  by the user.
     595
     596\item The IPP Image Server maintains a record of all image copies
     597  currently available in the repository.  This record includes at
     598  least the image name, location (which machine), the image size, and
     599  the state of the image (available, locked,
     600  deleted).
     601
     602\item The IPP Image Server locks images in the repository on request.
     603  Both read (shared) and write (exclusive) locks are provided.  A read
     604  lock prevents write access to the file; a write lock prevents both
     605  read and write access.  Access prevention may be advisory rather
     606  than rigorously enforced.
     607
     608\item The IPP Image Server return the image location (the computer or
     609  computers on which it resides) upon request.
     610
     611\item The IPP Image Server provides a specified image upon request.
     612
     613\item The IPP Image Server deletes images in the repository on
     614  request.
     615\end{itemize}
     616
     617%% IPP Image Server Requirements
     618
     619IPP Image Server requirements:
     620
    528621\begin{enumerate}
    529622\item The IPP Image Server shall accept raw images from the summit at
    530  a sustained rate of 1 exposure (2~GB) per \tbr{40
    531  seconds}. \VER{TEST}{TLR:17, TLR:23}
    532 
    533 \item The IPP Image Server shall store images on a distributed
    534   collection of computer disks.  Individual instances of a file are
    535   only required to be stored on a single machine (striping across
    536   computers is not a requirement).\VER{TEST}{TLR:17, TLR:23}
    537 
    538 \item The IPP Image Server shall attempt to store an image on a
    539   specific machine if requested by the user.\VER{TEST}{TLR:17, TLR:23}
    540 
    541 \item If such a request cannot be honored (ie, the machine is down),
    542   the IPP Image Server shall select an appropriate machine and notify
    543   the requesting agent of the new location.\VER{TEST}{TLR:17, TLR:23}
    544 
    545 \item The IPP Image Server shall store multiple copies of each image
    546   upon request, the number of copies specified independently for each
    547   file by the user.\VER{TEST}{TLR:17, TLR:23}
    548 
    549 \item The IPP Image Server shall maintain a record of all image copies
    550   currently available in the repository.  This record shall include at
    551   least the image name, location (which machine), the image size, and
    552   the state of the image (available, locked, deleted).\VER{INSPECT}{TLR:17, TLR:23}
    553 
    554 \item The IPP Image Server shall lock images in the repository on
    555   request.  Both read (shared) and write (exclusive) locks shall be
    556   provided.  A read lock shall prevent write access to the file; a
    557   write lock shall prevent both read and write access.  \tbr{Access
    558   prevention may be advisory rather than enforced.} \VER{TEST}{TLR:17, TLR:23}
    559 
    560 \item The IPP Image Server shall return the image location (the
    561   computer or computers on which it resides) upon request.\VER{TEST}{TLR:17, TLR:23}
    562 
    563 \item The IPP Image Server shall provide a specified image upon request.\VER{TEST}{TLR:17, TLR:23}
    564 
    565 \item The IPP Image Server shall delete images in the repository on request.\VER{TEST}{TLR:17, TLR:23}
     623 a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds.}
     624 \VER{TEST}{TLR:17, TLR:23}
     625
     626\item The IPP Image Server nodes shall not be offline for more than 12 hours
     627  consecutively or 36 hours per year.\VER{ANALYSIS}{TLR:17}
     628
     629\item The IPP Image Server shall provide a seekable, unix file-system
     630  reference to the specified image.\VER{TEST}{allocated}
    566631
    567632\end{enumerate}
    … …  
    569634\subsubsection{AP Database}
    570635
    571 The purpose of the AP Database is:
    572 \begin{itemize}
    573 \item to enable the photometric calibration of images
    574 \item to enable the astrometric calibration of images
    575 \item to enable the construction of flat-field correction frames
    576 \item to enable the construction of a photometric calibration catalog
    577 \item to enable the construction of an astrometric calibration catalog
    578 \item to monitor the system photometry calibration parameters
    579 \item to monitor the system astrometry calibration parameters
    580 \item to perform the identification of single-occurance transients
    581 \end{itemize}
    582 
     636%%% Table: AP DB parameters
    583637\begin{table}
    584638\begin{center}
    … …  
    603657\end{table}
    604658
    605 \begin{enumerate}
    606 \item The AP Database shall accept and store individual detections and
    607   collections of detections along with information about the image
    608   which provided the detections.\VER{TEST}{TLR:2, TLR:3, TLR:22, TLR:24}
    609 
    610 \item Detections shall be saved as one of several detection classes
    611   (P2, P4$\Sigma$, P4$\Delta$, SS) and the AP Database shall store the
    612   appropriate parameters, listed in Table~\ref{APdetections}, for each
    613   class.\VER{TEST}{TLR:2, TLR:3, TLR:22, TLR:24}
    614 
    615 \item The AP Database shall identify the image which provided the
    616   detection, or in the case of external references, an identifier
    617   specific to the reference source.\VER{TEST}{TLR:2, TLR:3}
    618 
    619 \item The AP Database shall group detections into objects and measure
    620   average parameters of those objects.\VER{ANALYSIS}{TLR:2, TLR:3, TLR:22}
    621 
    622 \item The AP Database shall store parallax and proper motion parameters
    623   for a subset of the average objects.\VER{TEST}{TLR:2, TLR:3, TLR:22}
    624 
    625 \item The AP Database shall store image and filter calibration
    626   information necessary to convert between instrumental magnitudes and
    627   calibrated magnitudes in standard systems.\VER{INSPECT}{TLR:3}
    628 
    629 \item The AP Database shall perform at least the follow queries, with
    630   constraints on the output based on at least time ranges, magnitude
    631   limits, error limits:
    632 
    633  \begin{enumerate}
    634  \item given $(RA,DEC)$ and a Radius, return all objects and/or
    635  detections in the region.\VER{TEST}{TLR:2, TLR:3}
    636 
    637  \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all objects and/or
    638    detections in the region.\VER{TEST}{TLR:2, TLR:3}
    639 
    640  \item given $(RA,DEC)$, return closest object.\VER{ANALYSIS}{TLR:2, TLR:3, TLR:22}
    641 
    642  \item given object ID, return all detections\VER{TEST}{TLR:2, TLR:3}
    643 
    644  \item given detection, return source image data.\VER{TEST}{TLR:2, TLR:3}
    645 
    646  \item given detection, return object.\VER{TEST}{TLR:2, TLR:3, TLR:22}
    647 
    648  \item given $(RA,DEC)$, return all images overlapping coordinate.\VER{ANALYSIS}{TLR:2, TLR:3}
    649 
    650  \item given $(RA,DEC)$ and a Radius, return all images overlapping region.\VER{ANALYSIS}{TLR:2, TLR:3}
    651 
    652  \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping region.\VER{ANALYSIS}{TLR:2, TLR:3}
    653 
    654  \item given detection instrumental magnitude, return derived
    655    magnitudes based on calibration information.\VER{TEST}{TLR:2, TLR:3}
    656 
    657  \item given a collection of detections in a filter, determine the
    658    object average magnitude in that filter.\VER{ANALYSIS}{TLR:2, TLR:3}
    659 
    660  \item given a collection of objects and detections, determine the
    661    individual image zero-points.\VER{ANALYSIS}{TLR:2, TLR:3}
    662 
    663  \item given a region, return all possible combinations of the object
    664    or detection magnitudes $(M_1 - M_2)$.\VER{TEST}{TLR:2, TLR:3}
    665 
    666  \item given a list of $(RA,DEC)$ entries, return all nearest objects.\VER{ANALYSIS}{TLR:2, TLR:3}
    667 
    668  \item given a filter, telescope, or detector, return all calibration
    669    terms and history.\VER{TEST}{TLR:2, TLR:3}
    670 
    671  \item given a detection, return all non-detections from images which
    672    overlapped the detection coordinates.\VER{ANALYSIS}{TLR:2, TLR:3, TLR:22}
    673  \end{enumerate}
    674 
    675 \item The AP Database shall accept detection IDs of moving objects and
    676   label the detections with the identified object.\VER{TEST}{TLR:2, TLR:3, TLR:22}
    677 
    678 \item The AP Database shall accept new detections at the rate
    679   generated by the telescope from the Phase 2 and Phase 4 analysis.
    680   \tbr{Except within 10 degrees of the galactic plane, the AP Database
    681   shall keep up with the incoming rates.}  The expected rates are
    682   listed in Table~\ref{APrates}, along with the total data volume
    683   required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2, TLR:3, TLR:22}
    684 
    685 \item The AP Database shall provide access to external Pan-STARRS
    686   clients to the detected objects within \tbr{5 minute} after the
    687   image is obtained.\VER{TEST}{TLR:22}
    688 \label{IPP:DeReq:29c}
    689 \end{enumerate}
    690 
     659%%% Table: AP DB Throughput
    691660\begin{table}
    692661\begin{center}
    … …  
    713682\end{table}
    714683
     684%% IPP AP DB T & F
     685
     686The purpose of the AP Database is:
     687\begin{itemize}
     688\item to enable the photometric calibration of images
     689\item to enable the astrometric calibration of images
     690\item to enable the construction of flat-field correction frames
     691\item to enable the construction of a photometric calibration catalog
     692\item to enable the construction of an astrometric calibration catalog
     693\item to monitor the system photometry calibration parameters
     694\item to monitor the system astrometry calibration parameters
     695\item to perform the identification of single-occurance transients
     696\end{itemize}
     697
     698The tasks and functions of the AP Database include:
     699
     700\begin{itemize}
     701\item The AP Database accepts and stores individual detections and
     702  collections of detections along with information about the image
     703  which provided the detections.
     704
     705\item Detections are saved as one of several detection classes (P2,
     706  P4$\Sigma$, P4$\Delta$, SS) and the AP Database stores the
     707  appropriate parameters, listed in Table~\ref{APdetections}, for each
     708  class.
     709
     710\item The AP Database identifies the image which provided the
     711  detection, or in the case of external references, an identifier
     712  specific to the reference source.
     713
     714\item The AP Database groups detections into objects on the basis of
     715  positional coincidence and measures average parameters of those
     716  objects.
     717
     718\item The AP Database stores parallax and proper motion parameters for
     719  a subset of the average objects.
     720
     721\item The AP Database stores image and filter calibration information
     722  necessary to convert between instrumental magnitudes and calibrated
     723  magnitudes in standard systems.
     724
     725\item The AP Database performs at least the follow queries, with
     726  constraints on the output based on at least time ranges, magnitude
     727  limits, error limits:
     728
     729 \begin{itemize}
     730 \item given $(RA,DEC)$ and a Radius, return all objects and/or
     731 detections in the region.
     732
     733 \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all objects and/or
     734   detections in the region.
     735
     736 \item given $(RA,DEC)$, return closest object.
     737
     738 \item given object ID, return all detections.
     739
     740 \item given detection, return source image data.
     741
     742 \item given detection, return object.
     743
     744 \item given $(RA,DEC)$, return all images overlapping coordinate.
     745
     746 \item given $(RA,DEC)$ and a Radius, return all images overlapping region.
     747
     748 \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping region.
     749
     750 \item given detection instrumental magnitude, return derived
     751   magnitudes based on calibration information.
     752
     753 \item given a collection of detections in a filter, determine the
     754   object average magnitude in that filter.
     755
     756 \item given a collection of objects and detections, determine the
     757   individual image zero-points.
     758
     759 \item given a region, return all possible combinations of the object
     760   or detection magnitudes $(M_1 - M_2)$.
     761
     762 \item given a list of $(RA,DEC)$ entries, return all nearest objects.
     763
     764 \item given a filter, telescope, or detector, return all calibration
     765   terms and history.
     766
     767 \item given a detection, return all non-detections from images which
     768   overlapped the detection coordinates.
     769 \end{itemize}
     770
     771\item The AP Database shall accept detection IDs of moving objects and
     772  label the detections with the identified object.
     773\end{itemize}
     774
     775%% IPP AP DB Requirements
     776The IPP AP Database has the following performance requirements:
     777
     778\begin{enumerate}
     779\item The AP Database shall accept new detections at the rate
     780  generated by the telescope from the Phase 2 and Phase 4 analysis.
     781  \tbr{Except within 10 degrees of the galactic plane, the AP Database
     782  shall keep up with the incoming rates.}  The expected rates are
     783  listed in Table~\ref{APrates}, along with the total data volume
     784  required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2, TLR:3, TLR:22}
     785
     786\item The AP Database shall provide access to external Pan-STARRS
     787  clients to the detected objects within \tbr{5 minute} after the
     788  image is obtained.\VER{TEST}{TLR:22}
     789  \label{IPP:DeReq:29c}
     790\end{enumerate}
     791
    715792\subsubsection{Metadata Database}
    716793
     794%% Table: Metadata data classes
    717795\begin{table}
    718796\begin{center}
    … …  
    739817\end{table}
    740818
    741 \begin{enumerate}
    742 \item The IPP Metadata Database shall accept metadata from the summit
    743  at a sustained rate of \tbr{1 MB per second}.\VER{TEST}{TLR:17, TLR:21, TLR:25}
    744 
    745 \item The Metadata Database shall store the classes of data listed in
    746   Table~\ref{metadata}.  Thus, the Metadata Database shall store and
    747   provide metadata for all raw images, for processed images, for the
     819%% Metadata DB T & F
     820
     821The Metadata Database tasks and functions:
     822
     823\begin{itemize}
     824\item The Metadata Database stores the classes of data listed in
     825  Table~\ref{metadata}.  Thus, the Metadata Database stores and serves
     826  metadata for all raw images, for processed images, for the
    748827  calibration images (both raw and master), for the extracted object
    749828  lists.  Metadata describing the environmental conditions at the
    750   telescope shall also be stored and provided as needed.
    751   Database.\VER{INSPECT}{TLR:21, TLR:25}
    752 
    753 \item The Metadata Database queries shall have a latency of $< 0.1$ seconds.\VER{TEST}{TLR:17}
    754 
    755 \item The Metadata Database shall be capable of at least 100 queries per second.\VER{TEST}{TLR:17}
     829  telescope is also stored and provided as needed. 
     830
     831\item The Metadata Database responds to simple queries which return
     832  the data in the categories listed in Table~\ref{metadata} based on
     833  the primary data key and with basic constraints of time ranges and
     834  other simple conditional constraints.
     835
     836\item The Metadata database stores the configuration information with
     837  restricted access so that only specific people may change the
     838  information (eg, science parameters available to the science team;
     839  software configuration parameters available to the system
     840  maintainers).
     841\end{itemize}
     842
     843%% Metadata DB Requirements
     844
     845The Metadata Database has the following requirements:
     846
     847\begin{enumerate}
     848\item The IPP Metadata Database shall accept metadata from the summit
     849   at a sustained rate of \tbr{1 MB per 40 second.}\VER{TEST}{TLR:17,
     850   TLR:21, TLR:25}
     851
     852\item The Metadata Database queries shall have a latency of $< 0.1$
     853  seconds.\VER{TEST}{TLR:17}
     854
     855\item The Metadata Database shall be capable of at least 100 queries
     856  per second.\VER{TEST}{TLR:17}
    756857
    757858\item The Metadata Database shall be capable of accepting a total data
    758859  volume after 2 years of operation of 280 GB. \VER{INSPECT}{TLR:25}
    759 
    760 \item The Metadata Database shall respond to simple queries which
    761   return the data in the categories listed in Table~\ref{metadata}
    762   based on the primary data key and with basic constraints of time
    763   ranges and other simple conditional constraints.\VER{TEST}{TLR:17}
    764 
    765 \item The Metadata shall store descriptive information about the raw
    766   images received from the summit and the current state of the data
    767   processing.\TASK
    768 
    769 \item The Metadata shall also store descriptive information for each of
    770   the static sky images currently available.\TASK
    771 
    772 \item Software configuration parameters shall be stored in and
    773   extracted from the Metadata Database.\TASK
    774 
    775 \item The Metadata database shall store the configuration information
    776   with restricted access so that only specific people may change the
    777   information.\VER{TEST}{allocated}
    778 
    779 \item User-configurable software parameters shall be stored in and
    780   extracted from the Metadata Database.\TASK
    781860
    782861\item The Metadata Database shall restrict write access of the
    783862  scientific parameters to a different group from the software and
    784863  hardware configuration parameters.\VER{TEST}{allocated}
    785 
    786864\end{enumerate}
    787865
    788866\subsubsection{Controller}
     867
     868%% IPP Controller T & F
     869
     870 IPP Controller tasks and functions:
     871
     872\begin{itemize}
     873
     874\item On startup, the IPP Controller attempts to establish
     875  communication with all of its computers and set their state to be
     876  {\tt alive} or {\tt dead} based on the success of the
     877  connection.
     878
     879\item The IPP Controller detects computers which crash or stop
     880  responding and set their state to {\tt dead}.
     881
     882\item The IPP Controller attempts to re-establish communication with
     883  {\tt dead} computers.
     884
     885\item The IPP Controller accepts tasks from external users and
     886  systems, which may specify a desired CPU (node) and priority in
     887  addition to the task command.
     888
     889\item The IPP Controller attempts to run pending tasks on the desired
     890  node, if available (not {\tt dead} or {\tt off}).
     891
     892\item If the node is unavailable, the IPP Controller attempts to run
     893  the task on another node.
     894
     895\item If the node is available, the IPP Controller attempts to run a
     896  given task only if no higher-priority tasks are available and no
     897  task is currently being executed.
     898
     899\item The IPP Controller monitors the output from the task and writes
     900  it to an associated log destination.
     901
     902\item The IPP Controller monitors the execution status of each task
     903  currently executing on a node and performs the following actions:
     904
     905  \begin{itemize}
     906  \item identify the task as successful if it has a valid exit status.
     907  \item identify the task as unsuccessful if it has an error exit status.
     908  \item identify the task as unattempted if the computer crashed.
     909  \end{itemize}
     910
     911\item The IPP Controller accepts and performs the following external
     912  commands:
     913  \begin{itemize}
     914  \item add a task to the pending task list.
     915  \item delete a specific task from the pending task list.
     916  \item return the current status of a specific task.
     917  \item return a list of all pending and non-pending tasks.
     918  \item set a specified computer state to {\tt off} or {\tt dead}.
     919  \item restrict a specified CPU to a class of tasks.
     920  \item halt execution of a specified task.
     921  \item set the IPP Controller state to {\tt finish}, {\tt abort}, or {\tt stop}.
     922  \end{itemize}
     923\end{itemize}
     924
     925%% IPP Controller Requirements
     926
     927IPP Controller requirements:
     928
    789929\begin{enumerate}
    790930
    … …  
    792932  computers.\VER{TEST}{TLR:17}
    793933
    794 \item On startup, the IPP Controller shall attempt to establish
    795   communication with all of its computers and set their state to be
    796   {\tt alive} or {\tt dead} based on the success of the connection.\VER{TEST}{TLR:17}
    797 
    798 \item The IPP Controller shall detect computers which crash or stop
    799   responding and set their state to {\tt dead}.\VER{TEST}{TLR:17}
    800 
    801 \item The IPP Controller shall attempt to re-establish communication
    802   with {\tt dead} computers.\VER{TEST}{TLR:17}
    803 
    804 \item The IPP Controller shall accept tasks from external users and
    805   systems, which may specify a desired CPU (node) and priority in
    806   addition to the task command.\VER{TEST}{TLR:17}
    807 
    808 \item The IPP Controller shall attempt to run pending tasks on the
    809   desired node, if available (not {\tt dead} or {\tt off}).\VER{TEST}{TLR:17}
    810 
    811 \item If the node is unavailable, the IPP Controller shall attempt to
    812   run the task on another node.\VER{TEST}{TLR:17}
    813 
    814 \item If the node is available, the IPP Controller shall attempt to run
    815   a given task only if no higher-priority tasks are available and no
    816   task is currently being executed.\VER{TEST}{TLR:17}
    817 
    818 \item The IPP Controller shall monitor the output from the task and
    819   write it to an associated log destination.\VER{TEST}{TLR:17}
    820 
    821 \item The IPP Controller shall monitor the execution status of each
    822   task currently executing on a node and perform the following
    823   actions:
    824 
    825   \begin{enumerate}
    826   \item identify the task as successful if it has a valid exit status.\VER{TEST}{TLR:17}
    827   \item identify the task as unsuccessful if it has an error exit status.\VER{TEST}{TLR:17}
    828   \item identify the task as unattempted if the computer crashed.\VER{TEST}{TLR:17}
    829   \end{enumerate}
    830 
    831 \item The IPP Controller shall accept and perform the following
    832   external commands:
    833   \begin{enumerate}
    834   \item add a task to the pending task list.\VER{TEST}{TLR:17}
    835   \item delete a specific task from the pending task list.\VER{TEST}{TLR:17}
    836   \item return the current status of a specific task.\VER{TEST}{TLR:17}
    837   \item return a list of all pending and non-pending tasks.\VER{TEST}{TLR:17}
    838   \item set a specified computer state to {\tt off} or {\tt dead}.\VER{TEST}{TLR:17}
    839   \item restrict a specified CPU to a class of tasks.\VER{TEST}{TLR:17}
    840   \item halt execution of a specified task.\VER{TEST}{TLR:17}
    841   \item set the IPP Controller state to {\tt finish}, {\tt abort}, or {\tt stop}.\VER{TEST}{TLR:17}
    842   \end{enumerate}
    843 
    844934\item The IPP Controller shall limit command latency to \tbr{$< 0.1$} seconds.\VER{TEST}{TLR:17}
    845935
    … …  
    856946
    857947\subsubsection{Scheduler}
    858 \begin{enumerate}
    859 \item The IPP Scheduler shall send the analysis tasks which it
    860   initiates to the IPP Controller.\VER{TEST}{TLR:17}
    861 
    862 \item All analysis tasks sent by the IPP Scheduler shall include a
    863   complete UNIX command with necessary arguments, the priority of the
    864   task, and optionally the desired processing node.\VER{INSPECT}{TLR:17}
     948
     949%% IPP Scheduler T & F
     950
     951The IPP Scheduler tasks and functions:
     952
     953\begin{itemize}
     954\item The IPP Scheduler sends the analysis tasks which it initiates to
     955  the IPP Controller.
     956
     957\item All analysis tasks sent by the IPP Scheduler include a complete
     958  UNIX command with necessary arguments, the priority of the task, and
     959  optionally the desired processing node.
     960
     961\item When the IPP Scheduler is placed in the {\em paused state}, it
     962  only initiates User-requested tasks.
     963
     964\item When the IPP Scheduler is placed in the {\em interactive state},
     965  it initiates User-requested tasks as well as data transfer tasks.
     966
     967\item When the IPP Scheduler is placed in the {\em automatic state},
     968  it initiates the most appropriate task based on the inputs and
     969  dependency rules.
     970
     971\item The IPP Scheduler sends the exit status of the analysis tasks to
     972  the appropriate destination as defined by the task dependency table.
     973\end{itemize}
     974
     975%% IPP Scheduler Requirements
     976
     977The IPP Scheduler requirements:
     978
     979\begin{enumerate}
     980\item The IPP Scheduler shall publish the static sky images to the
     981  Pan-STARRS PSPS on a time-scale of \tbr{6 month}.\VER{TEST}{TLR:19}
    865982
    866983\item The IPP Scheduler shall query the Databases on a regular basis
    … …  
    869986
    870987\item The IPP Scheduler shall accept new User input in real-time:
    871 within 0.1 seconds of the request.\VER{TEST}{TLR:17}
    872 
    873 \item When the IPP Scheduler is placed in the {\em paused state}, it
    874   shall only initiate User-requested tasks.\VER{TEST}{TLR:17}
    875 
    876 \item When the IPP Scheduler is placed in the {\em interactive state},
    877   it shall initiate User-requested tasks as well as data transfer
    878   tasks.\VER{TEST}{TLR:17}
    879 
    880 \item When the IPP Scheduler is placed in the {\em automatic state},
    881   it shall initiate the most appropriate task based on the inputs and
    882   dependency rules.\VER{TEST}{TLR:17}
    883 
    884 \item The IPP Scheduler shall send the exit status of the analysis
    885   tasks to the appropriate destination as defined by the task
    886   dependency table.\VER{TEST}{TLR:17}
    887 
    888 \item The IPP Scheduler shall publish the static sky images to the
    889   Pan-STARRS PSPS on a time-scale of \tbr{6 month}.\VER{TEST}{TLR:19}
     988  within 0.1 seconds of the request.\VER{TEST}{TLR:17}
    890989
    891990\item The IPP Scheduler shall publish the detected objects to the
    … …  
    9031002  to the MOPS subsystem within 5 minutes of the image exposure
    9041003  time.\VER{TEST}{TLR:22}
    905 
    906 \end{enumerate}
    907 
    908 \subsection{Analysis Stages}
    909 
    910 We now consider the requirements of the analysis tasks which shall be
    911 performed by the IPP.  These tasks represent the core of the required
    912 IPP functionality; the architectural components discussed above can be
    913 viewed as primarily supporting infrastructure to enable the analysis
    914 tasks to be executed on the appropriate data and to store the results.
    915 
    916 \subsubsection{Science Image Analysis}
     1004\end{enumerate}
     1005
     1006%%%%%% Analysis Stages
     1007
     1008%%%% Science Image Analysis Stages
     1009
     1010\subsection{Science Image Analysis Stages}
     1011
     1012We now consider the requirements of the science image analysis tasks
     1013which are performed by the IPP.  These tasks represent the core of the
     1014required IPP functionality; the architectural components discussed
     1015above can be viewed as primarily supporting infrastructure to enable
     1016the analysis tasks to be executed on the appropriate data and to store
     1017the results.
    9171018
    9181019The Science Image analysis stages together represent the basic data
    919 analysis required by the IPP.  There are several requirements which
    920 shall be met by the collection of science image analysis stages as a
    921 group.
     1020analysis required by the IPP.  Integral to our operational concept for
     1021the IPP is the division of the science image analysis into four
     1022phases, each of which represents a complete analysis on a particular
     1023unit of data.  The tasks and functions of these separate stages are
     1024discussed below.
     1025
     1026\subsubsection{General Science Image Analysis Requirements}
     1027There are several requirements which shall be met by the collection of
     1028science image analysis stages as a group.
    9221029
    9231030\begin{enumerate}
    … …  
    9371044 static sky image, and update the corresponding exposure (S/N) maps,
    9381045 at a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17}
    939 
    940 \item The IPP Science Analysis shall detect and measure parameters of
    941 objects on the pre-processed science images.\VER{TEST}{TLR:12}
    942 
    943 \item The IPP Science Analysis shall detect and measure parameters of
    944 objects on the stacked science images.\VER{TEST}{TLR:13}
    945 
    946 \item The IPP Science Analysis shall detect and measure parameters of
    947 objects on the static sky images.\VER{TEST}{TLR:14}
    948 
    949 \item The IPP Science Analysis shall detect and measure parameters of
    950 objects on the difference images.\VER{TEST}{TLR:15}
    951 
    952 \item The IPP Science Analysis shall determine astrometry of the
    953  detected objects relative to an external astrometric reference with
    954  an accuracy of \tbr{750 mas} (for bright objects) in the
    955  Commissioning phase of the telescope.\VER{TEST}{TLR:4, TLR:3}
    956 
    957 \item The IPP Science Analysis shall determine astrometry of the
    958  detected objects relative to an external astrometric reference with
    959  an accuracy of \tbr{250 mas} (for bright objects) during the
    960  construction of the Pan-STARRS Astrometric Reference Catalog.\VER{ANALYSIS}{TLR:4, TLR:3}
    961 
    962 \item The IPP Science Analysis shall determine astrometry of the
    963  detected objects relative to the Pan-STARRS Astrometric Reference
    964  with an accuracy of \tbr{100 mas} (for bright objects) during normal
    965  operations.\VER{ANALYSIS}{TLR:4, TLR:3}
    966 
    967 \item The IPP Science Analysis shall determine photometry of the
    968  detected objects within an internal photometric system with scatter
    969  less than \tbr{25 millimags} (for bright objects) during the
    970  Commissioning phase of the telescope in photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2}
    971 
    972 \item The IPP Science Analysis shall determine photometry of the
    973  detected objects within an internal photometric system with scatter
    974  less than \tbr{10 millimags} (for bright objects) during the
    975  construction of the Pan-STARRS Photometric Reference Catalog in
    976  photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2}
    977 
    978 \item The IPP Science Analysis shall determine photometry of the
    979  detected objects within an internal photometric system with scatter
    980  less than \tbr{5 millimags} (for bright objects) during normal
    981  operations in photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2}
    982 
    983 \item The IPP Science Analysis shall determine photometry of the
    984  detected objects in an external photometric system with scatter less
    985  than \tbr{10 millimags} (for bright objects) during normal operations
    986  in photometric weather.\VER{ANALYSIS}{TLR:1, TLR:2}
    9871046
    9881047\item The maximum latency between the acquisition of an image and the
    … …  
    9971056\end{enumerate}
    9981057
     1058%% Phase 1
    9991059\subsubsection{Phase 1 : image processing preparation}
    10001060
    1001 \begin{enumerate}
    1002 \item the Phase 1 analysis shall execute within 2 seconds for a
    1003   complete FPA image.\VER{TEST}{TLR:17}
    1004 
    1005 \item The Phase 1 analysis stage shall determine the astrometric
    1006   solution of the complete camera (FPA image) with an accuracy of
    1007   \tbr{1 arcsec} peak-to-peak deviation.\VER{TEST}{TLR:3}
    1008 
    1009 \item The Phase 1 analysis stage shall load the guide star pixel and
    1010   celestial coordinates.\TASK
    1011 
    1012 \item If guide stars are not available, the Phase 1 analysis stage
    1013   shall extract bright stars from the image.\TASK
    1014 
    1015 \item This extraction shall be done in less than \tbr{1 second}.\VER{TEST}{TLR:17}
    1016  
     1061Phase 1 is the image processing preparation stage.  The analysis is
     1062performed on a complete FPA.  At the end of this analysis, the FPA is
     1063ready to be analysed in detail in Phase 2.  The Phase 1 tasks and
     1064functions are:
     1065
     1066\begin{itemize}
     1067
     1068\item Extract FPA guide stars to determine astrometry across the full FPA
     1069
     1070\item If no guide stars are available, phase 1 must measure the pixel
     1071  coordinates of known bright stars expected in the field from the
     1072  image data.
     1073
    10171074\item The total number of stars and size of the bright-star
    10181075  acquisition box shall be a user-configurable parameter in the range
    1019   20 - 250.\TASK
    1020 
     1076  20 - 250.
     1077
     1078\item Calculate the Image cell / Sky cell overlaps for each image.
     1079  Sky cells which do not have sufficient science image overlap \tbr{$<
     1080  5\%$} are excluded from the overlap table.
     1081
     1082\end{itemize}
     1083
     1084The Phase 1 requirements are:
     1085
     1086\begin{enumerate}
     1087\item the Phase 1 analysis shall execute within 2 seconds for a
     1088  complete FPA image.\VER{TEST}{TLR:17, allocated}
     1089
     1090\item The Phase 1 analysis stage shall determine the astrometric
     1091  solution of the complete camera (FPA image) with an accuracy of 1
     1092  arcsec peak-to-peak deviation.\VER{TEST}{TLR:3}
     1093
     1094\item Bright-star extraction from the image data shall be performed in
     1095  less than \tbr{1 second}.\VER{TEST}{TLR:17}
     1096 
    10211097\item In order for blind astrometry of an image to succeed, it is
    10221098  necessary that approximate image coordinates be known.  The Phase 1
    1023   analysis shall be able to succeed despite initial coordinate errors
    1024   as large as \tbr{20\arcsec}.\VER{TEST}{TLR:3}
    1025 
    1026 \item The Phase 1 analysis stage shall construct a table of the
    1027   overlaps between the science image to be processed and the static
    1028   sky images.\TASK
    1029 
    1030 \item The overlaps shall be overestimated by a small amount so that
    1031   errors in astrometry at Phase 1 will not cause any valid static sky
    1032   / science image pairs to be missed.\TASK
    1033 
    1034 \item The amount of overlap shall be a user-configurable parameter.\VER{TEST}{TLR:6, TLR:11}
     1099  analysis shall succeed despite initial coordinate errors as large as
     1100  \tbr{20\arcsec}.\VER{TEST}{TLR:3}
    10351101 
    1036 \item Sky cells which do not have sufficient science image overlap
    1037   \tbr{$< 5\%$} shall be excluded from the overlap table.\VER{TEST}{TLR:6, TLR:11}
    1038 
    1039 \item It is not unusual for an image to be obtained with invalid
    1040   coordinates or without any valid stars.  For example, the telescope
    1041   control system may make an error and report the wrong time or
    1042   coordinates.  Or, the image may be obtained in exceptionally poor
    1043   conditions with no detected stars.  Phase 1 shall return a
    1044   descriptive error message in these conditions.\TASK
    1045 \end{enumerate}
    1046 
     1102\end{enumerate}
     1103
     1104%% Phase 2
    10471105\subsubsection{Phase 2 : image reduction}
    10481106
    1049 The Phase~2 analysis is the detrend stage, in which the images from
    1050 the detector are processed to remove instrumental signatures. 
    1051 
    1052 \paragraph{Timing}
    1053 The complete Phase~2 analysis shall be performed in $< 38$ seconds for
    1054 up to 4 complete FPA images at one time. \VER{TEST}{TLR:17}
    1055 
    1056 \paragraph{Processing Recipe}
    1057 \begin{enumerate}
    1058 \item The Phase 2 analysis stage shall consult the processing recipe
    1059   to define the necessary analysis steps performed by the Phase 2
    1060   stage.\TASK
    1061 
    1062 \item Phase 2 shall perform the analysis steps only if required by the
    1063   processing recipe.\TASK
    1064 
    1065 \item The processing recipe shall define the stages to be executed with
    1066   optional exposure time and background flux limits to require or
    1067   exclude select certain stages.\TASK
    1068 \end{enumerate}
    1069 
    1070 \paragraph{Detrend Image Convolutions}
    1071 \begin{enumerate}
    1072 
    1073 \item The Phase 2 analysis stage shall convolve the flat-field and
    1074   high-spatial-frequency fringe images with the OT kernel.\VER{TEST}{TLR:1}
    1075 
    1076 \item The Phase 2 analysis stage shall determine the OT kernel from the
    1077   IPP Metadata Database.\TASK
    1078 
    1079 \item If no OT kernel exists, this step shall be silently skipped.\TASK
    1080 \end{enumerate}
    1081 
    1082 \paragraph{Flag bad and saturated pixels}
    1083 \begin{enumerate}
    1084 
    1085 \item The Phase 2 analysis shall load the basic bad pixel map appropriate to
    1086 the detector of interest.\VER{TEST}{TLR:18}
    1087 
    1088 \item The Phase 2 analysis shall use the OT kernel to grow the traps in the
    1089 raw bad pixel map.  \VER{TEST}{TLR:18}
    1090 
    1091 \item The Phase 2 analysis shall mask saturated pixels and a user-specified
    1092 number of surrounding pixels.\VER{TEST}{TLR:18}
    1093 
    1094 \item The Phase 2 analysis shall mask ghosts of bright stars.\VER{TEST}{TLR:18}
    1095 
    1096 \item Different bits shall be set to identify different reasons for masking
    1097 the pixels.\VER{TEST}{TLR:21}
    1098 \end{enumerate}
    1099 
    1100 \paragraph{Bias correction via overscan subtraction}
    1101 \begin{enumerate}
    1102 
    1103 \item Phase 2 shall perform bias subtraction on the image.\VER{TEST}{TLR:1}
    1104 
    1105 \item Phase 2 shall choose the bias subtraction method and analysis statistic
    1106 based on the user-configured parameters.\TASK
    1107 
    1108 \item The bias correction shall be measured from the image overscan region.\TASK
    1109 
    1110 \item The overscan region shall be determined from the Metadata DB.\TASK
    1111 
    1112 \item The bias subtraction shall be capable of using one of following
    1113 bias corrections, depending on the user parameters:
    1114 
    1115 \begin{enumerate}
    1116 \item subtract a single constant from the image.  \VER{TEST}{TLR:1}
    1117 
    1118 \item subtract a 1-D bias which varies along the overscan.  The function to be used shall include
    1119 a spline or a Chebychev polynomial derived from the data values along
    1120 the overscan, as specified by the user parameters. \VER{TEST}{TLR:1}
    1121 
    1122 \item correct the overscan {\em and} subtract a 2-D bias image which
    1123   has been overscan corrected using one of the two methods above.\VER{TEST}{TLR:1}
    1124 \end{enumerate}
    1125 
    1126 \item The statistic used to calculate the overscan constant or the
    1127 inputs to the spline and polynomial fits shall be derived from groups
    1128 of pixels on the basis of one of several possible statistics, as
    1129 specified by the user parameters.\VER{TEST}{TLR:1}
    1130 
    1131 \item The choice of statistics shall include the sample and robust
    1132 mean, median, and modes.\VER{TEST}{TLR:1}
    1133 
    1134 \item In the case of a single constant, all of the overscan pixel
    1135 values are used in the calculation of this statistic.\VER{TEST}{TLR:1}
    1136 
    1137 \item In the case of the 1D functional representation, the input
    1138 values to the fit shall represent the coordinate along the overscan,
    1139 with the statistic derived from the pixels in the perpendicular
    1140 direction at each location.\VER{TEST}{TLR:1}
    1141 
    1142 \item If specified in the user parameters, sigma-clipping shall be
    1143 performed on the input data values.\VER{TEST}{TLR:1}
    1144 
    1145 \item The bias subtraction shall leave no residuals greater than \tbr{1 DN}
    1146 peak-to-peak.\VER{TEST}{TLR:1}
    1147 \end{enumerate}
    1148 
    1149 \paragraph{Trim object image}
    1150 \begin{enumerate}
    1151 
    1152 \item The Phase 2 analysis shall trim the non-imaging pixels from the
    1153 image.\TASK
    1154 
    1155 \item The definition of the imaging area shall be determined from the
    1156 Metadata Database.\TASK
    1157 
    1158 \item Phase 2 shall trim pixel near the edges that have been
    1159 compromised due to OT operation.\VER{TEST}{TLR:1}
    1160 \end{enumerate}
    1161 
    1162 \paragraph{Correct for non-linearity}
    1163 
    1164 If required by the recipe, each chip shall be independently corrected for the
    1165 effects of non-linearity.\VER{TEST}{TLR:1}
    1166 
    1167 \paragraph{Flat-field correction}
    1168 \begin{enumerate}
    1169 
    1170 \item The Phase 2 analysis shall divide the science image by the
    1171   provided flat-field image.\VER{TEST}{TLR:1}
    1172 
    1173 \item The division shall handle zero-valued pixels in the flat-field
    1174   image without raising floating point exceptions, setting the
    1175   corresponding bit value in the mask.\VER{TEST}{TLR:1}
    1176 
    1177 \item The flat-field images shall be appropriately normalized (see
    1178   section \ref{mkcal}).\VER{TEST}{TLR:1}
     1107Phase 2 is the detrend stage, in which each detector is separately
     1108processed to remove instrumental signatures.  The result of Phase 2 is
     1109an image with high-quality astrometric and photometric calibrations, a
     1110collection of objects detected in the image and characterized in a
     1111rudimentary way (star / non-stellar), and a measurement of the PSF
     1112across the detector. 
     1113
     1114The tasks and functions of Phase 2 are as follows:
     1115
     1116\begin{itemize}
     1117
     1118\item Convolve the flat-field and high-spatial-frequency fringe images
     1119  with the OT kernel.
     1120
     1121\item Mask ghosts of bright stars which introduce residual feature
     1122  more significant than \tbr{1\%} of the background.
     1123
     1124\item Bias subtract the image.
     1125
     1126\item Correct each chip independently for non-linearity.
     1127
     1128\item Flat-field correct the image.
     1129
     1130\item Subtract a fit to the detector-dependent fringing pattern.
     1131
     1132\item Subtract a fit to the low-spatial frequency sky background.
     1133
     1134\item Identify `cosmic rays' on the basis of morphology.
     1135
     1136\item Perform (positive) object detection on the processed images,
     1137  down to a user-configured threshold, likely to be $\sim 20\sigma$.
     1138  The detection threshold may optionally be a function of the average
     1139  background flux or the local noise level.
     1140
     1141\item Measure the following object parameters:
     1142
     1143  \begin{itemize}
     1144  \item object centroid and position errors.
     1145  \item an extended object position ($x_g, y_g$).
     1146  \item instrumental PSF magnitude and error.
     1147  \item local background level and error.
     1148  \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) of the object
     1149    and their covariance matrix.
     1150  \end{itemize}
     1151
     1152\item Perform minimal object classification to distinguish objects
     1153  which are consistent with a single PSF, objects which are
     1154  inconsistently large, objects which are inconsistently small, and
     1155  objects which are saturated.
     1156
     1157\item Match the detected objects with known astrometric reference
     1158  objects, including proper-motion compensation.
     1159
     1160\item Fit the reference and detected object coordinates to determine
     1161  astrometric parameters for the individual OTAs, including
     1162  polynomials of the coordinates up to 3rd order (user-specified
     1163  parameter).  The Cell astrometric parameters are not allowed to vary
     1164  in the fit, which uses outlier rejection to determine a robust
     1165  solution.
     1166
     1167\item Extract subrasters (`postage stamps') surrounding a
     1168  user-specified list of coordinates from the flattened
     1169  images, to be saved in the IPP Image Server.
     1170
     1171\item measure the PSF variation as a function of detector position.
     1172
     1173\end{itemize}
     1174 
     1175The Phase 2 requirements are:
     1176
     1177\begin{enumerate}
     1178\item The complete Phase~2 analysis shall be performed in $< 38$
     1179  seconds for up to 4 complete FPA images at one
     1180  time. \VER{TEST}{TLR:17}
     1181
     1182\item The bias subtraction shall leave no residuals greater than
     1183  \tbr{1 DN} peak-to-peak for images within the normal range of bias
     1184  variations.\VER{TEST}{TLR:1}
     1185
     1186\item The Phase 2 flat-field correction shall handle zero-valued
     1187  pixels in the flat-field image without raising floating point
     1188  exceptions, setting the corresponding bit value in the
     1189  mask.\VER{TEST}{TLR:1}
    11791190
    11801191\item The flat-fielded image shall have a consistent photometric
    1181   zero-point across the chip, and across the full FPA, to within 0.2\%
    1182   with peak-to-peak deviations of \tbr{0.5\%}.\VER{TEST}{TLR:1}
    1183 \end{enumerate}
    1184 
    1185 \paragraph{Sky \& Fringe subtraction}
    1186 \begin{enumerate}
    1187 
    1188 \item The Phase 2 analysis shall subtract the sky (and fringe where
    1189   needed) contributions from the images.\VER{TEST}{TLR:1, TLR:5}
     1192  zero-point across the chip, and across the full FPA, with scatter $<
     1193  0.2\%$ and peak-to-peak deviations of $< 0.5\%$.\VER{ANALYSIS}{TLR:1}
    11901194
    11911195\item The residual after the background subtraction shall have an
    11921196  average offset of 0 counts, excluding the signal from astronomical
    1193   features.\VER{TEST}{TLR:5}
     1197  features.\VER{ANALYSIS}{TLR:5}
    11941198
    11951199\item The background residuals shall have peak-to-peak variations of
    1196   less than \tbr{1\%} of the input background amplitude.\VER{TEST}{TLR:5}
     1200  less than \tbr{1\%} of the input background amplitude.\VER{ANALYSIS}{TLR:5}
    11971201
    11981202\item The background residuals shall have a scatter of less than
    11991203  \tbr{1\%} of the input background amplitude for apertures of less
    1200   than \tbr{10~arcsec}.\VER{TEST}{TLR:1}
    1201 \end{enumerate}
    1202 
    1203 \paragraph{Identify `cosmic rays'}
    1204 \begin{enumerate}
    1205 
    1206 \item The Phase 2 analysis shall detect cosmic rays with flux $>
    1207   5\sigma$ by morphology in single images with an efficiency of $> 95$\%.
    1208   \VER{TEST}{TLR:18}
    1209 
    1210 \item The Phase 2 analysis shall mask detected cosmic rays with a
    1211   unique bit value in the mask.\TASK
    1212 
    1213 \item The Phase 2 analysis shall extend the masked region by a
    1214   user-configurable growth factor.\TASK
    1215 
    1216 \item The Phase 2 analysis shall perform the cosmic ray detection only
    1217   if it is required by the analysis recipe.\TASK
    1218 \end{enumerate}
    1219 
    1220 \paragraph{Find objects in the image}
    1221 \begin{enumerate}
    1222 
    1223 \item The Phase 2 analysis shall perform object detection on the
    1224   processed images.\VER{TEST}{TLR:12}
    1225 
    1226 \item The object detection process shall detect all objects above a
    1227   user-configured threshold.\TASK
    1228 
    1229 \item The threshold shall be a positive value; negative values shall
    1230   invoke an error.\TASK
    1231 
    1232 \item The detection threshold shall optionally be a function of the
    1233   average background flux or the local noise level.\TASK
    1234 
    1235 \item The object detection shall measure the following object
    1236   parameters:
    1237   \begin{enumerate}
    1238   \item object centroid and position errors\VER{TEST}{TLR:12}
    1239   \item an extended object position ($x_g, y_g$)\VER{TEST}{TLR:12}
    1240   \item instrumental PSF magnitude and error\VER{TEST}{TLR:12}
    1241   \item local background level and error\VER{TEST}{TLR:12}
    1242   \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) of the object
    1243     and their covariance matrix\VER{TEST}{TLR:12}
    1244   \end{enumerate}
    1245 
    1246 \item Minimal object classification shall be performed to distinguish
    1247   objects which are consistent with a single PSF, objects which are
    1248   inconsistently large, objects which are inconsistently small, and
    1249   objects which are saturated.\VER{TEST}{TLR:12}
    1250 
    1251 \item The resulting collection of detected objects shall be saved along
    1252   with the relevant image metadata (\ie filter, exposure time, etc).\VER{TEST}{TLR:20}
    1253 \end{enumerate}
    1254 
    1255 \paragraph{Astrometry}
    1256 \begin{enumerate}
    1257 
    1258 \item The Phase 2 analysis shall match the detected objects with known
    1259   astrometric reference objects.\VER{TEST}{TLR:3}
    1260 
    1261 \item The astrometric reference object coordinates shall be adjusted
    1262   for proper motion.\VER{TEST}{TLR:3}
    1263 
    1264 \item The reference and detected object coordinates shall be fit to
    1265   determine astrometric parameters for the individual OTAs.\VER{TEST}{TLR:3}
    1266 
    1267 \item The OTA astrometric parameters shall include polynomials of the
    1268 coordinates up to 3rd order.\VER{TEST}{TLR:3}
    1269 
    1270 \item The fitted number of polynomial orders shall be a user-configured
    1271   parameter.\TASK
    1272 
    1273 \item The Cell astrometric parameters shall not be allowed to vary in
    1274   the fit.\VER{}{}
    1275 
    1276 \item The fit shall be robust, rejecting outlier matches (either stars
    1277   with poorly determined proper motion or spurious matches).\VER{TEST}{TLR:3}
     1204  than \tbr{10~arcsec}.\VER{ANALYSIS}{TLR:1}
     1205
     1206\item The Phase 2 analysis shall detect cosmic rays with flux $> 5\sigma$ by
     1207  morphology in single images with an efficiency of $> 95$\% for
     1208  images which are not undersampled.  \VER{TEST}{TLR:18}
    12781209
    12791210\item The resulting astrometric solution shall be consistent across the
    … …  
    12811212\end{enumerate}
    12821213
    1283 \paragraph{Postage Stamps}
    1284 \begin{enumerate}
    1285 
    1286 \item The Phase 2 analysis shall extract subrasters (`postage stamps')
    1287   surrounding a user-specified list of coordinates from the flattened
    1288   images.\VER{TEST}{TLR:12}
    1289 
    1290 \item The postage stamp images shall be saved in the IPP Image Server.\VER{TEST}{TLR:12}
    1291 \end{enumerate}
    1292  
     1214%% Phase 3
    12931215\subsubsection{Phase 3 : exposure analysis}
    1294 \begin{enumerate}
    1295 
    1296 \item The Phase 3 analysis shall use the objects detected in Phase 2,
    1297   matched with a user-specified reference photometry catalog, to
    1298   determine the image photometric zero point and zero-point variations
    1299   across the field.\VER{TEST}{??}
     1216
     1217The Phase 3 analysis uses the objects detected in Phase 2 and external
     1218reference catalogs to determine improved photometric and astrometric
     1219calibrations for the FPA as a whole, and to improve the measurement of
     1220the PSF and sky variations across the field.  The Phase 3 tasks and
     1221functions are as follows:
     1222
     1223\begin{itemize}
     1224
     1225\item Phase 3 uses the objects detected in Phase 2, matched with a
     1226  user-specified reference photometry catalog, to determine the image
     1227  photometric zero point and zero-point variations across the field.
    13001228
    13011229\item If zero-point variations are significant (\tbr{$> 0.01$ mag
    1302   peak-to-peak}), the zero-point variations shall be modeled with a
    1303   polynomial correction of order 3 or less.\VER{TEST}{TLR:1}
    1304 
    1305 \item The photometric nature of the FPA image shall be categorized on
    1306   the basis of the zero-point consistency, the transparency compared
    1307   with recent long-term measurements in the filter, and the external
    1308   indicators of photometricity.\VER{TEST}{TLR:2}
    1309 
    1310 \item The Phase 3 analysis shall use the objects detected in Phase 2,
    1311   matched with an appropriate astrometric reference catalog, to
    1312   improve the distortion model used for the image.\VER{TEST}{TLR:3}
    1313 
    1314 \item The resulting astrometric accuracy shall be consistent across
    1315 the field to 30 mas.\VER{TEST}{TLR:4}
    1316 
    1317 \item The resulting astrometric accuracy shall be limited by the
     1230  peak-to-peak}), the zero-point variations are modeled with a
     1231  polynomial correction of order 3 or less.
     1232
     1233\item The photometric nature of the FPA image is categorized on the
     1234  basis of the zero-point consistency, the transparency compared with
     1235  recent long-term measurements in the filter, and the external
     1236  indicators of photometricity.
     1237
     1238\item Phase 3 uses the objects detected in Phase 2, matched with an
     1239  appropriate astrometric reference catalog, to improve the distortion
     1240  model used for the image.  The resulting astrometric accuracy is
     1241  consistent across the field to 30 mas, and is limited by the
    13181242  astrometric reference catalog, (eg, 100 - 250 mas for
    1319   USNO-B1.0).\VER{TEST}{TLR:3}
    1320 
    1321 \item The Phase 3 analysis shall modify the background correction of
    1322 Phase 2 based on the full-field statistics to achieve an accuracy of 1\%
    1323 of the background.\VER{TEST}{TLR:5}
     1243  USNO-B1.0).
     1244
     1245\item The Phase 3 analysis modifies the background correction of Phase
     1246  2 based on the full-field statistics to achieve an accuracy of 1\%
     1247  of the background.
     1248
     1249\end{itemize}
     1250
     1251The Phase 3 requirements are:
     1252
     1253\begin{enumerate}
    13241254
    13251255\item The complete Phase~3 analysis shall be performed in $< 2$
    13261256seconds for up to 4 complete FPA images at one time. \VER{TEST}{TLR:17}
    13271257
    1328 \end{enumerate}
    1329 
     1258\item For images obtained under normal observing conditions, the
     1259  resulting astrometric solution shall have a residual scatter of $<
     1260  30$ milliarcseconds when calibrated with the AP Survey reference
     1261  catalog and $< 100$ milliarcseconds when calibrated with the USNO-B
     1262  catalog.\VER{ANALYSIS}{TLR:}
     1263
     1264\item For images obtained under normal observing conditions, the
     1265  resulting astrometric solution shall have a precision relative to
     1266  ICRS of better than 100 milliarcseconds.\VER{ANALYSIS}{TLR:}
     1267
     1268\item For images obtained under photometric conditions or minimal
     1269  cirrus conditions ($< 0.1$ mag total extinction), the resulting
     1270  photometric calibration shall have a relative accuracy of 5
     1271  millimagnitudes.\VER{ANALYSIS}{TLR:}
     1272
     1273\item For images obtained under photometric conditions or minimal
     1274  cirrus conditions ($< 0.1$ mag total extinction), the resulting
     1275  photometric calibration shall have an absolution photometric
     1276  accuracy of 10 millimagnitudes when calibrated relative to the AP
     1277  Survey reference catalog.\VER{ANALYSIS}{TLR:}
     1278
     1279\item For images obtained under photometric conditions or minimal
     1280  cirrus conditions ($< 0.1$ mag total extinction) and under the moon
     1281  conditions listed in Table~\ref{moonconditions}, the resulting sky
     1282  background subtraction shall leave behind peak-to-peak residuals $<
     1283  1$\% of the input sky flux.\VER{ANALYSIS}{TLR:}
     1284
     1285\end{enumerate}
     1286
     1287%% Phase 4
    13301288\subsubsection{Phase 4 : image combination}
    13311289
    13321290Phase 4 is the image combination stage, in which multiple images of
    13331291the same portion of the sky are merged and confronted with the static
    1334 sky image.  Requirements for the different steps of the process are
    1335 given below.
    1336 
    1337 \paragraph{Extract image pixels}
    1338 \begin{enumerate}
    1339 
    1340 \item The Phase 4 analysis shall determine the corresponding set of
    1341   image pixels for a given sky cell.\TASK
    1342 
    1343 \item The corresponding image pixels shall be extracted from the input
    1344   images, using the astrometric information for each OTA and Cell to
    1345   determine the exact overlaps.\TASK
    1346 
     1292sky image.  The Phase 4 tasks and functions are as follows:
     1293
     1294\begin{itemize}
     1295
     1296\item The Phase 4 analysis determines the corresponding set of image
     1297  pixels for a given sky cell.
     1298
     1299\item These pixels are extracted from the input images, using the
     1300  astrometric information for each OTA and Cell to determine the exact
     1301  overlaps.
     1302
     1303\item The Phase 4 analysis skips any sky cells with fewer than 5\% of
     1304  their pixels overlapping the input images.
     1305
     1306\item Pixels which have been extracted from the input images are
     1307  geometrically warped to match the corresponding pixels in the sky
     1308  image.  This transformation is based on the measured astrometric
     1309  solution for the input images relative to the reference catalog used
     1310  to generate the static sky image.  The warping may use a
     1311  locally-linear astrometric solution to speed the processing.
     1312 
     1313\item Phase 4 determines the appropriate photometry scaling factors
     1314  needed to combine the images photometrically.
     1315
     1316\item When multiple images are combined, the group of input pixels
     1317  which contribute to an output pixel are examined and pixels from the
     1318  group of images which are inconsistent with the ensemble (by an
     1319  amount defined by the user-configurable parameters) are identified
     1320  and flagged, though this outlier rejection shall be performed
     1321  optionally.
     1322
     1323\item The resulting collection of pixels is used to construct a single
     1324  output image, cleaned of the outliers.
     1325
     1326\item The cleaned, combined image is PSF matched with the static sky
     1327  image.
     1328
     1329\item The static sky image is subtracted from the stacked, cleaned
     1330  image, resulting in the difference image (P4$\Delta$ image)
     1331
     1332\item The Phase 4 analysis performs object detection on the difference
     1333  images.  All objects in the difference image above a user-configured
     1334  signficance threshold are detected, including both positive and
     1335  negative flux objects.  The detection threshold may optionally be a
     1336  function of the average background flux or the local noise
     1337  level.  The likely significance threshold is $\sim 3\sigma$.
     1338
     1339\item P4$\Delta$ objects have the following object parameters
     1340  measured:
     1341  \begin{itemize}
     1342  \item object centroid and position errors.
     1343  \item instrumental PSF magnitude and error.
     1344  \item local background level and error.
     1345  \item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$.
     1346  \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix.
     1347  \end{itemize}
     1348
     1349\item Minimal object classification is performed to distinguish
     1350  objects which are consistent with a single PSF, objects which are
     1351  inconsistent, and objects which are saturated.
     1352
     1353\item The pixels belonging to variable sources are masked in the
     1354  input image.
     1355
     1356\item A new, cleaned image is constructed from the masked input images
     1357  (P4$\Sigma$ image)
     1358
     1359\item Object detection is performed on the cleaned, summed image to a
     1360  user-configured significance threshold ($\sim 7\sigma$).  Only
     1361  positive flux object are considered.  The detection threshold may
     1362  optionally be a function of the average background flux or the local
     1363  noise level.
     1364
     1365\item P4$\Sigma$ objects have the following object parameters
     1366  measured:
     1367  \begin{itemize}
     1368  \item object centroid and position errors.
     1369  \item an extended object position ($x_g, y_g$).
     1370  \item instrumental PSF magnitude and error.
     1371  \item local background level and error.
     1372  \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their
     1373    covariance matrix.
     1374  \item the Petrosian radius, magnitude, axis ratio, and angle.
     1375  \item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$.
     1376  \end{itemize}
     1377
     1378\item Minimal object classification is performed to distinguish
     1379  objects which are consistent with a single PSF, objects which are
     1380  inconsistent, and objects which are saturated.
     1381
     1382\item Before the image is added to the static sky, it must pass Q/A
     1383  tests:
     1384  \begin{itemize}
     1385  \item the measured photometry scatter for the image must be less
     1386      than \tbr{1\%}.
     1387
     1388  \item the measured astrometry scatter for the image must be less
     1389  than \tbr{30 mas}.
     1390  \end{itemize}
     1391
     1392\item The final, cleaned input image is added to the static sky so
     1393  that an incrementally-deeper static sky image may be
     1394  made.
     1395\end{itemize}
     1396
     1397The Phase 4 requirements are:
     1398
     1399\begin{enumerate}
    13471400\item The Phase 4 analysis shall not miss any pixels in this match, and
    13481401  it shall read no more than 20\% more pixels than necessary from the
    13491402  input images.\VER{TEST}{TLR:17}
    13501403
    1351 \item The Phase 4 analysis shall skip any sky cells with fewer than 5\%
    1352   of their pixels overlapping the input images.\VER{TEST}{TLR:17}
    1353 \end{enumerate}
    1354 
    1355 \paragraph{Transform pixel coordinates}
    1356 \begin{enumerate}
    1357 
    1358 \item Pixels which have been extracted from the input images shall be
    1359   mapped to the corresponding pixels in the sky image.\TASK
    1360 
    1361 \item The transformation shall be based on the measured astrometric
    1362   solution for the input images relative to the reference catalog used
    1363   to generate the static sky image.\VER{TEST}{TLR:3}
    1364 
    1365 \item This warping shall use a locally-linear astrometric solution.\VER{TEST}{TLR:17}
    1366  
    1367 \item The output image shall maintain photometric consistency with the
    1368   input image to within 0.2\%.\VER{TEST}{TLR:1}
    1369 \end{enumerate}
    1370 
    1371 \paragraph{Flux matching}
    1372 
    1373 The Phase 4 analysis shall determine appropriate photometry scaling
    1374 factors needed to combine the images photometrically.\TASK
    1375 
    1376 \paragraph{Image outlier pixel rejection}
    1377 \begin{enumerate}
    1378 
    1379 \item When multiple images are combined, the group of input pixels
    1380   which contribute to an output pixel shall be examined and pixels from
    1381   the group of images which are inconsistent with the ensemble (by an
    1382   amount defined by the user-configurable parameters) shall be
    1383   identified and flagged.\VER{TEST}{TLR:18}
    1384 
    1385 \item This outlier rejection shall be performed optionally.\TASK
    1386 
    1387 \end{enumerate}
    1388 
    1389 \paragraph{Initial cleaned image}
    1390 
    1391 The resulting collection of pixels shall be used to construct a single
    1392 output image, cleaned of the outliers.\VER{TEST}{TLR:18}
    1393 
    1394 \paragraph{PSF matching}
    1395 
    1396 The cleaned, combined image shall be PSF matched with the static sky image.\VER{TEST}{TLR:15}
    1397 
    1398 \paragraph{Image Subtraction}
    1399 
    1400 The static sky image shall be subtracted from the stacked, cleaned
    1401 image.  \VER{TEST}{TLR:15}
    1402 
    1403 \paragraph{Find objects in the image}
    1404 \begin{enumerate}
    1405 
    1406 \item The Phase 4 analysis shall perform object detection on the
    1407   difference images.\VER{TEST}{TLR:15}
    1408 
    1409 \item All objects in the difference image shall be detected and the
    1410   pixels belonging to variable sources flagged in the input image.\VER{TEST}{TLR:15}
    1411 
    1412 \item The object detection shall detect all objects above a
    1413   user-configured threshold.\VER{TEST}{TLR:15}
    1414 
    1415 \item Both positive and negative objects shall be detected: the
    1416   specified threshold shall define the absolute value of the detection
    1417   thresholds.\VER{TEST}{TLR:15}
    1418 
    1419 \item The detection threshold shall optionally be a function of the
    1420   average background flux or the local noise level.\VER{TEST}{TLR:15}
    1421 
    1422 \item The object detection shall measure the following object parameters:
    1423   \begin{enumerate}
    1424   \item object centroid and position errors\VER{TEST}{TLR:15}
    1425   \item instrumental PSF magnitude and error\VER{TEST}{TLR:15}
    1426   \item local background level and error\VER{TEST}{TLR:15}
    1427   \item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$\VER{TEST}{TLR:15}
    1428   \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix\VER{TEST}{TLR:15}
    1429   \end{enumerate}
    1430 
    1431 \item Minimal object classification shall be performed to distinguish
    1432   objects which are consistent with a single PSF, objects which are
    1433   inconsistent, and objects which are saturated.\VER{TEST}{TLR:15, TLR:18}
    1434 
    1435 \item The resulting collection of detected objects shall be saved along
    1436   with the relevant image metadata (\ie filter, exposure time, etc).\VER{TEST}{TLR:22}
    1437 \end{enumerate}
    1438 
    1439 \paragraph{Cleaned Input Image}
    1440 \begin{enumerate}
    1441 
    1442 \item The pixels flagged as being from the difference image sources
    1443   shall be masked in the input images.\VER{TEST}{TLR:6, TLR:11}
    1444 
    1445 \item A new, cleaned image shall be constructed from the masked input
    1446   images.\VER{TEST}{TLR:6, TLR:11}
    1447 
    1448 \end{enumerate}
    1449 
    1450 \paragraph{Find objects in the image}
    1451 \begin{enumerate}
    1452 
    1453 \item The Phase 4 analysis shall perform object detection on the
    1454   cleaned, summed image.\VER{TEST}{TLR:13}
    1455 
    1456 \item The object detection shall detect all objects above a
    1457   user-configured threshold.\VER{TEST}{TLR:13}
    1458 
    1459 \item The threshold shall be a positive value; negative values shall
    1460   invoke an error.\VER{TEST}{TLR:13}
    1461 
    1462 \item The detection threshold optionally shall be a function of the
    1463   average background flux or the local noise level.\VER{TEST}{TLR:13}
    1464 
    1465 \item The object detection shall measure the following object parameters:
    1466   \begin{enumerate}
    1467   \item object centroid and position errors\VER{TEST}{TLR:13}
    1468   \item an extended object position ($x_g, y_g$)\VER{TEST}{TLR:13}
    1469   \item instrumental PSF magnitude and error\VER{TEST}{TLR:13}
    1470   \item local background level and error\VER{TEST}{TLR:13}
    1471   \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their
    1472     covariance matrix\VER{TEST}{TLR:13}
    1473   \item the Petrosian radius, magnitude, axis ratio, and angle\VER{TEST}{TLR:13}
    1474   \item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$.\VER{TEST}{TLR:13}
    1475   \end{enumerate}
    1476 
    1477 \item Minimal object classification shall be performed to distinguish
    1478   objects which are consistent with a single PSF, objects which are
    1479   inconsistent, and objects which are saturated.\VER{TEST}{TLR:13}
    1480 
    1481 \item The resulting collection of detected objects shall be saved along
    1482   with the relevant image metadata (\ie filter, exposure time, etc).\VER{TEST}{TLR:20}
    1483 \end{enumerate}
    1484 
    1485 \paragraph{Image Processing Q/A}
    1486 
    1487 Before the image is added to the static sky, it shall pass Q/A tests:
    1488 
    1489 \begin{enumerate}
    1490 \item the measured photometry scatter for the image shall be less than
    1491       \tbr{1\%}.\VER{TEST}{TLR:1}
    1492 
    1493 \item the measured astrometry scatter for the image shall be less than
    1494   \tbr{30 mas}.\VER{TEST}{TLR:3}
    1495 
    1496 \end{enumerate}
    1497 
    1498 \paragraph{Update static sky}
    1499 
    1500 The final, cleaned input image shall be added to the static sky so that
    1501 an incrementally-deeper static sky image may be made.\VER{TEST}{TLR:6, TLR:11}
    1502 
    1503 \paragraph{Timing}
    1504 The complete Phase~4 analysis shall be performed in $< 38$ seconds for
    1505 up to 4 complete FPA images at one time. \VER{TEST}{TLR:17}
    1506 
    1507 \subsubsection{Calibration Stages}
     1404\item The warped images shall maintain photometric consistency with
     1405  the input image to within 0.2\%.\VER{TEST}{TLR:1}
     1406
     1407\item The sky representation shall degrade the image quality by less
     1408  than 10 milliarcseconds added in quadrature to the input image
     1409  quality.\VER{TEST}{TLR:1}
     1410
     1411\item The complete Phase~4 analysis shall be performed in $< 38$
     1412  seconds for up to 4 complete FPA images at one
     1413  time. \VER{TEST}{TLR:17}
     1414
     1415\item completeness?
     1416
     1417\item contamination?
     1418
     1419\end{enumerate}
     1420
     1421\subsection{Calibration Stages}
    15081422\label{mkcal}
     1423
     1424\subsubsection{General Calibration Construction Requirements}
    15091425
    15101426The Calibration analysis stages construct the various types of
    … …  
    15211437\end{enumerate}
    15221438
    1523 Requirements for each of the individual calibration analysis stages
    1524 are discussed in detail below.
    1525 
    1526 \paragraph{bias images}
    1527 \begin{enumerate}
    1528 
    1529 \item The \code{bias} calibration stage shall construct a master bias
    1530   image from a collection of raw bias images.\TASK
    1531 
    1532 \item The \code{bias} calibration stage shall correct the input images
    1533   based on the overscan region.\TASK
    1534 
    1535 \item The \code{bias} calibration stage shall combine the input images
    1536   using the statistic specified by the user, selected from one of the
     1439\subsubsection{Bias Image Creation}
     1440
     1441The Bias calibration stage constructs a master bias image from a
     1442collection of raw bias images.  The tasks and functions include:
     1443
     1444\begin{itemize}
     1445
     1446\item The Bias calibration stage corrects the input images based on
     1447  the overscan region, determined from either the header or from
     1448  metadata.
     1449
     1450\item The Bias calibration stage combines the input images using the
     1451  statistic specified by the user, selected from one of the following:
     1452  sample mean, median, and mode, robust mean, median, and mode, and
     1453  the clipped mean and median.
     1454
     1455\item The Bias calibration stage construct residual images, in which
     1456  the master bias is applied to the input images.
     1457
     1458\item Outlier residual images, those for which the residual bias and
     1459  variance in the bias image are excessive, are excluded from the
     1460  input image stack and the bias image reconstructed.
     1461\end{itemize}
     1462
     1463\subsubsection{Dark Image Creation}
     1464
     1465The Dark calibration stage shall construct a master dark image from a
     1466  collection of raw dark images.  The tasks and functions include:
     1467
     1468\begin{itemize}
     1469
     1470\item The Dark calibration stage raises an error if the input images
     1471  have exposure times which deviate by more than
     1472  \tbr{2\%}.
     1473
     1474\item The Dark calibration stage corrects the input dark images for
     1475  the bias.
     1476
     1477\item The Dark calibration stage combines the input images using the
     1478  statistic specified by the user, selected from one of the following:
     1479  sample mean, median, and mode, robust mean, median, and mode, and
     1480  the clipped mean and median.
     1481
     1482\item The Dark calibration stage constructs residual images, in which
     1483  the master dark is applied to the input images.
     1484
     1485\item Outlier residual images, those for which the residual level and
     1486  variance are excessive, are excluded from the input image stack and
     1487  the dark image reconstructed.
     1488\end{itemize}
     1489
     1490\subsubsection{Flat-field Image Creation}
     1491
     1492The Flat-field calibration stage constructs a master flat-field image
     1493from a collection of raw flat-field images.  The tasks and functions
     1494include:
     1495
     1496\begin{itemize}
     1497
     1498\item The Flat-field calibration stage accepts a group of images from
     1499  one of the following flat-field sources: dome, twilight,
     1500  night-sky.
     1501
     1502\item The flat-field calibration stage raises an error if the
     1503  input images in a single stack used more than one of the above
     1504  flat-field sources or multiple filters.
     1505
     1506\item The Flat-field calibration stage corrects the input flat-field
     1507  images for the bias and dark.
     1508
     1509\item The Flat-field calibration stage combines the input images using
     1510  the statistic specified by the user, selected from one of the
    15371511  following: sample mean, median, and mode, robust mean, median, and
    1538   mode, and the clipped mean and median.\TASK
    1539 
    1540 \item The \code{bias} calibration stage shall construct residual
    1541   images, in which the master bias is applied to the input images.\TASK
    1542 
    1543 \item Outlier residual images, those for which the residual bias and
    1544   variance in the bias image are excessive ($> 1DN$), shall be excluded
    1545   from the input image stack the the bias image reconstructed.\VER{TEST}{TLR:1}
    1546 \end{enumerate}
    1547 
    1548 \paragraph{dark images}
    1549 \begin{enumerate}
    1550 
    1551 \item The \code{dark} calibration stage shall construct a master dark
    1552   image from a collection of raw dark images.\TASK
    1553 
    1554 \item The \code{dark} calibration stage shall raise an error if the
    1555   input images have exposure time which deviate by more than
    1556   \tbr{2\%}.\VER{TEST}{TLR:1}
    1557 
    1558 \item The \code{dark} calibration stage shall correct the input dark
    1559   images for the bias.\TASK
    1560 
    1561 \item The \code{dark} calibration stage shall combine the input images
    1562   using the statistic specified by the user, selected from one of the
    1563   following: sample mean, median, and mode, robust mean, median, and
    1564   mode, and the clipped mean and median.\VER{TEST}{TLR:1}
    1565 
    1566 \item The \code{dark} calibration stage shall construct residual
    1567   images, in which the master dark is applied to the input images.\TASK
    1568 
    1569 \item Outlier residual images, those for which the residual level and
    1570   variance are excessive ($> 1DN$), shall be excluded from the input
    1571   image stack the the dark image reconstructed.\VER{TEST}{TLR:1}
    1572 \end{enumerate}
    1573 
    1574 \paragraph{flat-field images}
    1575 \begin{enumerate}
    1576 
    1577 \item The \code{flat-field} calibration stage shall construct a master
    1578   flat-field image from a collection of raw flat-field images.\VER{TEST}{TLR:1}
    1579 
    1580 \item The \code{flat-field} calibration stage shall accept a group of
    1581   images from one of the following flat-field sources: dome, twilight,
    1582   night-sky.\VER{TEST}{TLR:1}
    1583 
    1584 \item The \code{flat-field} calibration stage shall raise an error if
    1585   the input images in a single stack used more than one of the above
    1586   flat-field sources or multiple filters.\TASK
    1587 
    1588 \item The \code{flat-field} calibration stage shall correct the input
    1589   flat-field images for the bias and dark.\TASK
    1590 
    1591 \item The \code{flat-field} calibration stage shall combine the input
    1592   images using the statistic specified by the user, selected from one
    1593   of the following: sample mean, median, and mode, robust mean,
    1594   median, and mode, and the clipped mean and median.\VER{TEST}{TLR:1}
    1595 
    1596 \item The \code{flat-field} calibration stage shall construct residual
    1597   images, in which the master flat-field is applied to the input
    1598   images.\TASK
     1512  mode, and the clipped mean and median.
     1513
     1514\item The Flat-field calibration stage constructs residual images, in
     1515  which the master flat-field is applied to the input images.
    15991516
    16001517\item Outlier residual images, those for which the residual level and
    16011518  variance are excessive ($> 0.1$\%, or 1.02 times the Poisson limit
    1602   of the flat-field image), shall be excluded from the input image
    1603   stack the the flat-field image reconstructed.\VER{TEST}{TLR:1}
    1604 \end{enumerate}
    1605 
    1606 \paragraph{mask images}
    1607 \begin{enumerate}
    1608 
    1609 \item The \code{mask} calibration stage shall construct a bad-pixel
    1610   mask from a stack of raw flat-field images and a master flat-field
    1611   image.\VER{TEST}{TLR:1}
    1612 
    1613 \item The \code{mask} calibration stage shall mask the pixels which are
     1519  of the flat-field image), are excluded from the input image stack
     1520  and the flat-field image reconstructed.
     1521\end{itemize}
     1522
     1523\subsubsection{Mask Image Creation}
     1524
     1525The Mask calibration stage constructs a bad-pixel mask from a stack of
     1526raw flat-field images and a master flat-field image.  The tasks and
     1527functions include:
     1528
     1529\begin{itemize}
     1530
     1531\item The Mask calibration stage masks the pixels which are
    16141532  inconsistent in the input flats by more than \tbr{1\%}, given
    1615   sufficient signal-to-noise in the input flats.\VER{TEST}{TLR:1}
    1616 
    1617 \item The \code{mask} calibration stage shall mask the pixels which are
     1533  sufficient signal-to-noise in the input flats.
     1534
     1535\item The Mask calibration stage mask the pixels which are
    16181536  consistently low or high in the input flats by more than a factor of
    1619   \tbr{3} beyond the typical pixel.\VER{TEST}{TLR:1}
    1620 
    1621 \item The \code{mask} calibration stage shall mask the pixels
    1622   identified in a table of bad pixels generated externally to the
    1623   calibration stage.\TASK
    1624 
    1625 \item The \code{mask} calibration stage shall use multiple bit values
    1626   to identify the different types of masked pixels.\TASK
    1627 
    1628 \item The \code{mask} calibration stage shall raise an error if the
    1629   input images generate too many bad pixels in the mask.\TASK
    1630 \end{enumerate}
    1631 
    1632 \paragraph{fringe frames}
    1633 \begin{enumerate}
    1634 
    1635 \item The \code{fringe} calibration stage shall construct a master fringe
     1537  \tbr{3} beyond the typical pixel.
     1538
     1539\item The Mask calibration stage masks the pixels identified in a
     1540  table of bad pixels generated externally to the calibration stage.
     1541
     1542\item The Mask calibration stage uses multiple bit values to identify
     1543  the different types of masked pixels.
     1544
     1545\item The Mask calibration stage raises an error if the input images
     1546  generate too many bad pixels in the mask.
     1547\end{itemize}
     1548
     1549\subsubsection{Fringe-frame Creation}
     1550
     1551The Fringe-frame Creation calibration stage constructs a master fringe
    16361552frame from a stack of raw night-time sky images or from a stack of
    1637 dome fringe frames.\VER{TEST}{TLR:1, TLR:5}
    1638 
    1639 \item The \code{fringe} calibration stage shall raise an error if the input
    1640 stack consists is images generated with more than one type of fringe
    1641 source or with multiple filters.\TASK
    1642 
    1643 \item The \code{fringe} calibration stage shall flatten the input images
    1644 to remove the pixel-to-pixel sensitivity variations of the detector.\VER{TEST}{TLR:1}
    1645 
    1646 \item The \code{fringe} calibration stage shall measure the fringe amplitude
    1647 on the input fringe images.\TASK
    1648 
    1649 \item The \code{fringe} calibration stage shall scale the input fringe images
    1650 based on the fringe amplitude.\TASK
    1651 
    1652 \item The \code{fringe} calibration stage shall combine the scaled input
    1653 images using the statistic specified by the user, selected from one of
    1654 the following: sample mean, median, and mode, robust mean, median, and
    1655 mode, and the clipped mean and median.\VER{TEST}{TLR:5}
    1656 
    1657 \item The \code{fringe} calibration stage shall construct residual images, in
    1658 which the master fringe image is applied to the input images, along
    1659 with all necessary preceding calibration images.\TASK
    1660 
    1661 \item The \code{fringe} calibration stage shall measure the residual fringe
    1662 amplitude on the residual images.\TASK
    1663 \end{enumerate}
    1664 
    1665 \paragraph{low-spatial-frequency sky models}
    1666 
    1667 The \code{sky model} calibration stage shall construct a sky model
    1668 image from a stack of raw night-time sky images.\VER{TEST}{TLR:5}
    1669 
    1670 \paragraph{Flat-field correction frame}
    1671 \begin{enumerate}
    1672 
    1673 \item The \code{flat-field correction} calibration stage shall construct a
    1674 flat-field correction image from dithered observations of a stellar
    1675 field.\VER{TEST}{TLR:1}
    1676 
    1677 \item The \code{flat-field correction} calibration stage shall construct a
    1678 flat-field correction image for each filter and camera independently.\TASK
    1679 
    1680 \item The \code{flat-field correction} calibration stage shall construct a
    1681 correction image which makes the photometry of multiple observations
    1682 of the same stellar source consistent at different locations in the
    1683 focal plane.\VER{TEST}{TLR:1}
    1684 
    1685 \item The \code{flat-field correction} calibration stage shall construct
    1686 corrected flat-field images using the measured correction.\VER{TEST}{TLR:1}
    1687 
    1688 \item The \code{flat-field correction} calibration stage shall determine the
    1689 consistency of the corrected flat-field images using the dithered
    1690 stellar field observations flattened with the corrected flat-field
    1691 image.\TASK
    1692 \end{enumerate}
    1693 
    1694 \paragraph{Non-linearity correction frames}
    1695 \begin{enumerate}
    1696 
    1697 \item The \code{non-linear correction} calibration stage shall construct a
    1698 non-linear correction from a collection of images of a constant source
    1699 with varying exposure times.\VER{TEST}{TLR:1}
    1700 
    1701 \item The \code{non-linear correction} calibration stage shall construct a
    1702 non-linear correction which linearizes the detector fluxes $<0.5\%$.\VER{TEST}{TLR:1}
    1703 
    1704 \item The \code{non-linear correction} calibration stage shall determine the
    1705 saturation regime, in which the non-linear correction is no longer
    1706 consistent to $<0.5\%$.\VER{TEST}{TLR:1}
    1707 \end{enumerate}
    1708 
    1709 \paragraph{Telescope Astrometry Parameters}
    1710 
    1711 \begin{enumerate}
    1712 \item The IPP Calibration system shall construct static models of the
     1553dome fringe frames.  The tasks and functions include:
     1554
     1555\begin{itemize}
     1556
     1557\item The Fringe-frame Creation calibration stage raises an error if
     1558  the input stack consists is images generated with more than one type
     1559  of fringe source or with multiple filters.
     1560
     1561\item The Fringe-frame Creation calibration stage flattens the input
     1562  images to remove the pixel-to-pixel sensitivity variations of the
     1563  detector.
     1564
     1565\item The Fringe-frame Creation calibration stage measures the fringe
     1566  amplitude on the input fringe images.
     1567
     1568\item The Fringe-frame Creation calibration stage scales the input
     1569  fringe images based on the fringe amplitude.
     1570
     1571\item The Fringe-frame Creation calibration stage combines the scaled
     1572  input images using the statistic specified by the user, selected
     1573  from one of the following: sample mean, median, and mode, robust
     1574  mean, median, and mode, and the clipped mean and median.
     1575
     1576\item The Fringe-frame Creation calibration stage constructs residual
     1577  images, in which the master fringe image is applied to the input
     1578  images, along with all necessary preceding calibration images.
     1579
     1580\item The Fringe-frame Creation calibration stage measures the
     1581  residual fringe amplitude on the residual images.
     1582\end{itemize}
     1583
     1584\subsubsection{Low-spatial-frequency Sky Models}
     1585
     1586The Sky Model calibration stage constructs a sky model image set from
     1587a stack of raw night-time sky images.
     1588
     1589\subsubsection{Flat-field correction Frame Creation}
     1590
     1591The Flat-field correction calibration stage constructs a flat-field
     1592correction image from dithered observations of a stellar field.  The
     1593tasks and functions include:
     1594
     1595\begin{itemize}
     1596
     1597\item The Flat-field correction calibration stage constructs a
     1598  flat-field correction image from dithered observations of a stellar
     1599  field.
     1600
     1601\item The Flat-field correction calibration stage constructs a
     1602  flat-field correction image for each filter and camera
     1603  independently.
     1604
     1605\item The Flat-field correction calibration stage constructs a
     1606  correction image which makes the photometry of multiple observations
     1607  of the same stellar source consistent at different locations in the
     1608  focal plane.
     1609
     1610\item The Flat-field correction calibration stage constructs corrected
     1611  flat-field images using the measured correction.
     1612
     1613\item The Flat-field correction calibration stage determines the
     1614  consistency of the corrected flat-field images using the dithered
     1615  stellar field observations flattened with the corrected flat-field
     1616  image.
     1617\end{itemize}
     1618
     1619\subsubsection{Non-linearity correction}
     1620
     1621The Non-linear correction calibration stage constructs a correction
     1622model for low-level non-linearity effects in the detector.  The tasks
     1623and functions include:
     1624
     1625\begin{itemize}
     1626
     1627\item The Non-linear correction calibration stage constructs a
     1628  non-linear correction from a collection of images of a constant
     1629  source with varying exposure times.
     1630
     1631\item The Non-linear correction calibration stage construct a
     1632  non-linear correction which linearizes the detector fluxes
     1633  $<0.5\%$.
     1634
     1635\item The Non-linear correction calibration stage determines the
     1636  saturation regime, in which the non-linear correction is no longer
     1637  consistent to $<0.5\%$.
     1638\end{itemize}
     1639
     1640\subsubsection{Telescope Astrometry Parameters}
     1641
     1642\begin{itemize}
     1643\item The IPP Calibration system constructs static models of the
    17131644  telescope astrometry parameters (e.g., distortion, detector warps)
    1714   once per week.\VER{INSPECT}{TLR:4}
    1715 
    1716 \item The IPP Calibration system shall construct static models of the
     1645  once per week.
     1646
     1647\item The IPP Calibration system constructs static models of the
    17171648  telescope astrometry parameters (e.g., distortion, detector warps)
    17181649  with an accuracy to produce astrometry consistent to 30
    1719   milliarcsec.\VER{TEST}{TLR:4}
    1720 
    1721 \item The IPP Calibration system shall monitor changes in the
    1722   telescope astrometry parameters and issue a warning if the
    1723   parameters change by more than \tbr{2\%}.\VER{INSPECT}{TLR:4}
    1724 \end{enumerate}
    1725 
    1726 \paragraph{Zero-Point Monitoring}
    1727 
    1728 The IPP Calibration system shall determine telescope filter and camera
    1729 zero-points on a \tbd{timescale} with an accuracy sufficient to
     1650  milliarcsec.
     1651
     1652\item The IPP Calibration system monitors changes in the telescope
     1653  astrometry parameters and issue a warning if the parameters change
     1654  by more than \tbr{2\%}.
     1655\end{itemize}
     1656
     1657\subsubsection{Zero-Point Monitoring}
     1658
     1659The IPP Calibration system determines telescope filter and camera
     1660zero-points on a nightly basis with an accuracy sufficient to
    17301661determine photometry in the native filter systems to 5 millimags.
    1731 
    1732 \subsubsection{Reference Catalog Creation}
    1733 
    1734 For PS-1, one of the primary goals is the creation of photometric and
    1735 astrometric reference catalogs for the general community and for the
    1736 future Pan-STARRS calibration.  The generation of these catalogs is
    1737 inherently a research project, and will require human control and
    1738 intervention.  The IPP shall provide the data access, manipulation and
    1739 visualization tools needed to construct these reference catalogs and
    1740 to assess their quality.  In this section, we list the requirements of
    1741 the tools needed for this effort.
    1742 
    1743 \paragraph{Astrometry Reference Creation}
    1744 
    1745 \begin{table}
    1746 \begin{center}
    1747 \caption{Astrometric Reference Catalogs\label{AstroRefs}}
    1748 \begin{tabular}{lrrrrl}
    1749 \hline
    1750 \hline
    1751 Name       & scatter limit   & proper  & depth   & Nstars     & filters \\
    1752            & (milli-arcsec)  & motion? &(mag)    & (millions) &         \\
    1753 \hline
    1754 Hipparcos  &   1             & 2       &  7.3    &    0.1     & V       \\
    1755 Tycho2     &  10             & 1       & 11.5    &    2.5     & B,V     \\
    1756 UCAC-2     &  20             & 1       & 16.0    &   48.0     & R       \\
    1757 USNO-A2.0  & 250             & N/A     & 19.0?   &  526.2     & B,R     \\
    1758 USNO-B1.0  & 200             & 20?     & 21.0    & 1042.6     & B,R     \\
    1759 2MASS      &  70             & N/A     & 15.0?   &  470.0     & J,H,K   \\
    1760 \hline
    1761 \end{tabular}
    1762 \end{center}
    1763 \end{table}
    1764 
    1765 The IPP will generate an astrometric reference on the basis of the
    1766 observations obtained by the AP survey.  The IPP shall provide the
    1767 analysis tools needed to generate the master astrometric reference
    1768 catalog.  Much of the required functionality is covered by the AP
    1769 Database.  The specific requirements for the Astrometric Reference
    1770 creation are listed below:
    1771 
    1772 \begin{enumerate}
    1773 \item The IPP Reference Creation System shall produce an astrometric
    1774   reference catalog from the extracted objects within 6 months of the
    1775   end of the AP Survey.\VER{TEST}{TLR:3, TLR:4}
    1776 
    1777 \item The IPP Reference Creation System shall produce an astrometric
    1778   reference catalog with an absolute accuracy of \tbr{100 mas} and a
    1779   local relative accuracy of \tbr{30 mas} for bright objects.\VER{TEST}{TLR:3}
    1780 
    1781 \item The IPP Reference Creation System shall produce an astrometric
    1782   reference catalog with proper motions measurements for
    1783   non-solar-system objects with an accuracy of \tbr{20 mas / year} for
    1784   unsaturated, bright stars.\VER{TEST}{TLR:3}
    1785 
    1786 \item The Astrometry Reference creation tools shall return the match between
    1787 stars observed with Pan-STARRS and any of several astrometric
    1788 reference catalogs listed in Table~\ref{AstroRefs}.\TASK
    1789 
    1790 \item The tools shall convert the reference catalog object coordinates to all
    1791 of the coordinate frames of relevance in the telescope and camera
    1792 system:
    1793 \begin{enumerate}
    1794 \item Catalog to mean positions\VER{TEST}{TLR:3}
    1795 \item Mean to apparent positions\VER{TEST}{TLR:3}
    1796 \item Apparent positions + pointing to tangent plane coordinates\VER{TEST}{TLR:3}
    1797 \item Apparent positions + pointing to focal plane coordinates\VER{TEST}{TLR:3}
    1798 \item focal plane to specific detector (OTA)\VER{TEST}{TLR:3}
    1799 \item specific detector to detector cell\VER{TEST}{TLR:3}
    1800 \end{enumerate}
    1801 
    1802 \item The tools shall provide the necessary calibration data: the telescope
    1803 and camera optical distortion models and the variation of the image
    1804 PSF across the camera field, as a function of color.\TASK
    1805 
    1806 \item The tools shall fit the observed stellar coordinates to the astrometric
    1807 reference catalog coordinates to determine improved astrometric
    1808 solutions for both the stars and the detectors.  \TASK
    1809 
    1810 \item The tools shall determine improved telescope optical distortion models
    1811 based on the astrometric solutions. \VER{TEST}{TLR:3}
    1812 
    1813 \item The tools shall optionally determine the fit coefficients as a function
    1814 of position along, or with combinations of magnitude or color.  \VER{TEST}{TLR:3}
    1815 
    1816 \item The fitting method shall include robust outlier rejection.  \VER{TEST}{TLR:3}
    1817 
    1818 \item The tools shall identify objects which are detected in the catalog, but
    1819 not the science image or vice-versa.\TASK
    1820 
    1821 \item The tools shall determine average centroiding errors for each object.\TASK
    1822 
    1823 \item The tools shall plot the fit residuals against a wide variety of
    1824 parameters: the object positions, magnitudes, colors, etc.\TASK
    1825 
    1826 \item The tools shall allow the fit to exclude subsets of objects from the
    1827 fits on the basis of these parameters.\TASK
    1828 
    1829 \item The tools shall provide coordinates of the guide stars in the
    1830 same frame of reference as the normal image data to within 30
    1831 mas.\VER{TEST}{TLR:3}
    1832 
    1833 \item The tools shall perform the various fitting steps for the guide stars
    1834 rather than for the normal image data.\TASK
    1835 \end{enumerate}
    1836 
    1837 \paragraph{Photometry Reference Creation}
    1838 
    1839 \begin{table}
    1840 \begin{center}
    1841 \caption{Photometric Reference Catalogs\label{PhotoRefs}}
    1842 \begin{tabular}{lrrr}
    1843 \hline
    1844 \hline
    1845 Name       & scatter & depth & filters \\
    1846            & mmag    & mag   &         \\
    1847 \hline
    1848 SDSS       & 15?     & 16?   & {\em u,g,r,i,z} \\
    1849 CFHT-LS    & 10?     & 18    & {\em u,g,r,i,z} \\
    1850 Landolt    & 10-20   & 15    & {\em U,B,V,R,I} \\
    1851 \hline
    1852 \end{tabular}
    1853 \end{center}
    1854 \end{table}
    1855 
    1856 The IPP will generate a photometric reference catalog on the basis of
    1857 the observations obtained by the AP survey.  The IPP shall provide the
    1858 analysis tools needed to generate a master photometric reference
    1859 catalog.  Much of the required functionality is covered by the AP
    1860 Database.  The specific requirements for the Photometric Reference
    1861 creation are listed below:
    1862 
    1863 \begin{enumerate}
    1864 \item The IPP Reference Creation System shall produce a photometric
    1865   reference catalog from the extracted point-source objects within 6
    1866   months of the end of the AP Survey.\VER{TEST}{TLR:1}
    1867 
    1868 \item The IPP Reference Creation System shall produce a photometric
    1869   reference catalog with a consistency across the sky of \tbr{5
    1870   millimag}.\VER{TEST}{TLR:1}
    1871 
    1872 \item The IPP Reference Creation System shall produce a photometric
    1873   reference catalog with an absolute calibration to the external
    1874   system (defined by \tbr{SDSS} and the CFHT Legacy Survey Standards)
    1875   with an accuracy of \tbr{10 millimag} (for bright objects).\VER{TEST}{TLR:1}
    1876 
    1877 \item The Photometry Reference creation tools shall return the match between
    1878 stars observed with Pan-STARRS and any of several photometric
    1879 reference catalogs listed in Table~\ref{PhotoRefs}.\TASK
    1880 
    1881 \item The tools shall convert between different photometric systems, including:
    1882 \begin{enumerate}
    1883 \item instrumental to nominal detector magnitude\VER{TEST}{TLR:1}
    1884 \item nominal detector magnitude to average filter system\VER{TEST}{TLR:1}
    1885 \item average filter system to reference photometry system\VER{TEST}{TLR:1}
    1886 \end{enumerate}
    1887 
    1888 \item These transformations shall account for color and airmass terms.  \VER{TEST}{TLR:1}
    1889 
    1890 \item The tools shall measure and apply relative photometry corrections
    1891 between images.\VER{TEST}{TLR:1}
    1892 
    1893 \item The tools shall determine photometric transformation fit coefficients
    1894 as a function of airmass, magnitude, color and detector coordinates,
    1895 or with combinations of the above.\TASK
    1896 
    1897 \item The fitting method shall include robust outlier rejection.\VER{TEST}{TLR:1}
    1898 
    1899 \item The tools shall reject specific objects from the fit on the basis of
    1900 object locations, instrumental magnitudes, observed and reference
    1901 errors, and in particular time of the observations. \TASK
    1902 
    1903 \item The tools shall plot the fit residuals against a wide variety of
    1904 parameters, including the object positions, magnitudes, colors, etc.\TASK
    1905 
    1906 \item The tools shall provide photometry from the guide stars in the same
    1907 system as observations of stars from the normal imaging data.\VER{TEST}{TLR:1}
    1908 
    1909 \item The tools shall perform the above fitting steps for the guide stars
    1910 rather than for the normal image data.\TASK
    1911 \end{enumerate}
    19121662
    19131663\subsection{Modules}
    … …  
    19681718\subsubsection{External Catalogs}
    19691719
    1970 The IPP AP Database shall be able to interact with the following
    1971 externally provided reference catalogs listed in Table~\ref{AstroRefs}
    1972 and Table~\ref{PhotoRefs}.\VER{TEST}{TLR:1, TLR:3}
    1973 
    1974 \subsubsection{Analysis Reference Data}
    1975 
    1976 The IPP shall store reference data describing the relevant Pan-STARRS
    1977 and IPP components, including the telescope, camera, detectors,
    1978 filters, clustered computers, and IPP software parameters.
     1720The IPP AP Database shall be able to interact with the externally
     1721provided reference catalogs listed in Table~\ref{AstroRefs} and
     1722Table~\ref{PhotoRefs}.\VER{TEST}{TLR:1, TLR:3}
    19791723
    19801724\subsubsection{Static Sky Pixel Size}
    … …  
    21371881images obtained at a cadence of 1 image per 40 seconds.\VER{TEST}{TLR:17}
    21381882
    2139 \item The IPP shall perform the Phase 2 analysis within an average
    2140 time of 40 seconds per single Gigapixel camera image.  The Phase 2
    2141 analysis has been measured to require 3200 GHz-sec on a Pentium-4
    2142 machine.\VER{TEST}{TLR:17}
    2143 
    2144 \item The IPP shall perform the Phase 4 analysis on a set of 4 input
    2145 frames within an average time of 180 seconds.  The Phase 4 analysis
    2146 has been measured to require a total of 7800 GHz-sec on a Pentium-4
    2147 machine for a major frame of 4 input Gigapixel camera
     1883\item The IPP shall perform the Phase 1 and Phase 2 analyses within an
     1884average time of 40 seconds per single Gigapixel camera image.  The
     1885Phase 2 analysis has been measured to require 3200 GHz-sec on a
     1886Pentium-4 machine.\VER{TEST}{TLR:17}
     1887
     1888\item The IPP shall perform the Phase 3 and Phase 4 analyses on a set
     1889of 4 input frames within an average time of 180 seconds.  The Phase 4
     1890analysis has been measured to require a total of 7800 GHz-sec on a
     1891Pentium-4 machine for a major frame of 4 input Gigapixel camera
    21481892images.\VER{TEST}{TLR:17}
    21491893\end{enumerate}
    … …  
    21981942assumptions of bandwidth and CPU speeds to estimate the number of
    21991943nodes required for the IPP.  Each CPU is matched with one network
    2200 adapter and one disk array.  :
     1944adapter and one disk array. 
    22011945\begin{enumerate}
    22021946\item The IPP requires at least 40 Phase 2 Nodes (OTA Nodes)\VER{TEST}{TLR:17}
    … …  
    22061950\item The IPP requires at least 10 AP DB Nodes\VER{TEST}{TLR:17}
    22071951\end{enumerate}
    2208 
    2209 \subsubsection{Availability}
    2210 
    2211 The IPP Image Server nodes shall not be offline for more than 12 hours
    2212   consecutively or 36 hours per year.\VER{ANALYSIS}{TLR:17}
    22131952
    22141953%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Note: See TracChangeset for help on using the changeset viewer.