IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 2544


Ignore:
Timestamp:
Nov 30, 2004, 1:16:03 PM (22 years ago)
Author:
eugene
Message:

mods from PDR prep

Location:
trunk/doc/design
Files:
2 edited

Legend:

Unmodified
Added
Removed

  • trunk/doc/design/ippSDRS.tex

    r2241 r2544  
    1 %%% $Id: ippSDRS.tex,v 1.14 2004-10-29 22:00:08 eugene Exp $
     1%%% $Id: ippSDRS.tex,v 1.15 2004-11-30 23:16:03 eugene Exp $
    22\documentclass[panstarrs]{panstarrs}
    33
    44% basic document variables
    5 \title{Pan-STARRS Image Processing Pipeline}
    6 \subtitle{Supplementary Design Requirements Specification}
    7 \shorttitle{IPP SDRS}
     5\title{Pan-STARRS PS-1 Image Processing Pipeline}
     6\subtitle{System/Subsystem Design Description}
     7\shorttitle{IPP SSDD}
    88\author{Eugene A. Magnier, Paul A. Price, Josh Hoblitt}
    99\audience{Pan-STARRS PMO}
    … …  
    1111\project{Pan-STARRS Image Processing Pipeline}
    1212\organization{Institute for Astronomy}
    13 \version{DR}
     13\version{00}
    1414\docnumber{PSDC-430-011}
    1515
    … …  
    2727DR.03     & 2004.03.25 & Section reorganization \\ \hline
    2828DR.04     & 2004.04.13 & Most sections fleshed out \\ \hline
    29 DR.05     & 2004.04.29 & Reorganisation for consistency \\ \hline
     29DR.05     & 2004.04.29 & Reorganization for consistency \\ \hline
    3030DR.06     & 2004.10.21 & Major revision in prep of PDR \\ \hline
    3131\RevisionsEnd
    3232
     33\inserttbd
     34\inserttbr
     35\pagebreak
     36
     37\tableofcontents
     38\pagebreak
     39
    3340\listoffigures
    34 
    35 \pagebreak
    36 
    37 \tableofcontents
    3841\pagebreak
    3942\pagenumbering{arabic}
    … …  
    8992\subsection{Document Overview}
    9093
    91 The Pan-STARRS IPP Software Requirements Specification contains the
    92 complete system requirements of the Pan-STARRS PS-1 IPP in order to
    93 achieve the top-level performance and operational requirements
    94 specified by the SCD.  The requirements flow begun in the SGS and
    95 continued in the SCD is further developed in this SRS to provide
    96 additional derived system and subsystem requirements.
     94The Pan-STARRS IPP System/Subsystem Design Description (SSDD) contains
     95the complete design description of the Pan-STARRS PS-1 IPP in order to
     96achieve the requirements specified by the Pan-STARRS PS-1 IPP Software
     97Requirements Specification (SRS; PSDC-430-005).  The requirements flow
     98begun in the SGS and SCD and continued in the SRS is used to guide the
     99design presented here.
    97100
    98101\subsection{Requirements Definitions}
    … …  
    103106that series is implied. 
    104107
    105 Open issues (TBDs) in this document are marked \tbd{in bold red}.
    106 
    107 Quantities which should be reviewed (TBRs) are marked \tbr{in bold
    108 blue}.
     108Open issues (TBDs) in this document are marked {\bf \color{red} in
     109bold red}.
     110
     111Quantities which should be reviewed (TBRs) are marked {\bf
     112\color{blue} in bold blue}.
    109113
    110114\subsubsection{``Shall''}  When used in this specification, the word
    … …  
    123127
    124128\DocumentsInternalSection
    125 PSDC-130-001  &   PS-1 Design Reference Mission \\ \hline
     129PSDC-230-001  &   PS-1 Design Reference Mission \\ \hline
     130PSDC-230-002  &   PS-1 System Concept Definition \\ \hline
     131PSDC-400-006  &   The Pan-STARRS IPP Computational Challenge \\ \hline
    126132PSDC-430-004  &   Pan-STARRS IPP C Code Conventions \\ \hline
    127 PSDC-430-006  &   Pan-STARRS IPP ADD \\ \hline
    128 PSDC-430-007  &   Pan-STARRS IPP PSLib SDR \\ \hline
     133PSDC-430-005  &   Pan-STARRS IPP PS-1 Software Requirements Specification \\ \hline
     134PSDC-430-006  &   Pan-STARRS IPP Algorithm Design Document \\ \hline
     135PSDC-430-007  &   Pan-STARRS IPP PSLib Supplementary Design Requirements Specification \\ \hline
     136PSDC-430-010  &   Pan-STARRS IPP Perl Code Conventions \\ \hline
     137PSDC-430-012  &   Pan-STARRS IPP Modules Supplementary Design Requirements Specification \\ \hline
     138PSDC-430-014  &   Pan-STARRS IPP PS-1 Cluster Support \\ \hline
    129139\DocumentsExternalSection
    130140Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\
    … …  
    140150Pan-STARRS.  It also is responsible for combining all of the science
    141151images in a given filter into a single representation of the
    142 non-variable component of the night sky called the ``Static Sky''.  To
    143 achieve these goals, the IPP also performs other analysis functions to
    144 generate the calibrations needed in the science image processing and
    145 to occasionally use the derived data to generate improved astrometric
    146 and photometric reference catalogs.  It also provides the
     152non-variable component of the night sky defined as the ``Static Sky''.
     153To achieve these goals, the IPP also performs other analysis functions
     154to generate the calibrations needed in the science image processing
     155and to occasionally use the derived data to generate improved
     156astrometric and photometric reference catalogs.  It also provides the
    147157infrastructure needed to store the incoming data and the resulting
    148158data products.
    … …  
    166176transient objects.  3) the Published Science Products Subsystem
    167177(PSPS), which will receive all data products of interest to the
    168 outside world, and will act as the long-term archive and publishing
    169 clearinghouse.
     178community external to the Pan-STARRS data processing systems, and will
     179act as the long-term archive and publishing clearinghouse.
    170180
    171181The IPP receives data from two Pan-STARRS subsystems: the Camera, from
    172182which it receives the large volume of image data, and OTIS
    173 (Observatory, Telesope and Infrastructure Subsystem), from which it
     183(Observatory, Telescope and Infrastructure Subsystem), from which it
    174184receives metadata describing the images and the environmental
    175185conditions.  The primary IPP hardware system on which the software
    176 operates will not be located at the summit.  Instead, because of
    177 thermal, power, and space constraints, the hardware will likely be
     186operates will probably not be located at the summit.  Instead, because
     187of thermal, power, and space constraints, the hardware will likely be
    178188located in a facility off the mountain.  A subset of processing tasks
    179189may eventually be assigned to machines at the summit if justified by
    … …  
    186196
    187197This document defines the design requirements of the IPP for the PS-1
    188 prototype telescope.  Even so, much of the IPP design for PS-4 will be
     198prototype telescope.  Much of the IPP design for PS-4 will be
    189199identical to or closely based on the PS-1 implementation.  The
    190 software organization and the infrastructure systems will be
    191 identical, with minor improvements in details.  The range analysis
    192 steps to be performed will be nearly identical, with some additional
    193 details added for PS-4 to improve the accuracy.
    194 
    195 In terms of the IPP, PS-1 differs from the complete PS-4 system in
    196 several important ways.  First, with only one telescope and camera,
    197 the data throughput rate is substantially reduce to a maximum of 1
    198 64-OTA image per 40 seconds rather than 4.  Since PS-1 is a prototype
    199 for testing the Pan-STARRS hardware and software subsystems, the
    200 observing strategy is not a fixed quantity.  The PS-1 Design Reference
    201 Mission (PSDC-xxx) provides some guidelines for the types of projects
    202 to be performed, including starting an AP Survey which will eventually
    203 cover the entire $3\pi$ steradians of the sky accessible to PS-4.  As
    204 a prototype, it is expected that much of the data collected by PS-1
    205 will be processed multiple times to test and tune the analysis steps.
    206 This difference in approach has implications for the storage required
    207 by PS-1: rather than delete images soon after they have been used, raw
    208 images must be stored for at least the first 18 months of PS-1
    209 operations.  We have used the PS-1 Design Reference Mission as a
    210 baseline for these storage requirements to drive our hardware design.
     200software organization and the infrastructure systems are expected to
     201be identical, with minor improvements in details.  The type of
     202analysis steps to be performed will be nearly identical, with some
     203additional details added for PS-4 to improve the accuracy.
     204
     205Although generally very similar, in terms of the IPP PS-1 differs from
     206the complete PS-4 system in several specific ways.  First, with only
     207one telescope and camera, the data throughput rate is substantially
     208reduced to a maximum of 1 64-OTA image per 40 seconds rather than 4.
     209Since PS-1 is a prototype for testing the Pan-STARRS hardware and
     210software subsystems, the observing strategy is not a fixed quantity.
     211The PS-1 Design Reference Mission (PSDC-230-001) provides some
     212guidelines for the types of observing tests which will probably be
     213performed, including possibly starting an Astrometric and Photometric
     214Survey which will eventually cover the entire $3\pi$ steradians of the
     215sky accessible to PS-4.  As a prototype, it is expected that much of
     216the data collected by PS-1 will be processed multiple times to test
     217and tune the analysis steps.  Compare with PS-4, this difference in
     218approach has implications for the storage required by PS-1: rather
     219than delete images soon after they have been used, raw images from
     220demonstration observations must be stored for at least the first two
     221years of PS-1 operations.  The PS-1 Design Reference Mission is used
     222as an upper limit for these storage requirements to drive the hardware
     223design.
    211224
    212225\subsection{System Design Decisions}
    213226
    214227Since Pan-STARRS is a survey project, all data from the telescopes
    215 will be uniformly analysed by the Pan-STARRS Image Processing Pipeline
    216 (IPP) and the appropriate resulting data products made available to
     228will be uniformly analyzed by the Pan-STARRS Image Processing Pipeline
     229(IPP), and the appropriate resulting data products made available to
    217230internal and external science analysis systems as they become
    218231available.  The processing performed by the IPP on the science images
    … …  
    223236object analysis of the static sky images.  In addition, the IPP will
    224237produce improved astrometric and photometric reference catalogs on an
    225 occasional basis as needed.  The output data products from the IPP
    226 consist of the calibration images, reduced images from the individual
    227 telescopes, combined images, difference images, the static sky image,
    228 object photometry, and reference astrometry and photometry.
    229 
    230 The requirements for the IPP, as identified in the IPP SRS (PSDC-REF)
    231 fall into several broad categories: Data analysis precision,
    232 throughput, system reliability, flexibility, testability, and
    233 traceability.  The details of the analysis tasks are specified in
     238as-needed basis.  The output data products from the IPP consist of the
     239calibration images, reduced images from the individual telescopes,
     240combined images, difference images, the static sky image, object
     241photometry, and reference astrometry and photometry.
     242
     243The requirements for the IPP, as identified in the PS-1 IPP SRS
     244(PSDC-430-005) fall into several broad categories: data analysis
     245precision, throughput, system reliability, flexibility, testability,
     246and traceability.  The details of the analysis tasks are specified in
    234247order to achieve the precision.  The architectural design as discussed
    235248below is motivated by the need for reliability and flexibility.  The
    236 hardware organization and the distributed / parallel processing model
    237 is motivated by the throughput requirements.  The need for flexibility
     249hardware organization and the distributed/parallel processing model is
     250motivated by the throughput requirements.  The need for flexibility
    238251and testability drives the software organization.  The need for simple
    239252testing procedures drives both the software organization and the
    … …  
    255268OTA number 61 from exposure 654321 to produce a specific set of output
    256269data products.  The analysis stages are discussed in detail in
    257 Section~\ref{IPP:AnalysisStages}.
    258 
    259 Depending on the particular stage, it may process individual images,
    260 collections of images, or derived data products.  Because of the
    261 nature of the image data, many of the analysis stages can be run in
    262 parallel.  For example, the analysis of a chip in one image does not
    263 depend on the results from another chip.
     270Section~\ref{sec:AnalysisStages}.
     271
     272A particular stage may process individual images, collections of
     273images, or derived data products.  Because of the nature of the image
     274data, many of the analysis stages can be run in parallel if needed to
     275increase the processing throughput.  For example, the analysis of a
     276chip in one image does not depend on the results from another chip.
    264277
    265278\subsection{Architectural Components}
    … …  
    268281\begin{center}
    269282\resizebox{6in}{!}{\includegraphics{pics/IPPoverview}}
    270 \caption{ \label{overview} IPP System Overview}
     283\caption{ \label{fig:overview} IPP System Overview}
    271284\end{center}
    272285\end{figure}
    273286
    274287In order to achieve the required functionality, the IPP provides an
    275 infrastructure within which the Analysis Stages above are exectuted.
    276 In order to facilitate the subsystem testing, we have divided the IPP
    277 software infrastructure into a number of clearly-defined architectural
    278 software units, listed as follows:
     288infrastructure within which the Analysis Stages described above are
     289executed.  In order to facilitate the subsystem testing, the IPP
     290software infrastructure has been divided into a number of
     291clearly-defined architectural software units as follows:
    279292
    280293\begin{itemize}
    … …  
    288301  restricted to imaging data: it is capable of storing any large data
    289302  files which are not well-suited for inclusion in a more structured
    290   relational database and for which access needs to be widely
     303  relational database, and for which access needs to be widely
    291304  available beyond the individual process which created the file.
    292305
    … …  
    295308  needed to perform the IPP analyses.  The metadata may include the
    296309  summary weather information for each night, or details about the
    297   filters, camera, telescopes, etc. 
     310  filters, camera, telescopes, etc.  Note that the IPP Metadata
     311  Database is not required to retain all archival engineering data
     312  from all of Pan-STARRS; other Pan-STARRS subsystems use their own
     313  internal databases to store engineering metadata and only the
     314  necessary subset is transferred to the IPP Metadata Database.
    298315
    299316\item {\bf Astrometry \& Photometry Database (AP DB):} This component
    … …  
    304321  query and manipulate the objects and detections.
    305322
    306 \item {\bf IPP Controller:} In order to perform the analysis stages
    307   required by the IPP, it is necessary to use distributed computing
    308   processes on a large number of computers.  The IPP Controller
    309   manages the collection of analysis tasks performed on these
    310   machines. 
     323\item {\bf IPP Controller:} In order to achieve the required
     324  processing throughput for the IPP analysis stages, it is necessary
     325  to use distributed computing processes on a large number of
     326  computers.  The IPP Controller manages the collection of analysis
     327  tasks performed on these machines.
    311328
    312329\item {\bf IPP Scheduler:} This component is a decision-making
    … …  
    318335
    319336The relationship between these software units is shown in
    320 Figure~\ref{overview}.  This figure also shows the interactions
     337Figure~\ref{fig:overview}.  This figure also shows the interactions
    321338between the IPP and other Pan-STARRS systems.  The architectural
    322 components are discussed in detail in
    323 Section~\ref{IPP:ArchComponents}.
     339components are discussed in detail in Section~\ref{sec:ArchComponents}.
    324340
    325341\begin{figure}
    326342\begin{center}
    327343\resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}}
    328 \caption{ \label{hardware} IPP Hardware Organization}
     344\caption{ \label{fig:hardware} IPP Hardware Organization}
    329345\end{center}
    330346\end{figure}
    … …  
    332348\subsection{IPP Hardware Organization}
    333349
    334 The IPP needs substantial computer resources, both in terms of
     350The IPP will utilize substantial computer resources, both in terms of
    335351computational power and in terms of data storage and network
    336352bandwidth.  The IPP requires relatively large amounts of data storage
    … …  
    354370the static sky storage nodes.
    355371
    356 Figure~\ref{hardware} shows our basic concept for the hardware
     372Figure~\ref{fig:hardware} presents the basic concept for the hardware
    357373organization for the IPP.  This diagram shows the two types of compute
    358 nodes: OTA-level processing and storage nodes and static sky
     374nodes: (1) OTA-level processing and storage nodes and (2) Static Sky
    359375processing and storage nodes.  Also shown are two switches which
    360376divide the network into OTA and Static-Sky portions.  In such an
    361 organization, the interswitch communication must meet the throughput
     377organization, the inter-switch communication must meet the throughput
    362378needs between these network portions (though a single switch may also
    363379be used if its backplane capacity is sufficient).  The additional data
    … …  
    367383
    368384\section{System Design : Architectural Components}
     385\label{sec:ArchComponents}
    369386
    370387\subsection{IPP Image Server}
     388
     389\subsubsection{Corresponding Requirements}
     390
     391The Image Server must meet the requirements specified in Section 3.4.1
     392of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005).  The specified design
     393is chosen to meet requirements 3.4.1.3, and 3.4.1.5.  The other three
     394requirements (3.4.1.1, 3.4.1.2, and 3.4.1.4) depend on the volume and
     395capabilities of the hardware, and are addressed in
     396Section~\ref{sec:Hardware}.
    371397
    372398\subsubsection{Image Server Overview}
    … …  
    378404Server include the raw images, the calibration images, intermediate
    379405processing stage images as needed, final processed images, difference
    380 images, image subsections, and any large non-imaging datafiles needed
     406images, image subsections, and any large non-imaging data files needed
    381407by the IPP.  The IPP Image Server must retain the files for as long as
    382408they are needed by the IPP.
    … …  
    396422There are three data concepts relevant to the IPP Image Server:
    397423\begin{itemize}
    398 \item {\bf storage object} This represents a single, unique data
     424\item {\bf Storage object:} This represents a single, unique data
    399425  entity in the Image Server.
    400426
    401 \item {\bf instance} A single copy of the storage object in the Image
     427\item {\bf Instance:} A single copy of the storage object in the Image
    402428  Server.  In general, a given storage object may have several instances
    403429  in the Image Server, normally on different computer nodes.
    404430
    405 \item {\bf file ID} This is the identifier of a particular storage
     431\item {\bf File ID:} This is the identifier of a particular storage
    406432  object in the Image Server.  The file ID is simply a unique string,
    407433  equivalent to the filename in a UNIX file system.
    … …  
    421447on some schedule.
    422448
    423 The IPP Image Server consists of the following components:
     449As shown in Figure~\ref{fig:ImageServer}, the IPP Image Server
     450consists of the following components:
    424451
    425452\begin{itemize}
    … …  
    428455\item Image Server daemon
    429456\item Image Server client APIs
    430 \item Image Server maintainence tools
     457\item Image Server maintenance tools (not shown)
    431458\end{itemize}
    432459
    … …  
    442469
    443470Clients interact with the IPP Image Server via a small number of C
    444 APIs (Bindings are also provided for Perl and Python and UNIX shell
    445 commands in some cases).  The client commands are:
     471APIs.  Bindings are also provided for Perl and Python and UNIX shell
     472commands in some cases.  The client commands are:
    446473
    447474\begin{itemize}
    … …  
    497524hardware resources.  A {\tt mysql} database engine is used to manage
    498525the database table.  The database tables defined for the Image Server
    499 are listed in Table~\ref{ImageServerTables}, and their contents are
    500 listed in Appendix A.  This database engine need not the same one as
    501 the one used for othe IPP subsystems.
     526are listed in Table~\ref{tab:ImageServerTables}, and their contents are
     527listed in Appendix~\ref{sec:ImageServerTableContents}.  This database
     528engine need not be the same one used for other IPP subsystems.
    502529%
    503 \begin{table}
    504 \begin{center}
    505 \caption{Image Server Database Tables\label{ImageServerTables}}
     530\begin{table}[ht]
     531\begin{center}
     532\caption{Image Server Database Tables\label{tab:ImageServerTables}}
    506533\begin{tabular}{ll}
    507534\hline
    … …  
    529556
    530557The IPP Image Server provides a collection of administration tools
    531 which allow for maintainence.  These are operations which may be
     558which allow for maintenance.  These are operations which may be
    532559automatically scheduled by the IPP or which may be initiated by a
    533 human via a command-shell interface.  The maintainence functions
    534 include migrating data between nodes to rebalance the available space
     560human via a command-shell interface.  The maintenance functions
     561include migrating data between nodes to re-balance the available space
    535562(this would only occur for instances which have not been placed on a
    536563specific node by the client API).  Other functions include checking
    … …  
    542569
    543570\subsection{Metadata Database}
    544 \label{Metadata}
     571\label{sec:Metadata}
     572
     573\subsubsection{Corresponding Requirements}
     574
     575The Metadata Database must meet the requirements specified in Section
     5763.4.2 of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005).  The specified
     577design is chosen to meet requirements 3.4.2.1, 3.4.2.2, 3.4.2.3,
     5783.4.2.4, 3.4.2.5.
     579
     580\subsubsection{Overview}
    545581
    546582The IPP Metadata Database acts as a repository for non-pixel data
    … …  
    558594Metadata Database may be collected and inserted by a separate,
    559595dedicated process.  Metadata which is large in volume or poorly
    560 structure may also be stored in an appropriate container file (FITS
     596structured may also be stored in an appropriate container file (FITS
    561597Table, FITS Header, XML File) in the Image Server with the Metadata DB
    562598providing pointers to these files.
    … …  
    568604\begin{table}[hb]
    569605\begin{center}
    570 \caption{Metadata Database Tables\label{MetadataDBTables}}
     606\caption{Metadata Database Tables\label{tab:MetadataDBTables}}
    571607\begin{tabular}{ll}
    572608\hline
    … …  
    597633\subsubsection{Metadata Tables}
    598634
    599 The contents of the Metadata Database will not be completely specified
    600 until the complete collection of data analysis scripts are available.
    601 Even so, we can identify the likely collection of long-term tables,
    602 and some of the temporary processing tables.
    603 Table~\ref{MetadtaDBTables} lists the Metadata tables identified to
     635Table~\ref{tab:MetadataDBTables} lists the Metadata tables identified to
    604636date for the Metadata Database.  The contents of these tables are
    605 outlined in Appendix~\ref{MetadataContents}, with examples for the
    606 data entries and thier data types in many cases.
     637outlined in Appendix~\ref{sec:MetadataTableContents}, with examples for
     638the data entries and their data types in many cases.  Additional
     639tables will be added as necessary as the data analysis scripts are
     640fleshed out in detail.  The Metadata Database, with a flat data
     641organization, is flexible enough to add additional information as it
     642is identified.
    607643
    608644\subsubsection{Metadata Queries}
    609645
    610646The IPP provides simple queries to the Metadata Database tables using
    611 autocoded APIs.  These queries return a single row or a collection of
     647auto-coded APIs.  These queries return a single row or a collection of
    612648rows based on the primary key.  The format of the API is identical for
    613649all Metadata tables.  New tables and APIs can be added to the IPP
    614 system by adding to the autocoding table description files.  See
    615 Appendix~\ref{Autocode} for futher information.
     650system by adding to the auto-code table description files.  The
     651auto-code API includes read and write access permissions to be set
     652for each table independently. See Appendix~\ref{sec:AutocodeIO} for
     653further information.
    616654
    617655%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    … …  
    619657\subsection{AP Database}
    620658
     659\subsubsection{Corresponding Requirements}
     660
     661The AP Database must meet the requirements specified in Section 3.4.3
     662of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005).  The specified design
     663is chosen to meet requirements 3.4.3.1 and 3.4.3.2.  In order to meet
     664the throughput requirements, the AP Database will be distributed
     665across 10 Nodes independent of the Image Server Nodes.  An alternative
     666organization of the database which will be studied will have the AP
     667Database co-located with the Image Server Phase 4 Nodes.
     668
    621669\subsubsection{Overview}
    622670
    623 The AP (Astrometry \& Photometry) Database is a mechanism to store
    624 data related to astronomical objects derived from various sources with
    625 a variety of associations.  The AP Database deals with two related
    626 concepts: {\em objects} and {\em detections}.  The objects are
    627 descriptions of astronomical objects while the detections are the
    628 specific measurements of those objects, typically measured from
     671The AP (Astrometry \& Photometry) Database is a CSCI which stores data
     672related to astronomical objects derived from various sources with a
     673variety of associations.  The AP Database deals with two related
     674concepts: {\em objects} and {\em detections}.  The {\em objects} are
     675descriptions of astronomical objects while the {\em detections} are
     676the specific measurements of those objects, typically measured from
    629677astronomical images.  A collection of {\em detections} may be used to
    630678derive average quantities which describe a particular {\em object}.  A
    631 third class of object information which must also be considered are
    632 those supplied by external references.  These may be treated as {\em
     679third class of measurement to be considered are those supplied by
     680external references.  Such measurements may be treated as {\em
    633681detections}, with the caveat that access to the raw measurements and
    634682metadata are usually unavailable: the reported measurements and errors
    … …  
    637685The AP Database stores the collections of detections which were
    638686derived from specific images from any of the analysis stages.  It
    639 provides a mechanism to determine and (in conjunction with the Image
    640 Server) locate the image from which a specific detection was derived.
    641 The AP Database also makes it possible to extract all detections
    642 derived from a specific image and to determine quantities such as the
    643 pixel coordinates of the detection on the image.
     687provides a mechanism to determine the image from which a specific
     688detection was derived, and in conjunction with the Image Server locate
     689the corresponding data file.  The AP Database also makes it possible
     690to extract all detections derived from a specific image and to
     691determine quantities such as the pixel coordinates of the detection on
     692the image.
    644693
    645694The AP Database also has the capability to associate multiple
    … …  
    648697
    649698First, the most distant stars, compact galaxies, and QSOs will have
    650 nearly fixed locations relative to other nearby stars, with only small
    651 deviations for individual measurements.  The association between
     699nearly fixed locations relative to other distant stars, with only
     700small deviations for individual measurements.  The association between
    652701multiple detections of such objects is made on the basis of their
    653702coincident positions.  The AP Database determines the average position
    … …  
    658707of such objects are linked by their orbits, and depend on both the
    659708position and the time of the image.  The AP Database does not attempt
    660 to make this link, which is the role of the MOPS system.  However, it
     709to make this link; this is the role of the MOPS system.  However, it
    661710has the ability to accept identifications made externally with
    662711specified detections and to return the identifier of the moving object
    … …  
    667716moving object detections from other types of queries.
    668717
    669 Third, stars in the general vicinity of the solar system fall in
     718Third, objects in the general vicinity of the solar system fall in
    670719between these first two classes of objects.  Their proper motion and
    671 parallax response is significant enough ($>1$ arcsec in 1 year) that
     720parallax response is significant enough ($>0.2$ arcsec in 1 year) that
    672721they are not well-described by an average location and a collection of
    673722offsets.  These objects are described by a distance and a proper
    … …  
    679728be associated with a specific astronomical object of any of the above
    680729classes and are treated as orphans.  Most of these will be spurious
    681 (not represent real objects), some will be from solar system objects
    682 for which orbits are not yet determined, some will be from faint stars
    683 near the detection limits, some will be from short-term transients
    684 which have only been detected once.  The AP Database maintains these
    685 detections until they have been associated with one of the objects
    686 above.  The AP Database provides mechanisms by which individual
    687 detections may be migrated back and forth between the orphan state and
    688 association with an astronomical object.
     730(not representing real objects), some will be from solar system
     731objects for which orbits are not yet determined, some will be from
     732faint stars near the detection limits, and some will be from
     733short-term transients which have only been detected once.  The AP
     734Database maintains these detections until they have been associated
     735with one of the objects above.  The AP Database provides mechanisms by
     736which individual detections may be migrated back and forth between the
     737orphan state and association with an astronomical object.
    689738
    690739For every object, and all orphaned detections, the AP Database also
    691 provides the capability to determine the images which observed the
     740provides the capability to determine the images containing the
    692741location of the object but for which no detection was made.  The
    693742minimum set of information which must be carried for these
    … …  
    695744
    696745The AP Database also stores the relationships between various
    697 photometric systems and, in some cases, the evolution of that
    698 relationship.  It provides mechanisms to convert between the measured
    699 instrumental magnitude of a detection with a specific filter,
    700 detector, and telescope, and at a particular time and the implied
    701 magnitude in the average Pan-STARRS photometry system, given a
    702 determined set of calibrations.  It also provides the capability to
    703 convert magnitudes in one system to the magnitudes in another system;
    704 an example of such a conversion is between the average Pan-STARRS
    705 filter systems and the various reference systems appropriate for those
    706 filters.
     746photometric systems and the evolution of that relationship.  It
     747provides mechanisms to convert between the measured instrumental
     748magnitude of a detection with a specific filter, detector, and
     749telescope, and at a particular time and the implied magnitude in the
     750average Pan-STARRS photometry system, given a determined set of
     751calibrations.  It also provides the capability to convert magnitudes
     752in one system to the magnitudes in another system; an example of such
     753a conversion is between the average Pan-STARRS filter systems and the
     754various reference systems appropriate for those filters.
    707755
    708756\begin{figure}
    … …  
    710758\resizebox{4.5in}{!}{\includegraphics{pics/APDB}}
    711759\caption{AP DB components}
    712 \label{fig:APDBRegions}
     760\label{fig:APDBComponents}
    713761\end{center}
    714762\end{figure}
    … …  
    726774time and date of the detection, etc.
    727775
    728 The IPP AP Database consists of the following components:
     776As shown in Figure~\ref{fig:APDBComponents}, the IPP AP Database
     777consists of the following components:
    729778
    730779\begin{itemize}
    … …  
    737786\subsubsection{AP Database Tables}
    738787
    739 Table~\ref{APDBTables} lists the tables used by the AP Database.  The
     788Table~\ref{tab:APDBTables} lists the tables used by the AP Database.  The
    740789contents of these tables are outlined in
    741 Appendix~\ref{APDBTableContents}.  Below, we discuss how these tables
    742 are used by the AP Database software.  Several of the tables are not
    743 just simple tables in the database but are instead divided into many
    744 subtables, each of which represents a portion of the sky.  These
    745 subtables may also be distributed across different computers to
    746 distribute the processing load.
     790Appendix~\ref{sec:APDBTableContents}.  Below, the use of these tables by
     791the AP Database software is discussed below.  Several of the tables
     792are not just simple tables in the database but are instead table
     793groups divided into many subtables, each of which represents a portion
     794of the sky (a {\tt region}).  These subtables may also be distributed
     795across different computers to distribute the processing load.
     796
     797\paragraph{Images Table Group}
    747798
    748799The {\tt Images} table group lists all of the images which provided
    749 the data in the AP Database.  These tables are subdivided by region.
    750 In general, the images listed in this table correspond to the Chips.
    751 This group of tables includes sufficient astrometric parameters to
    752 represent the coordinates of the detections to a sufficient accuracy.
     800the data in the AP Database.  These tables are subdivided by region on
     801the sky.  In general, the images listed in this table correspond to
     802the Chips.  This group of tables includes sufficient astrometric
     803parameters to represent the coordinates of the detections to a
     804sufficient accuracy.  Parallel to the Images table is the Mosaic
     805table.  This table is very similar to the Images table, but defines
     806the Mosaic which corresponds to a group of Images.  The parameters
     807include the astrometric information needed to define the camera
     808distortion.
     809
     810\paragraph{Image Overlaps Table Group}
    753811
    754812The specific subtable of {\tt Images} which contains a given image is
    755 the one which contains the center pixel \tbr{or 0,0 pixel} of that
    756 image.  An additional table group, {\tt Image Overlaps} (with the same
    757 subtable organization as the {\tt Images} subtables), lists images
    758 which overlap that specific subtable.  Thus, given a particular
    759 coordinate, in order to find that images which overlap that
    760 coordinate, it is necessary to search the images in the {\tt Images}
    761 subtable which includes that coordinate, and all images in the {\tt
    762 ImageOverlaps} subtable for that coordinate.
     813the one which contains the center pixel of that image.  An additional
     814table group, {\tt Image Overlaps} (with the same subtable organization
     815as the {\tt Images} subtables), lists images which overlap that
     816specific subtable.  Thus, given a particular coordinate, in order to
     817find that images which overlap that coordinate, it is necessary to
     818search the images in the {\tt Images} subtable which includes that
     819coordinate, and all images in the {\tt ImageOverlaps} subtable for
     820that coordinate.
    763821
    764822\begin{table}[hb]
    765823\begin{center}
    766 \caption{AP Database Tables\label{APDBTables}}
     824\caption{AP Database Tables\label{tab:APDBTables}}
    767825\begin{tabular}{ll}
    768826\hline
    … …  
    770828{\bf Table Name} & {\bf Description} \\
    771829\hline
    772 Images              & The images that have objects in the DB. \\
    773 Image Overlaps      & Image regions which are touched by specific images. \\
    774 Objects             & The objects --- average properties of multiple detections of the same object. \\
    775 Average Magnitudes  & Average photometry in multiple filters \\
    776 Matched Detections  & Detections of sources in an image identified with an Object. \\
    777 Orphaned Detections & Detections of sources in an image not identified with an Object. \\
    778 Non-detections      & Non-detections of objects in an image. \\
    779 Region Table        & spatial distribution of tables \\
    780 Filters             & Filters understood by the system. \\
    781 Photcodes           & Transformations between different photometric systems \\
    782 Database Machines   & computers used to store the tables \\
    783 % Zero Points       & Transformations between different photometric systems \\
    784 % Distortion Models & Transformations between different photometric systems \\
    785 % Solar System Objects & Identification of solar system objects \\
     830Images               & The images that have objects in the DB. \\
     831Image Overlaps       & Image regions which are touched by specific images. \\
     832Objects              & The objects --- average properties of multiple detections of the same object. \\
     833Average Magnitudes   & Average photometry in multiple filters \\
     834Solar System Objects & Identification of solar system objects \\
     835Matched Detections   & Detections of sources in an image identified with an Object. \\
     836Orphaned Detections  & Detections of sources in an image not identified with an Object. \\
     837Non-detections       & Non-detections of objects in an image. \\
     838Regions              & spatial distribution of tables \\
     839Filters              & Filters understood by the system. \\
     840Photcodes            & Transformations between different photometric systems \\
     841Zero Points          & History of Zero-point \& Airmass terms \\
     842Distortion Models    & History of Optical Distortion terms \\
     843Database Hosts       & computers used to store the tables \\
    786844\hline
    787845\end{tabular}
    788846\end{center}
    789847\end{table}
     848
     849\paragraph{Objects Table Group}
    790850
    791851The {\tt Objects} table group (also divided by region) stores the
    … …  
    797857be stored in a separate table. 
    798858
    799 A related table, also divided in the same regions, is the {\tt Average
    800 Magnitudes} table.  In this table, there are multiple rows per average
     859\paragraph{Average Magnitudes Table Group}
     860
     861A related table, also divided into the same regions, is the {\tt
     862Average Magnitudes} table.  In this table, there are multiple rows per
    801863object, one for each of the primary filters of interest for which
    802864photometric averaging is performed.  This organization makes the
    803865number of primary (averaged) filters a configurable value.
     866
     867\paragraph{Matched Detections Table Group}
    804868
    805869The {\tt Matched Detections} table stores all of the measurements of
    … …  
    814878quantities for these types of detections.)
    815879
     880\paragraph{Orphaned Detections Table Group}
     881
    816882The {\tt Orphaned Detections} table stores the detections which have
    817883not been correlated with an existing object.  This table is only
    818884populated for objects below a configuration-specified signal-to-noise
    819 limit (eg 5$\sigma$).  Bright orphaned detections are assigned an
     885limit (e.g., 5$\sigma$).  Bright orphaned detections are assigned an
    820886object and added to the {\tt Matched Detections} table.
     887
     888\paragraph{Non-detections Table Group}
    821889
    822890The {\tt Non-detections} table stores information about detection
    … …  
    827895non-detection statistics.
    828896
     897\paragraph{Regions Table}
     898
    829899The {\tt Regions} table is used to subdivide the tables of images,
    830900objects, and detections, etc, as discussed above.  The AP Database
    831901divides the sky into a hierarchy of regions (portions of the sky) each
    832 of which is in turn sub-divided into smaller portions.  Since nearly
     902of which is in turn subdivided into smaller portions.  Since nearly
    833903all interactions with the AP Database performed by the IPP are limited
    834904in spatial coverage, subdividing the tables allows a specific
    … …  
    846916detection data, the {\tt Regions} table allows for multiple computers
    847917to serve the database tables.  The region file specifies the machine
    848 which stores the specific table.  Figure~\ref{ABDBRegions} illustrates
    849 schematically the subdivision of the sky and the association between
    850 different levels of the hierarchy with different subtables.
     918which stores the specific table.  Figure~\ref{fig:APDBRegions}
     919illustrates schematically the subdivision of the sky and the
     920association between different levels of the hierarchy with different
     921subtables.
    851922
    852923\begin{figure}
    … …  
    857928\end{center}
    858929\end{figure}
     930
     931\paragraph{Other Reference Tables}
    859932
    860933The {\tt Filters} table identifies all of the physical filters
    … …  
    877950
    878951{\bf Option A:} A client chooses one of the machines and sends its
    879 query or data to be inserted to that machine.  The server then uses
    880 the region table to determine which machines contain the relevant
    881 portion of the sky.  The data to be inserted is divided into
    882 corresponding region chunks and sent to the appropriate servers.  In
    883 the case of queries, the queries are redirected to the appropriate
    884 server(s).  The original server may collect the results and return
    885 them to the original client.
     952query or data to that machine.  The server then uses the region table
     953to determine which machines contain the relevant portion of the sky.
     954Data to be added to the database is divided into corresponding region
     955chunks and sent to the appropriate servers.  Queries are redirected to
     956the appropriate server(s).  The original server may collect the
     957results and return them to the original client.
    886958
    887959{\bf Option B:} The client downloads the region table and performs the
    … …  
    893965and making each server symmetric.  The smaller tables (ie, Region,
    894966Filters, etc) could either be downloaded from a single server or
    895 replicated to all AP DB servers.
     967replicated to all AP DB servers.  For these reasons, Option A will be
     968used for the PS-1 IPP..
    896969
    897970\subsubsection{AP Database engine}
    … …  
    922995to the {\tt Matched Detections} table.  Any faint unmatched detections
    923996are added to the {\tt Orphaned Detections} table.  This division is
    924 important because it lets us automatically associate new detections
    925 with existing bright objects and limits the I/O volume required to
    926 make the detections.  In general, there will be many few {\tt Objects}
    927 than {\tt Detections}, and there will be fewer bright orphans than
    928 faint orphans.
     997important because it allows the automatic association of new
     998detections with existing bright objects while limiting the I/O volume
     999required to make the detections.  In general, there will be many fewer
     1000{\tt Objects} than {\tt Detections}, and there will be fewer bright
     1001orphans than faint orphans.
    9291002
    9301003\paragraph{Insert Reference Objects (addrefs)}
    … …  
    9411014This operation uses the overlaps of images and multiple observations
    9421015of the same objects to determine the relative photometry zero-points
    943 for a collection of images.  This is a task which would be run much
    944 more infrequently than the object insertion tasks. 
     1016for a collection of images.  This is a task that wil be run much more
     1017infrequently than the object insertion tasks.
    9451018
    9461019\paragraph{Determine Consistent Photometry Zero Points (uniphot)}
    … …  
    9511024atmospheric stability.
    9521025
    953 \paragraph{Determine Distortion Model (mosastro)}
     1026\paragraph{Determine Distortion and Static Astrometry Model (mosastro)}
    9541027
    9551028This operation uses the reference and image detections to determine an
    956 optical distortion model for the camera.
     1029optical distortion model for the camera and static astrometry model
     1030components.  The astrometry model includes: (1) field distortion
     1031introduced by the telescope optics, which is a smoothly-varying
     1032function of the field position relative to the center of the telescope
     1033boresite coordinates.  (2) focal plane geometry, which includes the
     1034chip positions and rotations in the focal relative to the boresite,
     1035along with chip-dependent plate-scale modifications needed to
     1036represent tilts or warps of the individual detectors relative to the
     1037ideal flat focal plane. .
    9571038
    9581039\begin{table}
    9591040\begin{center}
    960 \caption{AP Detection Classes \& Object Parameters\label{APdetections}}
     1041\caption{AP Detection Classes \& Object Parameters\label{tab:APdetections}}
    9611042\begin{tabular}{lrrrr}
    9621043\hline
    … …  
    9781059\end{table}
    9791060
    980 \subsubsection{Notes}
    981 
    982 discuss AP DB throughput issues
    983 
    984 how does the AP Database know about the relationship between a
    985 collection of chips? 
    986 
    987 what is astrometry representation in image table? 3rd order polynomial
    988 across the chip?
    989 
    990 does the AP Database know about FPA, Chip, Distortion Model, etc?  I
    991 think it probably needs to if it is going to solve for distortion
    992 models.  however, this operation may be a combination of AP DB
    993 interaction and MD DB interaction.
     1061\subsubsection{Throughput}
     1062
     1063The AP Database design partly driven by the need to make the
     1064detection-object associations quickly and to processes the incoming
     1065detections at a sufficiently high rate to meet the throughput
     1066requirements.  For each upload of the object detections from a
     1067complete FPA, the AP Database must match roughly $1.4 \times 10^{6}$
     1068detections from an FPA with roughly $6.4 \times 10^{6}$ objects,
     1069including orphaned bright detections.  This corresponds to roughly 640
     1070MB, if each object uses 100 bytes for its descriptive informations
     1071(more than is currently specified in the Object table).  With a
     1072throughput of 100 MB/s for reads from a RAID, the AP Database can
     1073perform the data read in a fraction of a second if the data is
     1074distributed across 10 computers.
    9941075
    9951076%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    9961077
    9971078\subsection{Controller}
     1079\label{sec:Controller}
     1080
     1081\subsubsection{Corresponding Requirements}
     1082
     1083The Controller must meet the requirements specified in Section 3.4.4
     1084of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005).  The design must meet
     1085requirements 3.4.4.1 - 3.4.4.7.  In particular, the Controller / Node
     1086Agent architecture is chosen to control the I/O flow between the
     1087Controller and the individual processes so that blocking on the I/O
     1088from many remote processes does not saturate the Controller
     1089processing.
     1090
     1091\subsubsection{Overview}
    9981092
    9991093\begin{figure}
    … …  
    10271121manage background tasks or if the IPP Controller should attempt to
    10281122send one task per CPU and let the operating system handle the I/O
    1029 load.
     1123load.  The relationship between the different components of the
     1124Controller is illustrated in Figure~\ref{fig:Controller} and discussed
     1125below.
    10301126
    10311127\subsubsection{Nodes}
    10321128
    1033 The Controller maintains a table of processing computers (`Nodes')
    1034 available to it and tracks the status of these Nodes.  Nodes managed
    1035 by the IPP Controller are allowed to be in one of several states, and
    1036 the IPP Controller must interact with it in an appropriate way for
    1037 each of those states.  A computer may be {\tt alive}, {\tt dead} or
    1038 {\tt off}.  If the computer is {\tt alive}, it responds to commands
    1039 from the IPP Controller and may be used for tasks subject to other
    1040 constraints.  If it is {\tt dead}, the computer is not responsive and
    1041 must not be used for executing tasks.  The IPP Controller must
    1042 identify computers which have died (not responding) and occasionally
    1043 test them to see if they are {\tt alive} again.  Computers which are
    1044 {\tt off} are not available for tasks and must not be tested.
    1045 Computers may be set to the {\tt off} or {\tt dead} states by external
    1046 subsystems; it is the responsibility of the IPP Controller to return a
    1047 computer to the {\tt alive} state if possible.
     1129The Controller maintains a table of available processing computers
     1130(`Nodes') and tracks the status of these Nodes.  Nodes managed by the
     1131IPP Controller are allowed to be in one of several states, and the IPP
     1132Controller must interact with it in an appropriate way for each of
     1133those states.  A Node may be {\tt alive}, {\tt dead} or {\tt off}.
     1134If the Node is {\tt alive}, it responds to commands from the IPP
     1135Controller and may be used for tasks subject to other constraints.  If
     1136it is {\tt dead}, the Node is not responsive and must not be used
     1137for executing tasks.  The IPP Controller must identify Nodes which
     1138have died (not responding) and occasionally test them to see if they
     1139are {\tt alive} again.  Nodes which are {\tt off} are not
     1140available for tasks and must not be tested.  Nodes may be set to
     1141the {\tt off} or {\tt dead} states by external subsystems; it is the
     1142responsibility of the IPP Controller to return a Node to the {\tt
     1143alive} state if possible.
    10481144
    10491145The IPP Controller must honor requests (normally from the users) to
    10501146change the mode of any computing node on demand between {\tt off} and
    1051 {\tt dead}.  This would normally be done after a computer has been
     1147{\tt dead}.  This would normally be done after a Node has been
    10521148rebooted and is released to the IPP Controller for its use.  It must
    10531149also be able to change the list of allowed tasks as requested by
    10541150external commands.
    10551151
    1056 Two example scenarios illustrate the transition between these states.
    1057 First, imagine a computer crashes.  At this point the IPP Controller
    1058 should detect that the computer is no longer responsive and mark it
    1059 {\tt dead}.  It should occasionally try to re-establish communication
    1060 with the computer, potentially with longer and longer delays between
    1061 attempts.  A human could be notified if the computer seems to remain
    1062 {\tt dead} for a very long time.  In another scenario, a person needs
    1063 to work on a computer.  They notify the IPP Controller that the
    1064 machine is {\tt off}, perhaps with a prior notification that the
    1065 machine should be prepared to go off.  When work on the machine is
    1066 complete, it should be placed in the {\tt dead} state.  Only when the
    1067 person is done working and testing the machine, and tells the IPP
    1068 Controller that the machine is now {\tt dead} can the IPP Controller
    1069 attempt to re-start communications and processing on that computer.
     1152Two example scenarios illustrate the transition between these states,
     1153and the basic concept of operations for the IPP Controller.  First,
     1154imagine a computer crashes.  At this point the IPP Controller should
     1155detect that the Node is no longer responsive and mark it as {\tt
     1156dead}.  It should occasionally try to re-establish communication with
     1157the Node, potentially with longer and longer delays between attempts.
     1158A human could be notified if the Node seems to remain {\tt dead} for a
     1159very long time.  In another scenario, a person needs to work on a
     1160Node.  They notify the IPP Controller that the machine is {\tt off},
     1161perhaps with a prior notification that the machine should be prepared
     1162to go off.  When work on the machine is complete, it should be placed
     1163in the {\tt dead} state.  Only when the person is done working and
     1164testing the machine, and tells the IPP Controller that the machine is
     1165now {\tt dead} can the IPP Controller attempt to re-start
     1166communications and re-new processing operations on that Node.
    10701167
    10711168\subsubsection{Node Agents}
    10721169
    10731170When the Controller starts, it attempts to launch a Node Agent on each
    1074 of the available processing Nodes.  Modes which are not responsive are
    1075 placed marked as {\tt dead} so they may be retried.  A Node Agent runs
    1076 on each of the individual nodes to execute the tasks as directed by
    1077 the Controller.  The Node Agents communicate with the Controller via a
     1171of the available processing Nodes.  Nodes which are not responsive are
     1172marked as {\tt dead} so they may be re-tried.  A Node Agent runs on
     1173each of the individual nodes to execute the tasks as directed by the
     1174Controller.  The Node Agents communicate with the Controller via a
    10781175socket connection.
    10791176
    1080 A Node Agent (which is only on Node in the {\tt alive} state) may be
    1081 in one of four modes: {\tt idle}, {\tt busy}, {\tt done}, {\tt crash}.
    1082 A Node Agent which is {\tt busy} currently has a task assigned to it
    1083 which is executing.  The IPP Controller may only assign one task to a
    1084 Node at a time.  A Node Agent which is in the {\tt idle} state may
    1085 have a task assigned to it.  When the Node Agent detects that a tasks
    1086 has finished, it changes to either the {\tt done} or {\tt crash}
    1087 states depending on the outcome of the process execution.  The IPP
    1088 Controller must also respect a list of task restrictions which may
    1089 require specific tasks to run on specific CPUs or exclude specific
    1090 tasks from specific CPUs.
     1177A Node Agent (which is only running on a Node in the {\tt alive}
     1178state) may be in one of four modes: {\tt idle}, {\tt busy}, {\tt
     1179done}, {\tt crash}.  A Node Agent which is {\tt busy} currently has a
     1180task assigned to it which is executing.  The IPP Controller may only
     1181assign one task to a Node at a time.  A Node Agent which is in the
     1182{\tt idle} state may have a task assigned to it.  When the Node Agent
     1183detects that a tasks has finished, it changes to either the {\tt done}
     1184or {\tt crash} states depending on the outcome of the process
     1185execution.  The IPP Controller must also respect a list of task
     1186restrictions which may require specific tasks to run on specific CPUs
     1187or exclude specific tasks from specific CPUs.
    10911188
    10921189A task being executed by the Node is run in the UNIX user space as a
    … …  
    11001197
    11011198The Node Agent returns its state ({\tt idle}, {\tt busy}, {\tt done},
    1102 {\tt crash'}) and the exit status of the current processing task, if
     1199{\tt crash}) and the exit status of the current processing task, if
    11031200available.  The reported exit state, if the process has completed
    11041201without crashing, is the UNIX exit state reported by the task: 0--256
    … …  
    11201217\paragraph{Kill task }
    11211218
    1122 The Node Agent should send a kill signal (signal 9 or 15) to the
    1123 current processing task.  When the processing task has exited, the
    1124 Node Agent should set its state to {\tt crash}.
     1219The Node Agent should send a kill signal (\code{KILL} or \code{TERM})
     1220to the current processing task.  When the processing task has exited,
     1221the Node Agent should set its state to {\tt crash}.
    11251222
    11261223\paragraph{Clear task}
    11271224
    11281225The Node Agent should set its state {\tt idle}.  If a processing stage
    1129 is currently running, it should be killed (signal 9 or 15) before the
    1130 task is cleared.
     1226is currently running, it should be killed (\code{KILL} or \code{TERM})
     1227before the task is cleared.
    11311228
    11321229\paragraph{Start processing stage}
    … …  
    11451242valid resource regardless of the node on which the task is executed.
    11461243Input and output data resources must be unique where necessary to
    1147 avoid conflicts.  \tbd{It is the responsibility of the programs to
    1148 wait for network lags (ie, NFS delays)}.  The IPP Controller gives
    1149 each task a unique identifier, which is returned to the requesting
    1150 entity.  The requestor may then use that ID to obtain status
    1151 information on that task or to send control signals to the specific
    1152 task.
     1244avoid conflicts.  It is the responsibility of the task to wait for
     1245network lags (ie, NFS delays).  The IPP Controller gives each task a
     1246unique identifier, which is returned to the requesting entity.  The
     1247requestor may then use that ID to obtain status information on that
     1248task or to send control signals to the specific task.
    11531249
    11541250Task requests may specify a desired node for the task execution.  The
    … …  
    11631259
    11641260Task requests may specify an urgency level.  The IPP Controller
    1165 determines the priority of the task on the basis of both the priority
     1261determines the priority of the task on the basis of both the urgency
    11661262and the age of the request.  An executing task must be completed on a
    11671263CPU before any new task is started on that CPU, regardless of
    1168 priority.  Tasks may be assigned a priority of 0 in which case they
    1169 are maintained in the queue and never executed.
     1264priority.  The urgency levels range from 0 to 2.  Tasks with an
     1265urgency of 1 are scheduled whenever they reach the top of the stack.
     1266Tasks with an urgency of 2 are sent immediately to the top of the
     1267stack. Tasks assigned a priority of 0 are maintained in the queue and
     1268never executed.
    11701269
    11711270It may be useful for the Controller to distinguish between tasks
    … …  
    11851284completed.
    11861285
    1187 \subsubsection{External Interfaces}
     1286\subsubsection{Controller Interfaces}
    11881287
    11891288The IPP Controller must accept commands from other IPP subsystems.
    … …  
    12371336\subsection{Scheduler}
    12381337
     1338\subsubsection{Corresponding Requirements}
     1339
     1340The Scheduler must meet the requirements specified in Section 3.4.5 of
     1341the Pan-STARRS PS-1 IPP SRS (PSDC-430-005).  The design must meet
     1342requirements 3.4.5.1 - 3.4.5.7.  In particular, the Task / Test
     1343division is chosen to prevent the Scheduler from blocking while an
     1344analysis process is performed.  Scheduling requirements will be met by
     1345defining appropriate Test periods for the different Tasks.
     1346
     1347\subsubsection{Overview}
     1348
    12391349The IPP is responsible for a variety of analysis jobs: processing of
    12401350the science images through several stages; routine assessment of the
    … …  
    12501360and initiate the actions.
    12511361
    1252 The IPP Scheduler acts as an intermediary between several components
    1253 of the IPP and also between the IPP and external agents such as OTIS
    1254 and the users who must monitor the behavior of the IPP.  The IPP
    1255 Scheduler may be viewed as the central brain of the IPP.
    1256 Figure~\ref{Scheduler} illustrates the design of the IPP Scheduler.
     1362The IPP Scheduler acts as an interface between several components of
     1363the IPP and also between the IPP and external agents such as OTIS and
     1364the users who must monitor the behavior of the IPP.  The IPP Scheduler
     1365may be viewed as the central brain of the IPP.
     1366Figure~\ref{fig:Scheduler} illustrates the design of the IPP
     1367Scheduler.
    12571368
    12581369\subsubsection{Scheduler Tasks and Tests}
    … …  
    12811392\begin{center}
    12821393\resizebox{6in}{!}{\includegraphics{pics/Scheduler}}
    1283 \caption{ \label{Scheduler} IPP Scheduler}
     1394\caption{ \label{fig:Scheduler} IPP Scheduler}
    12841395\end{center}
    12851396\end{figure}
    … …  
    12881399While the IPP Scheduler chooses the tasks to be performed, it is the
    12891400IPP Controller's responsibility to manage the specific tasks executing
    1290 on a given processing node.  This division of responsibilites allows
    1291 us to isolate and encapsulate the functionality of the IPP Scheduler
    1292 and the IPP Controller.  With this separation, the IPP Controller does
    1293 not need to have any information about the details of the tasks which
    1294 it executes, while the IPP Scheduler does not need to monitor the
     1401on a given processing node.  This division of responsibilities allows
     1402the different functionalities of the IPP Scheduler and the IPP
     1403Controller to be isolated and encapsulated.  With this separation, the
     1404IPP Controller does not information about the details of the tasks it
     1405executes, while the IPP Scheduler does not need to monitor the
    12951406computer hardware.
    12961407
    … …  
    12981409bi-directional; the IPP Scheduler sends tasks to the IPP Controller,
    12991410while the IPP Controller informs the IPP Scheduler of the outcome of
    1300 those tasks.  It is not specified whether the IPP Scheduler and IPP
    1301 Controller are components of a single software system or interacting
    1302 but distinct software components.
     1411those tasks.  For the PS-1 IPP, the IPP Scheduler and the IPP
     1412Controller are distinct, interacting software components.  The
     1413interface mechanisms are described in Section~\ref{sec:interfaces}.
    13031414
    13041415\subsubsection{Task Rules}
    … …  
    13061417The IPP Scheduler takes as input a collection of rules which define
    13071418the dependency of tasks on certain tests.  The IPP Scheduler must
    1308 choose between several types of analysis tasks based on those ruls and
    1309 on results of the tests.  The timescale on which different tasks (and
    1310 their related tests) are executed may vary from 10s of seconds to
    1311 hours, days, or even week.  The list of tasks which the IPP Scheduler
    1312 must decide between, and the relevant timescale, follow:
     1419choose between several types of analysis tasks based on those rules
     1420and on results of the tests.  The timescale on which different tasks
     1421(and their related tests) are executed may vary from 10s of seconds to
     1422hours, days, or even as long as a week.  The list of tasks which the
     1423IPP Scheduler must decide between, and the relevant timescale, follow:
    13131424\begin{itemize}
    13141425\item moving data from the Summit pixel server ($\sim 30$ second timescales)
    … …  
    13181429\item constructing new detrend images ($\sim$ weekly)
    13191430\end{itemize}
    1320 The scheduler may be viewed as a complex state machine.  Our goal is
     1431The scheduler may be viewed as a complex state machine.  The goal is
    13211432to design the scheduler so that rules may be specified independently
    1322 from the engine which parses the rules to detemine which specific jobs
     1433from the engine which parses the rules to determine which specific jobs
    13231434to send to the controller.
    13241435
    13251436\subsubsection{User Interface}
    13261437
    1327 The IPP Scheduler provides a user interface which allows a human
     1438The IPP Scheduler shall possess a user interface which allows a human
    13281439operator, or other processes, to monitor the current state of the
    13291440Scheduler.  Users have the option to specify that a particular task or
    1330 set of tasks is of higher or lower priority than the norm, or to
    1331 schedule a particular tasks on a different timescale from the basic
    1332 rule.
    1333 
    1334 The IPP Scheduler defines the operating state of the IPP.  When the
    1335 IPP is in the {\em automatic state}, the IPP Scheduler performs the
     1441set of tasks is of higher or lower urgency (as defined in
     1442Section~\ref{sec:Controller}) than the norm, or to schedule a
     1443particular tasks on a different timescale from the basic rule.
     1444
     1445The IPP Scheduler defines the operating state of the IPP and shares
     1446the same set of states:
     1447\begin{itemize}
     1448\item active state
     1449\item interactive state
     1450\item paused state
     1451\end{itemize}
     1452When the IPP Scheduler is in the {\em active state}, it performs the
    13361453most appropriate of all possible tasks at a particular time.  When the
    1337 IPP is in the {\em interactive state}, the IPP Scheduler performs only
    1338 the requested action regardless of the outcome of the decision trees.
    1339 In addition, in the interactive state, the IPP Scheduler must only
    1340 perform the requested actions and not attempt to perform the other
    1341 normally-required actions.  The only exception to this exclusion is
    1342 that, in the interactive state, data is still copied from the summit
    1343 system.  An additional IPP state is the {\em paused state}, intended
    1344 for tests or maintenance, in which case the IPP Scheduler does not
    1345 perform even the data copy tasks.  Every task is performed on demand
    1346 by the user.  A user command sets the IPP Scheduler in one of these
    1347 three states, {\em automatic}, {\em interactive}, and {\em paused}.
     1454IPP Scheduler is in the {\em interactive state}, it performs only a
     1455specific requested action regardless of the outcome of the decision
     1456trees.  In addition, in the interactive state, the IPP Scheduler must
     1457only perform the requested actions and not attempt to perform the
     1458other normally-required actions.  The only exception to this exclusion
     1459is that, in the interactive state, data is still copied from the
     1460summit system.  An additional IPP state is the {\em paused state},
     1461intended for tests or maintenance, in which case the IPP Scheduler
     1462does not perform even the data copy tasks.  Every task is performed on
     1463demand by the user.  A user command sets the IPP Scheduler in one of
     1464these three states, {\em active}, {\em interactive}, and {\em paused}.
    13481465
    13491466%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    13501467
    13511468\section{System Design : Science Analysis Tasks and Stages}
    1352 
    1353 In this section, we discuss the design of the science analysis stages
    1354 which perform the fundamental image analysis steps of the IPP.  The
    1355 IPP science image processing stages perform analyses on the night-sky
     1469\label{sec:AnalysisStages}
     1470
     1471This section describes the design of the science analysis stages which
     1472perform the fundamental image analysis steps of the IPP.  The IPP
     1473science image processing stages perform analyses on the night-sky
    13561474science images to extract the science data from these images.  These
    1357 consist of: Phase 1, the image processing preparation stage; Phase 2,
    1358 the image reduction stage; Phase 3, the exposure analysis stage; and
    1359 Phase 4, the image combination stage.  These analysis tasks must
    1360 process the images in a timely manner so that the incoming data stream
    1361 will not overload the IPP Image Server.  The decision to execute a
    1362 specific pipeline for a specific dataset is made by the Scheduler,
    1363 which sends the infomation to the Controller.  The Controller executes
    1364 the pipeline for the data on an appropriate machine and monitors the
    1365 success or failure of the processing stage.
     1475consist of:
     1476\begin{itemize}
     1477\item Phase 1, the image processing preparation stage,
     1478\item Phase 2, the image reduction stage
     1479\item Phase 3, the exposure analysis stage
     1480\item Phase 4, the image combination stage. 
     1481\end{itemize}
     1482These analysis tasks must process the images in a timely manner so
     1483that the incoming data stream will not overload the IPP Image Server.
     1484The decision to execute a specific pipeline for a specific dataset is
     1485made by the Scheduler, which sends the information to the Controller.
     1486The Controller executes the pipeline for the data on an appropriate
     1487machine and monitors the success or failure of the processing stage.
    13661488
    13671489The analysis stages are written as UNIX commands, which may be
    … …  
    13841506
    13851507The recipe is loaded as part of the runtime configuration information
    1386 loaded when the analysis script starts.  We define four levels of
    1387 runtime configuration information.  The {\tt site} configuration
     1508loaded when the analysis script starts.  Four levels of runtime
     1509configuration information are defined.  The {\tt site} configuration
    13881510defines values specific to the particular installation of the
    13891511software.  For example, the name of the machine which hosts the
    … …  
    14081530also be specified on the command line.  Examples of the recipe and
    14091531other runtime configuration options are given in
    1410 Appendix~\ref{RuntimeConfig}.
     1532Appendix~\ref{sec:RuntimeConfig}.
    14111533
    14121534%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    … …  
    14221544magnification.  The guide star coordinates are loaded from the
    14231545Metadata database.  These calculations are performed by comparing the
    1424 observed guide star detector coodinates with the known astrometic
     1546observed guide star detector coordinates with the known astrometric
    14251547positions of these same stars as reported by an external astrometric
    14261548reference.  The accuracy of the resulting astrometric solution is
    … …  
    14401562detection to determine the detector coordinates of those bright stars
    14411563which are not saturated but which are significantly above the
    1442 background level.  By targetting known locations in the image files,
     1564background level.  By targeting known locations in the image files,
    14431565only a small amount of data will have to be read.
    14441566
    … …  
    14551577phase.  It is acceptable for a small number of invalid overlaps to be
    14561578identified as these will be excluded in Phase 4.  Static Sky cells
    1457 which do not have sufficient science image overlap \tbr{$< 5\%$} need
    1458 not be processed because the few new measured pixels do not add
     1579which do not have sufficient science image overlap ($< 5\%$) need not
     1580be processed because the few new measured pixels do not add
    14591581significantly to the Static Sky.
    14601582
    1461 \subsubsection{Notes}
     1583\subsubsection{Examples}
     1584
     1585Examples of Phase 1 as called from the command line, with different
     1586types of images:
    14621587
    14631588\begin{verbatim}
    … …  
    15051630\subsubsection{Load Images}
    15061631
    1507 The Phase 2 analysis must load the science image to be analysed into
     1632The Phase 2 analysis must load the science image to be analyzed into
    15081633memory, as well as the corresponding metadata (either from the image
    15091634header and/or from the IPP Metadata Database).  It must use the
    … …  
    15201645
    15211646Science images which have been obtained with Orthogonal-Transfer
    1522 Guiding have had thier pixel response smoothed by the image correction
     1647Guiding have had their pixel response smoothed by the image correction
    15231648motion.  For these images, some of the detrend images need to be
    15241649convolved by the same OT kernel, so that they accurately represent the
    … …  
    15391664fringe frame(s) by the OT convolution kernel.  Specific flags in the
    15401665static bad pixel mask are also grown by the outline of the OT
    1541 convolution kernel (see Section \ref{ap:masks}).
     1666convolution kernel (see Section~\ref{sec:masks}).
    15421667
    15431668\subsubsection{Bias Correction / Overscan Subtraction}
    15441669
    15451670The image bias must be subtracted. Since different detectors behave in
    1546 different ways, several options for modelling the bias are available.
     1671different ways, several options for modeling the bias are available.
    15471672The bias is measured from the image overscan region.  The bias
    15481673subtraction method must be capable of subtracting a single constant
    15491674from the complete image, or to subtract a 1-D bias which varies as a
    15501675function along the overscan.  The function used to represent the
    1551 overscan region may be a spline or a chebychev polynomial derived from
     1676overscan region may be a spline or a Chebychev polynomial derived from
    15521677the data values along the overscan.  The values used to determine both
    15531678the single constant or the inputs to the spline and polynomial fits
    … …  
    15621687
    15631688\subparagraph{Flag bad and saturated pixels}
     1689\label{sec:masks}
    15641690
    15651691A static bad pixel mask is used to identify pixels which are known to
    … …  
    15671693image. Bad pixels which are charge traps are grown by the extent of
    15681694the OT convolution kernel.  Bad pixels above a charge trap (i.e.\ bad
    1569 colums) must not be grown, since they were not affected by pixel
     1695columns) must not be grown, since they were not affected by pixel
    15701696shifting, but only became bad at read-out.
    15711697
    … …  
    16331759artifacts generated by bright stars: bleeding columns, ghosts, or
    16341760other localized reflection effects.  This process also produces a
    1635 superbinned image of the background map which may be used as a
     1761super-binned image of the background map which may be used as a
    16361762debugging diagnostic.
    16371763
    … …  
    16471773\subsubsection{Detect and Measure objects}
    16481774
    1649 After the image have been processed by the preceeding steps, the Phase
     1775After the image have been processed by the preceding steps, the Phase
    165017762 analysis performs a basic object detection analysis.  Objects on the
    16511777flat-fielded object image are found, and general parameters are
    16521778measured.  Object detection is performed at several stages by the IPP,
    16531779with different object parameters measured in each case.
    1654 Table~\ref{APdetections} gives a list of the different detection
     1780Table~\ref{tab:APdetections} gives a list of the different detection
    16551781stages and the object parameters measured for those stages.  For the
    16561782Phase 2 analysis, the object parameters are: the object centroid and
    1657 the position covarience matrix, the instrumental PSF magnitude and
     1783the position covariance matrix, the instrumental PSF magnitude and
    16581784error, local background level and error, a measurement of the
    16591785star-galaxy separation, and a measurement of the object shape
    … …  
    16931819(either stars with poorly determined proper motion or spurious
    16941820matches).  The resulting astrometric solution is consistent across the
    1695 OTA field to within \tbr{0.2 arcsec}.
     1821OTA field to within 1.0 arcsec.
    16961822
    16971823\subsubsection{Perform Photometry}
    … …  
    17191845%\begin{center}
    17201846%\resizebox{6in}{!}{\includegraphics{pics/phase2}}
    1721 %\caption{ \label{phase2} Phase 2 dataflow - this diagram is old: update}
     1847%\caption{ \label{fig:phase2} Phase 2 dataflow - this diagram is old: update}
    17221848%\end{center}
    17231849%\end{figure}
    … …  
    17531879center, followed by a rotation to the average rotation of the FPA and
    17541880adjustment for the central plate scale.  The free parameters in this
    1755 stage are the boresite coordiates ($R_o, D_o$), the field rotation
     1881stage are the boresite coordinates ($R_o, D_o$), the field rotation
    17561882($\theta_o$) and the plate scale ($\rho_o$), and are fitted in Phase
    175718831.  These tangent plane coordinates are then distorted by the optical
    … …  
    17781904local reference catalog.  This analysis may only be performed if a
    17791905local reference is available.  Note that improved relative photometry
    1780 calculations may be performed in the absense of a reference catalog on
     1906calculations may be performed in the absence of a reference catalog on
    17811907the basis of image overlaps in the AP Database {\em after} the
    17821908detections have been added to the Database.  Such a relative
    … …  
    17841910performed as an independent analysis process.  Given the presence of a
    17851911local photometry reference, the zero point variations across the field
    1786 may be measured, and possibly modelled.  If the zero-point variations
     1912may be measured, and possibly modeled.  If the zero-point variations
    17871913are excessive, then the image is marked as non-photometric by the
    17881914analysis.
    … …  
    18101936same number of pixels as an OTA (4k x 4k) and represent a portion of a
    18111937local tangent plane projection.  In order to meet the image
    1812 degredation requirements, the pixel scale of the static sky is planned
     1938degradation requirements, the pixel scale of the static sky is planned
    18131939to be 0.2\arcsec, somewhat smaller than the 0.3\arcsec\ raw image
    18141940pixel scale.
    … …  
    18491975between the input image and the static sky image.  This will be done
    18501976by solving for a best-fit image kernel which minimizes the difference
    1851 image using a technique equivalent to the Allard-Lupton method.  The
    1852 modification we make is that, rather than represent the components of
    1853 the image difference kernel as a combination of Gaussians, we will
    1854 represent the kernel as a combination of pixels.  This method also
    1855 automatically determines a photometric match between the static sky
    1856 image and the input science image.
     1977image using a technique equivalent to the Allard-Lupton method.  One
     1978modification for the IPP is to represent the kernel as a combination
     1979of independent pixels rather than represent the components of the
     1980image difference kernel as a combination of Gaussians.  This method
     1981also automatically determines a photometric match between the static
     1982sky image and the input science image.
    18571983
    18581984\subsubsection{Object Detection and Measurement}
    … …  
    18601986Objects in the difference image are detected and a specific set of
    18611987object parameters are measured from these detections.
    1862 Table~\ref{APdetections} gives a list of the different detection
     1988Table~\ref{tab:APdetections} gives a list of the different detection
    18631989stages and the object parameters measured for those stages.  For the
    18641990Phase 4 difference image (P4$\Delta$), the measured object parameters
    1865 consist of: the object centroid and the position covarience matrix,
     1991consist of: the object centroid and the position covariance matrix,
    18661992the instrumental PSF magnitude and error, local background level and
    18671993error, a measurement of the star-galaxy separation, and a measurement
    … …  
    18782004Objects in the cleaned, summed image are detected and a specific set
    18792005of object parameters are measured from these detections.
    1880 Table~\ref{APdetections} gives a list of the different detection
     2006Table~\ref{tab:APdetections} gives a list of the different detection
    18812007stages and the object parameters measured for those stages.  For the
    18822008Phase 4 summed image (P4$\Sigma$), the measured object parameters
    1883 consist of: the object centroid and the position covarience matrix,
     2009consist of: the object centroid and the position covariance matrix,
    18842010the instrumental PSF magnitude and error, local background level and
    18852011error, a measurement of the star-galaxy separation, a measurement of
    … …  
    19212047%\begin{center}
    19222048%\resizebox{6in}{!}{\includegraphics{pics/phase4}}
    1923 %\caption{ \label{phase4} Phase 4 dataflow}
     2049%\caption{ \label{fig:phase4} Phase 4 dataflow}
    19242050%\end{center}
    19252051%\end{figure}
    … …  
    19402066The Calibration analysis stages may be performed on whatever
    19412067timescales are appropriate and necessary to maintain the quality and
    1942 relevance of the calibration images.  Below, we list the specific
    1943 calibration data which must be constructed in the calibration analysis
    1944 stages. 
     2068relevance of the calibration images.  The specific calibration data
     2069which must be constructed in the calibration analysis stages is listed
     2070below.
    19452071
    19462072The IPP must generate basic calibration images using the raw bias,
    … …  
    20732199thin-film interference must also be detected and corrected.  Models of
    20742200this background structure may be a necessary input to the correction
    2075 proceedure.  The IPP must have the capability of generating image
     2201procedure.  The IPP must have the capability of generating image
    20762202models of the large-scale structure patterns observed with the
    20772203telescope
    … …  
    20862212moved to a variety of locations on the detector in a sequence of
    20872213images.  The flat-field correction frames analysis stage makes use of
    2088 targetted observations following a specified dither pattern, and
     2214targeted observations following a specified dither pattern, and
    20892215extracts the photometered objects from the AP Database to determine
    20902216the necessary photometric corrections.  The resulting image is applied
    … …  
    20922218performed by applying the correction to the basic master flat-field
    20932219image, applying that flat-field image to the dithered photometry
    2094 observations, and performing the object detections.  Comparion of the
     2220observations, and performing the object detections.  Comparison of the
    20952221photometry of individual stars at different locations on the mosaic
    20962222will demonstrate the consistency of the flat-field image.
    … …  
    21102236\section{System Design : Miscellaneous Tasks}
    21112237
    2112 In this section, we discuss additional operations which are performed
    2113 by the IPP but which do not fall under the analysis of the science
    2114 images or the creation of the calibration images. 
     2238This section discusses additional operations which are performed by
     2239the IPP but which do not fall under the analysis of the science images
     2240or the creation of the calibration images.
    21152241
    21162242\subsection{Retrieval}
    … …  
    21282254performed in the real-time analysis.  The currently envisioned
    21292255parameters to be measured for every object are listed in
    2130 Table~\ref{APdetections}.  The parameters include the object centroid
    2131 and the position covarience matrix, the instrumental PSF magnitude and
     2256Table~\ref{tab:APdetections}.  The parameters include the object centroid
     2257and the position covariance matrix, the instrumental PSF magnitude and
    21322258error, local background level and error, a measurement of the
    21332259star-galaxy separation, a measurement of the object shape ($\sigma_x,
    … …  
    22002326\subsection{Pan-STARRS Library}
    22012327
    2202 The Pan-STARRS Library will consist of C structures describing the basic
    2203 data types needed by the IPP and C functions which perform the basic
    2204 data manipulation operations.  Note that a subset of the library
     2328The Pan-STARRS Library will consist of C structures describing the
     2329basic data types needed by the IPP and C functions which perform the
     2330basic data manipulation operations.  Note that a subset of the library
    22052331functions will be provided with SWIG interfaces as well to allow for
    22062332their use in the creation of the processing stages.  Examples of the
    2207 Pan-STARRS Library are fourier transforms and transforming between pixel
    2208 and celestial coordinates.
    2209 
    2210 \subsection{Modules}
     2333Pan-STARRS Library are Fourier transforms and transforming between
     2334pixel and celestial coordinates.  The details of the Pan-STARRS
     2335Library are specified in the document Pan-STARRS IPP PSLib
     2336Supplementary Design Requirements Specification (PSDC-430-007), which
     2337also addresses coding requirements detailed in the IPP PS-1 SRS
     2338(PSDC-430-005), Section 3.3.
     2339
     2340\subsection{IPP Modules}
    22112341
    22122342The IPP analysis stages are broken down into modules which represent
    22132343specific functional operations.  The modules will be written in C
    2214 using the Pan-STARRS Library functions and will be grouped into a Pan-STARRS
    2215 Module Library.  The modules will be provided with SWIG interfaces to
    2216 all public APIs for their use in processing stages.  Examples of
    2217 modules are overscan subtraction and image combination.  Some modules
    2218 (e.g.\ find objects on an image) will be used by multiple stages.
    2219 
    2220 \subsection{Stages}
     2344using the Pan-STARRS Library functions and will be grouped into a
     2345Pan-STARRS Module Library.  The modules will be provided with SWIG
     2346interfaces to all public APIs for their use in processing stages.
     2347Examples of modules are overscan subtraction and image combination.
     2348Some modules (e.g.\ find objects on an image) will be used by multiple
     2349stages.  The details of the Pan-STARRS Modules are specified in the
     2350document Pan-STARRS IPP Modules Supplementary Design Requirements
     2351Specification (PSDC-430-012), which also addresses coding requirements
     2352detailed in the IPP PS-1 SRS (PSDC-430-005), Section 3.3.
     2353
     2354\subsection{IPP Stages}
    22212355
    22222356The major IPP processing tasks are organized into stages, which
    … …  
    22322366
    22332367\section{Interfaces}
     2368\label{sec:interfaces}
    22342369
    22352370\subsection{Internal Interfaces}
    … …  
    22562391
    22572392FITS Tables will be used to store and transport tabular data,
    2258 especially large queries from database subsystems.  The Autocoding
    2259 technique discussed in Appendix~\ref{Autocode} is used to define many
     2393especially large queries from database subsystems.  The Auto-coding
     2394technique discussed in Appendix~\ref{sec:AutocodeIO} is used to define many
    22602395different table interactions.
    22612396
    … …  
    22692404interface to the databases.
    22702405
     2406Within IPP and Pan-STARRS in general, process-to-process communication
     2407will be defined through auto-coded APIs which support a limited and
     2408validated communication protocol.  The APIs will be coded based on a
     2409table which defines the allowed command set and the grammar to be
     2410used.  This mechanism will allow a single code block to define
     2411inter-process communication methods for many Pan-STARRS subsystems,
     2412including, within the IPP, the Scheduler-Controller communications.
     2413
    22712414\subsection{External Interfaces}
    22722415
    22732416This subsection describes the interfaces between the IPP and other
    22742417Pan-STARRS systems and the external clients.  The interfaces are
    2275 illustrated in Figure~\ref{overview}. 
     2418illustrated in Figure~\ref{fig:overview}. 
    22762419
    22772420\subsubsection{OTIS}
    … …  
    22942437\subsubsection{PSPS}
    22952438
    2296 The details of the transfer mechanism have \tbd{not been worked out}.
    2297 The data to be transfered include:
     2439Data will be sent to PSPS from the IPP as part of a daily or weekly
     2440analysis process on the Static Sky.  The data will be pushed from the
     2441IPP to PSPS when they are available.  The data to be transfered
     2442include:
    22982443\begin{itemize}
    2299 \item Static Sky images
    2300 \item Postage Stamps
    2301 \item Metadata tables
    2302 \item Detections \& Object associations.
     2444\item Static Sky images - to be transferred as FITS images or
     2445  FITS triangular image regions.
     2446\item Postage Stamps - to be transferred as FITS images.
     2447\item Metadata tables - to be transferred as FITS tables
     2448\item Detections \& Object associations - to be transferred as FITS tables.
    23032449\end{itemize}
    23042450
    23052451\subsubsection{MOPS}
    23062452
    2307 The details of the transfer mechanism have \tbd{not been worked out}.
    2308 The data to be transfered include:
     2453Data will be sent to MOPS from the IPP as part of the Phase 4
     2454analysis.  The data will be pushed from the IPP to MOPS when they are
     2455available.  The data to be transfered include:
    23092456\begin{itemize}
    2310 \item Image Metadata tables
    2311 \item Orphaned Detections
     2457\item Image Metadata tables - to be transferred as FITS tables
     2458\item Orphaned Detections - to be transferred as FITS tables
    23122459\end{itemize}
    23132460
    23142461\subsubsection{Other Preferred Client Science Pipelines}
    23152462
    2316 The details of the transfer mechanism have \tbd{not been worked out}.
     2463These cannot be completely defined until the Clients are defined and
     2464their requirements are specified.  The expectation is that the data
     2465products will be the same as for the MOPS.  The data will be pushed
     2466from the IPP to the Client Science Pipeline when they are available.
    23172467
    23182468%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    23192469
    23202470\section{Computer Hardware}
     2471\label{sec:Hardware}
    23212472
    23222473\subsection{PS-1 Cluster Design}
    … …  
    23332484support the Metadata DB and the AP DB. 
    23342485
    2335 The IPP PS-1 SRS (PSDC-xxx) specifies the processing throughput
    2336 requirements for the IPP.  We have performed benchmark tests of the
    2337 processing needs in order to achieve this throughput.  The details of
    2338 this study are presented in the IPP Hardware Analysis (PSDC-xxx),
    2339 which we summarize here.  The analysis measures the processing time
    2340 (excluding I/O) for both Phase 2 and Phase 4 on an Intel Pentium 4
    2341 processor, and expresses the processing time in GHz-seconds, under the
    2342 assumption that a machine with the same architecture and twice the
    2343 processor speed with perform the same analysis in half the time.  This
    2344 is probably a valid assumption within a limited range on hardware
    2345 using the same architecture.  We independently find that 32-bit Pentium
    2346 processors perform somewhat slower (up to a factor of 2) than
    2347 equivalently rated 64 bit Opeteron processors.  This discrepancy makes
    2348 our numbers somewhat conservative, but may only compensate for the
    2349 simplistic analysis we have performed. 
    2350 
    2351 Our benchmarks show that the Phase 2 analysis takes 12000 GHz-seconds
    2352 for a complete major frame (4 FPAs) while the Phase 4 analysis takes
    2353 7800 GHz-seconds for the same major frame.  We also examine the total
    2354 data I/O required for each processing node both locally to disk and
    2355 across the network to other machines.  These numbers in turn depend on
    2356 whether the data is optimally stored on the OTA nodes (raw images
    2357 matched to their calibration images) or if the data are randomized.
    2358 There are also differences in the analysis for how many bits and
    2359 images are used in the processing.  For PS-1, the `minimal' data set
    2360 is approrpiate, resulting in a total Phase 2 I/O of 21 GBs per major
    2361 frame and a total Phase 4 I/O of 36 GBs.  We will use the randomized
    2362 numbers as a conservative estimate, and assume the network is the
    2363 dominant I/O bottleneck.
    2364 
    2365 The analysis assumes each CPU is associated with one RAID array
    2366 (maximum throughput 110 MB/sec) and one network controller (maximum
    2367 throughput 70 MB/s) and that each one is a 2.2 GHz processor. In this
    2368 case, given the CPU load and I/O throughput above, the Phase 2 will
    2369 require a total of 190 seconds of I/O and 5500 seconds of processing
    2370 distributed across the cluster.  Likewise, the Phase 4 analysis will
    2371 require a total of 330 sec of I/O and 3500 seconds of processing.
    2372 Given the 160 seconds available per major frame, these numbers imply a
    2373 total of 63 processors are needed to keep up with the processing and
    2374 I/O load. 
    2375 
    2376 The other major driver on the IPP PS-1 cluster are the data storage
    2377 requirements.  We are required to store the entire AP Survey data and
    2378 the IVP data, and to have storage enough to represent the Static Sky
    2379 by the end of the two year mission.  These storage requirements as a
    2380 function of time are shown in Figure~\ref{StorageProfile}.  Based on
    2381 the PS-1 Design Reference Mission (PSDC-xxx), by the end of the
    2382 second year, we will have total storage needs of 850 TB for raw images
    2383 and the Static Sky, and an additional \tbd{XXX} TB for the AP DB
    2384 storage. 
    2385 
    2386 To meet these requirements, we have designed the IPP cluster to use
    2387 fat bricks which will be capable of holding 24 disks each.  Before
    2388 PS-1 goes on line, we will purchase enough disks to fill 1/3 of the
    2389 disk slots.  After 9 months (2006 Sept), we will purchase the next 1/3
    2390 of the disks, and the remaining disks 9 months after that (2007 June).
    2391 We have made conservative estimates of the available disk sizes at
    2392 these purchase dates (400 GB, 600 GB, and 900 GB), allowing us to
    2393 determine the number of computers needed to meet the storage
    2394 specification.  We will purchase 80 computers, with the storage
    2395 profile shown in the figure, ending at a total capacity (after
    2396 discounting volume for RAID overhead and binary vs digital terabytes)
    2397 of 950 TBs.  The 80 computers will easily meet the processing and I/O
    2398 requirements given the above need to 63 processors. 
    2399 
    2400 There are two competing trades we will also want to make.  First, we
    2401 will want to duplicate data to multiple machines in the network to
    2402 protect against catastrophic failures on a single machine.  This
    2403 double the total data space needed.  To compensate, however, we will
    2404 also employ compression to data, especially data which is older.
    2405 These two factors will tend to cancel each other, so we have ignored
    2406 both in out calculations above.
    2407 
    2408 \tbd{switch information}
     2486The IPP PS-1 SRS (PSDC-430-005) specifies the processing throughput
     2487requirements for the IPP.  Benchmark tests of the IPP processing
     2488algorithms have been used to drive the design needed to achieve the
     2489throughput requirements.  The details of this study are presented in
     2490the IPP Computational Challenge (PSDC-400-006), summarized here.  The
     2491analysis measures the processing time (excluding I/O) for both Phase 2
     2492and Phase 4 on an Intel Pentium 4 processor, and expresses the
     2493processing time in GHz-seconds, under the assumption that a machine
     2494with the same architecture and twice the processor speed will perform
     2495the same analysis in half the time.  This is probably a valid
     2496assumption within a limited range on hardware using the same
     2497architecture.  Independent tests show that 32-bit Pentium processors
     2498perform somewhat slower (up to a factor of 2) than equivalently rated
     249964 bit Opteron processors.  This discrepancy makes the measured
     2500numbers somewhat conservative, and compensates for the simplified
     2501analysis performed.  The benchmarks show that the Phase 2 analysis
     2502takes 12000 GHz-seconds for a complete major frame (4 FPAs) while the
     2503Phase 4 analysis takes 7800 GHz-seconds for the same major frame.
     2504
     2505The total data I/O required for each processing node, both locally to
     2506disk and across the network to other machines, has also been measured.
     2507These numbers in turn depend on whether the data is optimally stored
     2508on the OTA nodes (raw images matched to their calibration images) or
     2509if the data are randomized across the storage nodes.  There are also
     2510differences in the analysis for the number of bits per pixel and the
     2511number of calibration images used in the processing.  For PS-1, the
     2512`minimal' data set is appropriate, resulting in a total Phase 2 I/O of
     251321 GBs per major frame and a total Phase 4 I/O of 36 GBs.  The
     2514randomized numbers are used as a conservative estimate, under the
     2515assumption the network, not local disk access, is the dominant I/O
     2516bottleneck.
     2517
     2518The analysis assumes each CPU (rated at 2.2 GHz) is associated with
     2519one RAID array (maximum throughput 110 MB/sec) and one network
     2520controller (maximum throughput 70 MB/s). In this case, given the CPU
     2521load and I/O throughput above, Phase 2 will require a total of 190
     2522seconds of I/O and 5500 seconds of processing distributed across the
     2523cluster.  Likewise, the Phase 4 analysis will require a total of 330
     2524sec of I/O and 3500 seconds of processing.  Given the 160 seconds
     2525available per major frame, these numbers imply a total of 63
     2526processors are needed to keep up with the processing and I/O load.
     2527
     2528The other major driver on the IPP PS-1 cluster is the data storage
     2529requirements.  It is necessary to store the raw images from the entire
     2530AP Survey, the MOPS Verification Program (MVP) and the IPP
     2531Verification Program (IVP), and to have storage enough to represent
     2532the Static Sky by the end of the two year mission.  These storage
     2533requirements as a function of time are shown in
     2534Figure~\ref{fig:StorageProfile}.  Based on the PS-1 Design Reference
     2535Mission (PSDC-230-001), by the end of the second year, the total
     2536storage requirements for raw images and the Static Sky will be 850 TB,
     2537along with and an additional 55 TB needed for the AP DB storage
     2538
     2539To meet these requirements, the IPP cluster is designed to use fat
     2540bricks which will be capable of holding 24 disks each.  The 5U / 24
     2541disk rack mount computer cases are one of the highest density
     2542solutions currently available.  A 4U / 36 disk box is also available
     2543and will be considered.  The disk purchases will be staggered in three
     2544waves.  Before PS-1 goes on the sky, the first 1/3 of the disks (600
     2545disks total) will be purchased.  Since the lead time for disks is
     2546fairly short, the purchase will be made only when other portions of
     2547Pan-STARRS are clearly on a timeline to success.  After 9 months
     2548(tentatively 2006 September), the next 1/3 of the disks will
     2549purchased, and the remaining disks 9 months after that (tentatively
     25502007 June).  Using conservative estimates of the available disk sizes
     2551at these purchase dates (400 GB, 600 GB, and 900 GB), and allocating 1
     2552of 12 disks to the RAID and 10\% of the volume to file system and
     2553binary Gigabyte overheads, the disk purchases outlined above result in
     2554a total volume after the last purchase of 950 TB.  This meets the
     2555requirements with 10\% spare excess.  The disk volume profile is also
     2556shown in Figure~\ref{fig:StorageProfile} and shows that the disk space
     2557will be available in the time it is required.
     2558
     2559The total number of computers to be purchased is 80.  This provides
     2560the 1800 disk slots and more than enough processors to meet the
     2561processing requirements.  This also leaves 5 live spare machines.
     2562
     2563There are two details which are not included in the analysis above:
     2564compression and replication.  Compression of the older raw data will
     2565reduce the volume requirements by a factor of roughly two.  However,
     2566replication of the data across the network is necessary to ensure the
     2567data against catastrophic failures on a single machine.  Replication
     2568doubles the total data space needed.  These two factors will tend to
     2569cancel each other, and are ignored in the calculations above.
     2570
     2571The IPP PS-1 clusters will have the following allocations of computers
     2572from this cluster:
     2573\begin{itemize}
     2574\item Phase 2 Nodes: 32
     2575\item Phase 4 Nodes: 30
     2576\item AP Database: 10
     2577\item Metadata Database: 1
     2578\item Image Server Database: 1
     2579\item Controller /  Scheduler: 1
     2580\end{itemize}
     2581This distribution meets the projections for computational power for
     2582each of these data systems, and leaves 5 computers as live spares for
     2583redundancy.
    24092584
    24102585\subsection{PS-1 Cluster Expected Reliability}
    24112586
    2412 With 80 computers and 1920 disks, we must be cautious about component
    2413 failures and their impact on operations and data integrity.  There are
    2414 several factors which mitigate our exposure to hardware failures.
    2415 First, the use of RAID controllers and RAID-5 striping of the data
    2416 will protect the data on a single RAID set against the failure of a
    2417 single disk in the array.  Second, our plan to have duplication across
    2418 the cluster will protect us against catastrophic failures.  Finally,
    2419 the flexibility of the distributed computing plan makes it trivial to
    2420 handle the loss of individual machines as the system can automatically
    2421 redistribute the load across the cluster.
    2422 
    2423 The components which are most likely to fail in our experience are, in
    2424 order: hard drives, ram, power supplies, and other components.  The
    2425 hard drive failure rate is by far the dominant concern as it
    2426 potentially affects the data integrity. 
     2587With 80 computers and 1920 disks, component failures are inevitable.
     2588The cluster design and management must be chosen to minimize their
     2589impact on operations and data integrity.
     2590
     2591There are several factors which reduce the cluster's exposure to
     2592hardware failures.  First, the use of RAID controllers and RAID-5
     2593striping of the data will protect the data on a single RAID set
     2594against the failure of a single disk in the array.  Second,
     2595duplication of data across the cluster will protect against
     2596catastrophic failures of the array (loss of two disks, loss of the
     2597array controller card).  Finally, the flexibility of the distributed
     2598computing plan minimizes the impact the loss of individual machines
     2599has on operations by making changes in the data and processing
     2600assignments on the cluster a trivial matter.
     2601
     2602The components which are most likely to fail in the experience of our
     2603team are, in order: hard drives, RAID controllers, ram, power
     2604supplies, and other components.  The hard drive and RAID controller
     2605failure rates are by far the dominant concerns as they potentially
     2606affects the data integrity.
    24272607
    24282608Most sources (REFS: UCSD article, Samsung White Paper) currently imply
    24292609hard disk failure rates (MTBF) in the range 400,000 hours and 500,000
    2430 hours.  We take these as an upper limit, and instead adopt a
    2431 conservative value of 100,000 hours.  With 1920 disk, this MTBF
    2432 implies a failure of one disk every 2.2 days.  Since the disks are in
    2433 a RAID which reports the disk failures immediately and drops the array
    2434 into degraded mode, these failures will not have a huge impact on the
    2435 operations, and recovery time is only 10s of minutes.  This failure
    2436 rate implies that we should be checking for hard disk failures daily.
    2437 \tbd{is it necessary to catch failures at night or can the system run
    2438 with a degraded disk?}.  A catastrophic failure for the array would
    2439 require two of the 12 disks to fail before the first failed disk is
    2440 replaced.  If we assume that maintainence is poor and it is possible
    2441 for a disk to take 1 week to be replaced, we calculate a probability
    2442 of a catastrophe of 1.8\% each time a disk fails.  Combined with the
    2443 disk failure rate, we can expect a RAID catastrophe 6 times over the 2
    2444 year operation of PS-1.  We can use these numbers as a guideline for
    2445 our level of support needed to avoid these RAID failures.  Note that
    2446 these 6 failures should not cause loss of data since the data is
    2447 duplicated across the cluster, but they require over 1 day for
    2448 recovery (as the entire array must be replicated across the network).
    2449 
    2450 \subsection{PS-1 Cluster Support}
     2610hours.  These are used as an upper limit, with the more historically
     2611conservative value of 100,000 hours used instead.  With 1920 disk,
     2612this MTBF implies a failure of one disk every 2.2 days.  Since the
     2613disks are in a RAID which reports the disk failures immediately and
     2614drops the array into degraded mode, these failures will not have a
     2615huge impact on the operations, and recovery time is only 10s of
     2616minutes.  This failure rate implies that the maintenance plan must
     2617include checks for hard disk failures on a daily basis, and should
     2618make use of email notification and early warning information (ie,
     2619SMART messages). 
     2620
     2621A catastrophic failure for the array would require two of the 12 disks
     2622to fail before the first failed disk is replaced.  Assuming that
     2623maintenance is poor and it is possible for a disk to take 1 week to
     2624be replaced, the probability of a catastrophe is 1.8\% each time the
     2625first disk fails.  Combined with the disk failure rate, RAID
     2626catastrophes are expected 6 times over the 2 year operation of PS-1.
     2627These numbers can be used as a guideline for the level of support
     2628needed to avoid these RAID failures.  Note that these 6 failures
     2629should not cause loss of data since the data is duplicated across the
     2630cluster, but they require over 1 day for recovery (as the entire array
     2631must be replicated across the network).
     2632
     2633A detailed IPP computer cluster commissioning and maintenance plan is
     2634specified in the document `Pan-STARRS PS-1 IPP Cluster Support'
     2635(PSDC-430-014).
    24512636
    24522637\begin{figure}
    24532638\begin{center}
    24542639\resizebox{6in}{!}{\includegraphics[angle=-90]{pics/ps1_ipp_storage.ps}}
    2455 \caption{ \label{StorageProfile} Storage Profile}
     2640\caption{ \label{fig:StorageProfile} Storage Profile}
    24562641\end{center}
    24572642\end{figure}
    … …  
    24602645
    24612646\clearpage
    2462 
    2463 \section{Appendices}
    2464 
    2465 \subsection{Image Server Database Table Contents}
    2466 \label{ImageServerTableContents}
    2467 
    2468 Tables~\ref{ImageServerTables:SO} - \ref{ImageServerTables:VOL} list
     2647\appendix
     2648\section{Image Server Database Table Contents}
     2649\label{sec:ImageServerTableContents}
     2650
     2651Tables~\ref{tab:ImageServerTables:SO} - \ref{tab:ImageServerTables:VOL} list
    24692652the basic contents of the Image Server database tables. 
    24702653
    24712654\begin{table}[bh]
    24722655\begin{center}
    2473 \caption{Storage Object Table Contents\label{ImageServerTables:SO}}
     2656\caption{Storage Object Table Contents\label{tab:ImageServerTables:SO}}
    24742657\begin{tabular}{lll}
    24752658\hline
    … …  
    24882671\begin{table}[bh]
    24892672\begin{center}
    2490 \caption{Instance Table Contents\label{ImageServerTables:INT}}
     2673\caption{Instance Table Contents\label{tab:ImageServerTables:INT}}
    24912674\begin{tabular}{lll}
    24922675\hline
    … …  
    25082691\begin{table}[bh]
    25092692\begin{center}
    2510 \caption{Volume Table Contents\label{ImageServerTables:VOL}}
     2693\caption{Volume Table Contents\label{tab:ImageServerTables:VOL}}
    25112694\begin{tabular}{lll}
    25122695\hline
    … …  
    25152698\hline
    25162699\code{vol_id}     & integer        & internal volume identifier \\
    2517 \code{uri}        & string         & node name? \\
     2700\code{uri}        & string         & node name \\
    25182701\hline
    25192702\end{tabular}
    … …  
    25222705\clearpage
    25232706
    2524 \subsection{Metadata Database Table Contents}
    2525 \label{MetadataTableContents}
    2526 
    2527 Tables~\ref{WeatherTable} -- \ref{overlaps} list the basic contents of
    2528 each of the Metadata Database tables listed in Section~\ref{Metadata}.
     2707\section{Metadata Database Table Contents}
     2708\label{sec:MetadataTableContents}
     2709
     2710Tables~\ref{tab:WeatherTable} -- \ref{tab:overlaps} list the basic contents of
     2711each of the Metadata Database tables listed in Section~\ref{sec:Metadata}.
    25292712
    25302713\begin{table}[bh]
    25312714\begin{center}
    2532 \caption{Weather Table: some sample weather points\label{WeatherTable}}
     2715\caption{Weather Table: some sample weather points\label{tab:WeatherTable}}
    25332716\begin{tabular}{lll}
    25342717\hline
    … …  
    25492732\begin{table}[bh]
    25502733\begin{center}
    2551 \caption{SkyProbe Transparency Table (sample entries)\label{SkyprobeBVTable}}
     2734\caption{SkyProbe Transparency Table (sample entries)\label{tab:SkyprobeBVTable}}
    25522735\begin{tabular}{lll}
    25532736\hline
    … …  
    25692752\begin{table}[bh]
    25702753\begin{center}
    2571 \caption{Skyprobe Line Absorption Table (sample entries)\label{SkyprobeATable}}
     2754\caption{Skyprobe Line Absorption Table (sample entries)\label{tab:SkyprobeATable}}
    25722755\begin{tabular}{lll}
    25732756\hline
    … …  
    25922775\begin{table}[bh]
    25932776\begin{center}
    2594 \caption{Skyprobe Line Emission Table (sample entries)\label{SkyprobeETable}}
     2777\caption{Skyprobe Line Emission Table (sample entries)\label{tab:SkyprobeETable}}
    25952778\begin{tabular}{lll}
    25962779\hline
    … …  
    26132796\begin{table}[bh]
    26142797\begin{center}
    2615 \caption{DIMM Measurements Table\label{DimmTable}}
     2798\caption{DIMM Measurements Table\label{tab:DimmTable}}
    26162799\begin{tabular}{lll}
    26172800\hline
    … …  
    26342817\begin{table}[bh]
    26352818\begin{center}
    2636 \caption{Near IR Wide-field Camera Results Table\label{NIR-Table}}
     2819\caption{Near IR Wide-field Camera Results Table\label{tab:NIR-Table}}
    26372820\begin{tabular}{lll}
    26382821\hline
    … …  
    26532836\begin{table}[bh]
    26542837\begin{center}
    2655 \caption{Dome Status Table\label{DomeStatusTable}}
     2838\caption{Dome Status Table\label{tab:DomeStatusTable}}
    26562839\begin{tabular}{lll}
    26572840\hline
    … …  
    26712854\begin{table}[bh]
    26722855\begin{center}
    2673 \caption{Telescope Status\label{TelescopeStatusTable}}
     2856\caption{Telescope Status\label{tab:TelescopeStatusTable}}
    26742857\begin{tabular}{lll}
    26752858\hline
    … …  
    26902873\begin{table}[bh]
    26912874\begin{center}
    2692 \caption{Raw FPA Images\label{RawFPAs}}
     2875\caption{Raw FPA Images\label{tab:RawFPAs}}
    26932876\begin{tabular}{lll}
    26942877\hline
    … …  
    27202903\begin{table}[bh]
    27212904\begin{center}
    2722 \caption{Pending Science Chips\label{PendingChips}}
     2905\caption{Pending Science Chips\label{tab:PendingChips}}
    27232906\begin{tabular}{lll}
    27242907\hline
    … …  
    27362919\begin{table}[bh]
    27372920\begin{center}
    2738 \caption{Processed Science Chips\label{ProcessedChips}}
     2921\caption{Processed Science Chips\label{tab:ProcessedChips}}
    27392922\begin{tabular}{lll}
    27402923\hline
    … …  
    27532936\begin{table}[bh]
    27542937\begin{center}
    2755 \caption{Observation Group Information\label{OBS}}
     2938\caption{Observation Group Information\label{tab:OBSGroup}}
    27562939\begin{tabular}{lll}
    27572940\hline
    … …  
    27632946Type             & string          & Type of observation. \\
    27642947Status           & string          & Status of the observation group. \\
    2765 \tbd{etc} & \\
     2948etc & \\
    27662949\hline
    27672950\end{tabular}
    … …  
    27712954\begin{table}[bh]
    27722955\begin{center}
    2773 \caption{Observation Frame Information\label{OBS}}
     2956\caption{Observation Frame Information\label{tab:OBSFrame}}
    27742957\begin{tabular}{lll}
    27752958\hline
    … …  
    27812964Type             & string          & Type of observation. \\
    27822965Status           & string          & Status of the observation group. \\
    2783 \tbd{etc} & \\
     2966etc & \\
    27842967\hline
    27852968\end{tabular}
    … …  
    27892972\begin{table}[bh]
    27902973\begin{center}
    2791 \caption{Science Processing Stats\label{PSStats}}
     2974\caption{Science Processing Stats\label{tab:PSStats}}
    27922975\begin{tabular}{lll}
    27932976\hline
    … …  
    28273010\begin{table}[bh]
    28283011\begin{center}
    2829 \caption{Chip / Sky overlaps\label{overlaps}}
     3012\caption{Chip / Sky overlaps\label{tab:overlaps}}
    28303013\begin{tabular}{lll}
    28313014\hline
    … …  
    28433026\begin{table}[bh]
    28443027\begin{center}
    2845 \caption{Processed Sky-Cell stats\label{ProcessedSky}}
     3028\caption{Processed Sky-Cell stats\label{tab:ProcessedSky}}
    28463029\begin{tabular}{lll}
    28473030\hline
    … …  
    28503033\hline
    28513034Input Chips        & string        & Identification numbers of the chips used to produce the sky cell. \\
    2852 PSF adjustments    & string        & \tbd{Adjustments to the PSF.} \\
     3035PSF adjustments    & string        & Adjustments to the PSF. \\
    28533036CR rejection stats & string        & Statistics from the CR rejection (number of CRs, distribution, limiting flux). \\
    28543037Image comb params  & string        & Parameters used for the image combination. \\
    … …  
    28653048\clearpage
    28663049
    2867 \subsection{AP Database Table Contents}
    2868 \label{APDBTableContents}
    2869 
    2870 \tbd{Table contents to be defined}
     3050\section{AP Database Table Contents}
     3051\label{sec:APDBTableContents}
     3052
     3053\begin{table}[bh]
     3054\begin{center}
     3055\caption{Images\label{tab:images}}
     3056\begin{tabular}{lll}
     3057\hline
     3058\hline
     3059{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3060\hline
     3061Image ID          & & \\
     3062time/date         & & \\
     3063Exposure Time     & & \\
     3064Nstars            & & \\
     3065NX                & & \\
     3066NY                & & \\
     3067photcode          & & \\
     3068Mcal              & & \\
     3069Mcal error        & & \\
     3070Mcal chisq        & & \\
     3071Airmass           & & \\
     3072Astrometry        & & \\
     3073PSF               & & \\
     3074flags             & & \\
     3075Camera            & & \\
     3076\hline           
     3077\end{tabular}     
     3078\end{center}     
     3079\end{table}       
     3080                 
     3081\begin{table}[bh]
     3082\begin{center}
     3083\caption{Image Overlaps\label{tab:ImageOverlaps}}
     3084\begin{tabular}{lll}
     3085\hline
     3086\hline
     3087{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3088\hline
     3089Image ID          & & \\
     3090Region Table      & & \\
     3091\hline
     3092\end{tabular}
     3093\end{center}
     3094\end{table}
     3095
     3096\begin{table}[bh]
     3097\begin{center}
     3098\caption{Objects\label{tab:Objects}}
     3099\begin{tabular}{lll}
     3100\hline
     3101\hline
     3102{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3103\hline
     3104ID                & & \\
     3105$\alpha$          & & \\
     3106$\delta$          & & \\
     3107$\mu_{\alpha}$    & & \\
     3108$\mu_{\delta}$    & & \\
     3109$\sigma_{\alpha}$ & & \\
     3110$\sigma_{\delta}$ & & \\
     3111$\chi^2$ position & & \\
     3112$N_{\rm det}$     & & \\
     3113$N_{\rm miss}$    & & \\
     3114flags             & & \\
     3115\hline           
     3116\end{tabular}
     3117\end{center}
     3118\end{table}
     3119
     3120\begin{table}[bh]
     3121\begin{center}
     3122\caption{Average Magnitudes\label{tab:AveMags}}
     3123\begin{tabular}{lll}
     3124\hline
     3125\hline
     3126{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3127\hline
     3128object ID         & & \\
     3129$M_{\rm int}$     & & \\
     3130$M_{\rm ext}$     & & \\
     3131$\chi^2_{\rm mag}$& & \\
     3132$\sigma_{\rm mag}$& & \\
     3133photcode          & & \\
     3134\hline
     3135\end{tabular}
     3136\end{center}
     3137\end{table}
     3138
     3139\begin{table}[bh]
     3140\begin{center}
     3141\caption{Solar System Objects\label{tab:SSObjs}}
     3142\begin{tabular}{lll}
     3143\hline
     3144\hline
     3145{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3146\hline
     3147SSO ID            & & \\
     3148$N_{\rm det}$     & & \\
     3149\hline
     3150\end{tabular}
     3151\end{center}
     3152\end{table}
     3153
     3154\begin{table}[bh]
     3155\begin{center}
     3156\caption{Matched Detections\label{tab:Detections}}
     3157\begin{tabular}{lll}
     3158\hline
     3159\hline
     3160{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3161\hline
     3162$\alpha$          & & \\
     3163$\delta$          & & \\
     3164$\sigma_{\alpha}$ & & \\
     3165$\sigma_{\delta}$ & & \\
     3166$M_{\rm inst}$    & & \\
     3167$M_{\rm cal}$     & & \\
     3168$\sigma_{\rm mag}$& & \\
     3169photcode          & & \\
     3170type              & & \\
     3171flags             & & \\
     3172time/date         & & \\
     3173airmass           & & \\
     3174$\sigma_{x}$      & & \\
     3175$\sigma_{y}$      & & \\
     3176$\theta$          & & \\
     3177object ID         & & \\
     3178exptime           & & \\
     3179sky               & & \\
     3180$\sigma_{\rm sky}$& & \\
     3181etc               & & \\
     3182\hline
     3183\end{tabular}
     3184\end{center}
     3185\end{table}
     3186
     3187\begin{table}[bh]
     3188\begin{center}
     3189\caption{Orphaned Detections\label{tab:Orphans}}
     3190\begin{tabular}{lll}
     3191\hline
     3192\hline
     3193{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3194\hline
     3195$\alpha$          & & \\
     3196$\delta$          & & \\
     3197$\sigma_{\alpha}$ & & \\
     3198$\sigma_{\delta}$ & & \\
     3199$M_{\rm inst}$    & & \\
     3200$M_{\rm cal}$     & & \\
     3201$\sigma_{\rm mag}$& & \\
     3202photcode          & & \\
     3203type              & & \\
     3204flags             & & \\
     3205time/date         & & \\
     3206airmass           & & \\
     3207$\sigma_{x}$      & & \\
     3208$\sigma_{y}$      & & \\
     3209$\theta$          & & \\
     3210exptime           & & \\
     3211sky               & & \\
     3212$\sigma_{\rm sky}$& & \\
     3213etc               & & \\
     3214\hline           
     3215\end{tabular}
     3216\end{center}
     3217\end{table}
     3218
     3219\begin{table}[bh]
     3220\begin{center}
     3221\caption{Non-detections\label{tab:NonDetects}}
     3222\begin{tabular}{lll}
     3223\hline
     3224\hline
     3225{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3226\hline 
     3227object ID          & & \\
     3228$N_{\rm non-det}$          & & \\
     3229last time/date     & & \\
     3230last mag           & & \\
     3231faintest time/date & & \\
     3232faintest mag       & & \\
     3233\hline
     3234\end{tabular}
     3235\end{center}
     3236\end{table}
     3237
     3238\begin{table}[bh]
     3239\begin{center}
     3240\caption{Regions\label{tab:Regions}}
     3241\begin{tabular}{lll}
     3242\hline
     3243\hline
     3244{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3245\hline
     3246$\alpha_0$        & & \\
     3247$\alpha_1$        & & \\
     3248$\delta_0$        & & \\
     3249$\delta_1$        & & \\
     3250Region ID         & & \\
     3251Parent ID         & & \\
     3252Nchildren         & & \\
     3253Images            & & \\
     3254Objects           & & \\
     3255Detections        & & \\
     3256\hline
     3257\end{tabular}
     3258\end{center}
     3259\end{table}
     3260
     3261\begin{table}[bh]
     3262\begin{center}
     3263\caption{Filters\label{tab:Filters}}
     3264\begin{tabular}{lll}
     3265\hline
     3266\hline
     3267{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3268\hline
     3269Filter ID         & & \\
     3270Filter name       & & \\
     3271Photcode          & & \\
     3272$\lambda_0$       & & \\
     3273$\delta_\lambda$  & & \\
     3274$\epsilon$        & & \\
     3275transmission curve& & \\
     3276time/date         & & \\
     3277\hline           
     3278\end{tabular}     
     3279\end{center}
     3280\end{table}
     3281
     3282\begin{table}[bh]
     3283\begin{center}
     3284\caption{Photcodes\label{tab:Photcodes}}
     3285\begin{tabular}{lll}
     3286\hline
     3287\hline
     3288{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3289\hline
     3290Photcode          & & \\
     3291Telescope         & & \\
     3292Camera            & & \\
     3293Detector          & & \\
     3294Filter            & & \\
     3295Nominal ZP        & & \\
     3296airmass terms     & & \\
     3297color terms       & & \\
     3298Target            & & \\
     3299\hline
     3300\end{tabular}
     3301\end{center}
     3302\end{table}
     3303
     3304\begin{table}[bh]
     3305\begin{center}
     3306\caption{Zero Point History\label{tab:Zpts}}
     3307\begin{tabular}{lll}
     3308\hline
     3309\hline
     3310{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3311\hline
     3312Photcode          & & \\
     3313start Time/date   & & \\
     3314end Time/date     & & \\
     3315Zero Points       & & \\
     3316airmass           & & \\
     3317color             & & \\
     3318error             & & \\
     3319N measurements    & & \\
     3320N stars           & & \\
     3321photom ref set    & & \\
     3322\hline
     3323\end{tabular}
     3324\end{center}
     3325\end{table}
     3326
     3327\begin{table}[bh]
     3328\begin{center}
     3329\caption{Distortion History\label{tab:Distortions}}
     3330\begin{tabular}{lll}
     3331\hline
     3332\hline
     3333{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3334\hline
     3335Camera            & & \\
     3336Telescope         & & \\
     3337distortion terms  & & \\
     3338time/date         & & \\
     3339residuals / error & & \\
     3340N stars           & & \\
     3341N images          & & \\
     3342astrom ref set    & & \\
     3343\hline           
     3344\end{tabular}
     3345\end{center}
     3346\end{table}
     3347
     3348\begin{table}[bh]
     3349\begin{center}
     3350\caption{Database Hosts\label{tab:APDBHosts}}
     3351\begin{tabular}{lll}
     3352\hline
     3353\hline
     3354{\bf Column Name} & {\bf Datatype } & {\bf Description} \\
     3355\hline
     3356machine name      & & \\
     3357machine ID        & & \\
     3358\hline
     3359\end{tabular}
     3360\end{center}
     3361\end{table}
    28713362
    28723363%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    28733364
    2874 \subsection{Software Runtime Configuration Issues}
    2875 \label{RuntimeConfig}
     3365\section{Software Runtime Configuration Issues}
     3366\label{sec:RuntimeConfig}
    28763367
    28773368The IPP Software requires extensive runtime configuration information.
    … …  
    28813372Metadata Database or in configuration files available to the user.
    28823373Both methods are implemented in the current design.  In either method,
    2883 the necessary parameters are identical.  In this section, we discuss
    2884 the contents of specific portions of the runtime configuration.
    2885 
    2886 \subsubsection{Camera Definition Information}
     3374the necessary parameters are identical.  This section discusses the
     3375contents of specific portions of the runtime configuration.
     3376
     3377\subsection{Camera Definition Information}
    28873378
    28883379Every camera which may be analysed by the IPP has differences in how
    … …  
    29103401keywords for the same information at different times (major readout
    29113402software upgrades, for example, can be accompanied by keyword
    2912 revisions).  In addition, within Pan-STARRS and the IPP, we would like
    2913 the capability to refer to the Metadata database as the authoratative
    2914 sources of some of these entries rather than the image headers.  Given
    2915 this circumstance, it is at least necessary to define the appropriate
    2916 source for a given data concept appropriate to data from a specific
    2917 camera. 
     3403revisions).  In addition, within Pan-STARRS and the IPP, it is
     3404necessary to have the capability to refer to the Metadata database as
     3405the authoratative sources of some of these entries rather than the
     3406image headers.  Given this circumstance, it is at least necessary to
     3407define the appropriate source for a given data concept appropriate to
     3408data from a specific camera.
    29183409
    29193410The second problem arises when actually performing an analysis.  In
    … …  
    29333424In order to facilitate the operation of the IPP with a variety of
    29343425cameras, and to allow the software the flexibility to change the
    2935 camera defintion dynamically, we define a collection of software
    2936 runtime configuration information which defines a given camera.  This
    2937 information is represented below in the form of the PSLib Metadata
    2938 Config file, but may be stored in the Metadata Database or in an
    2939 alternate format as appropriate.   
    2940 
    2941 We start by noting that the a single camera is represented as a Focal
    2942 Plane Array (FPA), divided into Chips, divided into Cells.  For a
    2943 single FPA, all imaging data is stored in a FITS file or a collection
    2944 of FITS files.  Software needs to know where in a given file or set of
    2945 files to find a particular Cell, what Cells to expect, what chips to
    2946 expect, and the relationships between those entities, etc. 
     3426camera defintion dynamically, the IPP includes a collection of
     3427software runtime configuration information which defines a given
     3428camera.  This information is represented below in the form of the
     3429PSLib Metadata Config file, but may be stored in the Metadata Database
     3430or in an alternate format as appropriate.
     3431
     3432The a single camera is represented as a Focal Plane Array (FPA),
     3433divided into Chips, divided into Cells.  For a single FPA, all imaging
     3434data is stored in a FITS file or a collection of FITS files.  Software
     3435needs to know where in a given file or set of files to find a
     3436particular Cell, what Cells to expect, what chips to expect, and the
     3437relationships between those entities, etc.
    29473438
    29483439A single camera configuration file (or dataset) represents the
    … …  
    29553446NCELL       S32    NN
    29563447NCHIP       S32    NN
    2957 EXPTIME-SRC STR    HD:EXPTIME # need to specify PHU vs EXTNAME?
     3448EXPTIME-SRC STR    HD:EXPTIME # need to specify PHU vs EXTNAME
    29583449EXPTIME-KEY STR    EXPTIME 
    29593450DATE-KEY    STR    DATE-OBS
    … …  
    29653456\end{verbatim}
    29663457
    2967 \subsubsection{Analysis Recipe Information}
     3458\subsection{Analysis Recipe Information}
    29683459
    29693460In order to maintain flexibility in the analysis details, the IPP uses
    … …  
    29753466these may specify a specific value, or they may specify lookup methods
    29763467(database locations, or header locations).  The specifies of each
    2977 depends on the context.  Below, we provide an example recipe file for
    2978 the bias subtraction portion of Phase 2, giving several alternative
    2979 options for certain entries.  Note that, for example, the overscan
    2980 subtraction may be specified as using a particular region given in the
    2981 recipe file, or on the basis of a particular header keyword.
     3468depends on the context.  Below is an example recipe file for the bias
     3469subtraction portion of Phase 2, giving several alternative options for
     3470certain entries.  Note that, for example, the overscan subtraction may
     3471be specified as using a particular region given in the recipe file, or
     3472on the basis of a particular header keyword.
    29823473
    29833474\begin{verbatim}
    … …  
    30033494\end{verbatim}
    30043495
    3005 \subsection{I/O Code Autogeneration}
    3006 
    3007 Within IPP, we have a number of data collections which have multiple
    3008 representations.  We define a tool to automatically generate code to
    3009 provide I/O APIs to read and write these data and data structures to
    3010 carry them within program.  Within the IPP, we will use database
    3011 tables (ie, in the Metadata Database), FITS Tables (to exchange bulk
    3012 data), and XML (to exchange more complete datasets). 
     3496\section{I/O Code Autogeneration}
     3497\label{sec:AutocodeIO}
     3498
     3499The IPP includes a number of data collections which have multiple
     3500representations.  A software tool will be used to automatically
     3501generate code to provide I/O APIs to read and write these data and to
     3502define the data structures used to carry them within a program.
     3503Within the IPP, examples of these different data entities include
     3504database tables (ie, in the Metadata Database), FITS Tables (to
     3505exchange bulk data), and XML (to exchange more complete datasets).
    30133506
    30143507I/O API Autocode template (example.def):
    … …  
    30463539\end{verbatim}
    30473540
    3048 \bibliographystyle{plain}
    3049 \bibliography{panstarrs}
     3541%\bibliographystyle{plain}
     3542%\bibliography{panstarrs}
     3543
     3544\input{glossary.tex}
    30503545
    30513546\end{document}

  • trunk/doc/design/ippSRS.tex

    r2241 r2544  
    1  %%% $Id: ippSRS.tex,v 1.12 2004-10-29 22:00:08 eugene Exp $
     1 %%% $Id: ippSRS.tex,v 1.13 2004-11-30 23:16:03 eugene Exp $
    22\documentclass[panstarrs,spec]{panstarrs}
    33
    44% basic document variables
    5 \title{Pan-STARRS Image Processing Pipeline}
     5\title{Pan-STARRS PS-1 Image Processing Pipeline}
    66\subtitle{Software Requirements Specification}
    77\shorttitle{IPP SRS}
    … …  
    1111\project{Pan-STARRS Image Processing Pipeline}
    1212\organization{Institute for Astronomy}
    13 \version{DR}
     13\version{01}
    1414\docnumber{PSDC-430-005}
    1515
    … …  
    3434\RevisionsStart
    3535% version     Date         Description
    36 DR.01 & 2004.01.01 & First draft  \\ \hline
    37 DR.02 & 2004.03.10 & Second draft \\ \hline
    38 DR.03 & 2004.04.13 & Most paragraphs fleshed out \\ \hline
    39 DR.04 & 2004.04.27 & Basic text frozen for internal review \\ \hline
    40 DR.05 & 2004.05.24 & Incorporating comments from internal review \\ \hline
    41 DR.06 & 2004.08.06 & Revisions in prep of SRR \\ \hline
    42 DR.06 & 2004.10.22 & Revisions based on SRR \\ \hline
     36% DR.01 & 2004.01.01 & First draft  \\ \hline
     37% DR.02 & 2004.03.10 & Second draft \\ \hline
     38% DR.03 & 2004.04.13 & Most paragraphs fleshed out \\ \hline
     39% DR.04 & 2004.04.27 & Basic text frozen for internal review \\ \hline
     40% DR.05 & 2004.05.24 & Incorporating comments from internal review \\ \hline
     4100      & 2004.08.06 & Revisions in prep of SRR \\ \hline
     4201      & 2004.10.29 & Revisions based on SRR \\ \hline
    4343\RevisionsEnd
    4444
    … …  
    121121that series is implied. 
    122122
    123 Open issues (TBDs) in this document are marked \tbd{in bold red}.
    124 
    125 Quantities which should be reviewed (TBRs) are marked \tbr{in bold
    126 blue}.
     123Open issues (TBDs) in this document are marked {\bf \color{red} in
     124bold red}.
     125
     126Quantities which should be reviewed (TBRs) are marked {\bf
     127\color{blue} in bold blue}.
    127128
    128129\subsubsection{``Shall''}  When used in this specification, the word
    … …  
    141142
    142143\DocumentsInternalSection
    143 PSDC-130-001  &   PS-1 Design Reference Mission \\ \hline
    144 PSDC-130-xxx  &   PS-1 SCD \\ \hline
    145 PSDC-430-004  &   Pan-STARRS IPP C Code Conventions \\ \hline
    146 PSDC-430-006  &   Pan-STARRS IPP ADD \\ \hline
    147 PSDC-430-006  &   Pan-STARRS IPP SDRS \\ \hline
    148 PSDC-430-007  &   Pan-STARRS IPP PSLib SDRS \\ \hline
     144PSDC-230-001  &   PS-1 Design Reference Mission \\ \hline
     145PSDC-230-002  &   PS-1 System Concept Definition \\ \hline
     146PSDC-400-006  &   The Pan-STARRS IPP Computational Challenge \\ \hline
     147PSDC-430-004  &   Pan-STARRS PS-1 IPP C Code Conventions \\ \hline
     148PSDC-430-006  &   Pan-STARRS PS-1 IPP Algorithm Design Document \\ \hline
     149PSDC-430-007  &   Pan-STARRS PS-1 IPP PSLib Supplementary Design Requirements Specification \\ \hline
     150PSDC-430-010  &   Pan-STARRS PS-1 IPP Perl Code Conventions \\ \hline
     151PSDC-430-011  &   Pan-STARRS PS-1 IPP System/Subsystem Design Description \\ \hline
     152PSDC-430-012  &   Pan-STARRS PS-1 IPP Modules Supplementary Design Requirements Specification \\ \hline
    149153\DocumentsExternalSection
    150154Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\
    … …  
    179183The Pan-STARRS System Concept Definition (SCD) specifies the derived
    180184top-level requirements for the IPP, which we reproduce here (with
    181 numbering consistent with this document):
     185numbering consistent within this document):
    182186
    183187\begin{enumerate}
    … …  
    261265  \label{TLR:15}
    262266
    263 \item The IPP shall degrade the stacked image by no more than \tbr{10
    264   milliarcseconds (FWHM added in quadrature)} over the theoretical
     267\item The IPP shall degrade the stacked image by no more than 150
     268  milliarcseconds (FWHM added in quadrature) over the theoretical
    265269  limit for the stack under infinite
    266270  sampling.\VER{ANALYSIS}{SCD:3.5.2}
    … …  
    269273\item The IPP shall perform the processing of science images to the
    270274  level of transient detection and static sky inclusion at a rate such
    271   that exposures taken at an \tbr{average cadence of 40 seconds} do
    272   not accumulate in the processing buffer (average throughput
     275  that exposures taken at an average cadence of 40 seconds do not
     276  accumulate in the processing buffer (average throughput
    273277  requirement).\VER{TEST}{SCD:3.2.2.3}
    274278  \label{TLR:17}
    … …  
    281285
    282286\item The IPP shall perform transient detection to a completeness of
    283   99\% at the completeness for transient detections with a significant
     287  99\% at the completeness for transient detections with a significance
    284288  $> 5\sigma$.\VER{ANALYSIS}{SCD:xxx}
    285289
    … …  
    300304  \label{TLR:21}
    301305
    302 \item The IPP shall provide access to preferred Pan-STARRS science clients to the
    303   detected transient objects within \tbr{5 minutes}.\VER{TEST}{SCD:3.5.10}
     306\item The IPP shall provide access to preferred Pan-STARRS science
     307  clients to the detected transient objects within 15 minutes with at
     308  least 85\% reliability.\VER{TEST}{SCD:3.5.10}
    304309  \label{TLR:22}
    305310
    … …  
    375380\item Because the delivered code is required to run on UNIX machines, the delivered code shall be in compliance with the language-independent UNIX operating system standard POSIX (Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2004).\VER{INSPECT}{allocated}
    376381\item Source code files shall use the UNIX line-break convention (line-feed only).  \VER{INSPECT}{allocated}
    377 \item C coding style shall adhere to the standard defined in the document `Pan-STARRS C-coding standard' (PSDC-430-004).  \VER{INSPECT}{allocated}
    378 \item Perl coding shall follow the standard defined in the document \tbd{`Pan-STARRS Perl-coding standard' (PSDC-430-0XX)}.\VER{INSPECT}{allocated}
     382\item C coding style shall adhere to the standard defined in the document `Pan-STARRS IPP C-coding standard' (PSDC-430-004).  \VER{INSPECT}{allocated}
     383\item Perl coding shall follow the standard defined in the document `Pan-STARRS IPP Perl-coding standard' (PSDC-430-010).\VER{INSPECT}{allocated}
    379384\end{enumerate}
    380385
    … …  
    501506\subsubsection{Software Configuration}
    502507
    503 \paragraph{Version Management}
    504 
    505 The IPP software configuration management system shall ensure that
    506 validated versions of both internal and external software are used
    507 when the software is compiled.\VER{TEST}{allocated}
    508 
    509 \paragraph{Optional Modes}
    510 
    511 The IPP software configuration management system shall provide
    512 optionally selected software version sets under compilation
    513 conditions.  For example, compilation of the software for test
    514 purposes with a non-standard FFT tool shall be an
    515 option.\VER{TEST}{allocated}
     508The IPP software configuration management system shall follow the
     509processes outlined by the Pan-STARRS IPP Software Configuration
     510Management Place (PSDC-430-003).\VER{INSPECT}{allocated}
    516511
    517512\subsection{Architectural Components}
    … …  
    525520
    526521As discussed in the Pan-STARRS System Concept Definition
    527 (PSDC-250-002), the IPP is organized into a number of clearly-defined
    528 software elements.  The SCD provides a detailed description of the
    529 roles and responsibilities of these subsystems.  In brief, the IPP
    530 consists of: a collection of science analysis programs which perform
    531 the stages of the data analysis; a set of architectural components
    532 which provide the infrastructure needed to run the analysis programs;
    533 and a collection of hardware on which all of the software elements
    534 exist and operate.
     522(PSDC-230-002), the IPP is organized into a number of clearly-defined
     523software elements.  The SCD provides a detailed description of these
     524subsystems.  In brief, the IPP consists of: a collection of science
     525analysis programs which perform the stages of the data analysis; a set
     526of architectural components which provide the infrastructure needed to
     527run the analysis programs; and a collection of hardware on which all
     528of the software elements exist and operate.
    535529
    536530The architectural components consist of:
    … …  
    546540 it is no longer needed by other portions of the IPP.
    547541
     542\item {\bf IPP Metadata Database:} This component is used to store all
     543 other data which are neither image files nor astronomical object
     544 data.  The Metadata Database is the authoritative source for all
     545 metadata data, including metadata which may be duplicated elsewhere,
     546 such as in the headers of images in the image database.
     547
    548548\item {\bf Astrometry \& Photometry Database (AP):} This component is
    549549 used to store and manipulate astronomical objects detected in images
    … …  
    554554 needed to interpret the object data.
    555555
    556 \item {\bf IPP Metadata Database:} This component is used to store all
    557  other data which are neither image files nor astronomical object
    558  data.  The Metadata Database is the authoritative source for all
    559  metadata data, including metadata which may be duplicated elsewhere,
    560  such as in the headers of images in the image database.
    561 
    562556\item {\bf IPP Controller:} In order to perform the analysis stages
    563557 required by the IPP, it is necessary to use distributed computing
    … …  
    588582\begin{enumerate}
    589583\item The IPP Image Server shall accept raw images from the summit at
    590  a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds.}
     584 a sustained rate of 1 exposure (2~GB) per 40 seconds.
    591585 \VER{TEST}{TLR:17, TLR:23}
    592586
    … …  
    597591  reference to the specified image.\VER{TEST}{allocated}
    598592
    599 \item The IPP Image Server shall provide a total data capacity of 300
    600   TB after the first year of PS-1 operations and 900 TB after the
     593\item The IPP Image Server shall provide a total data capacity of 400
     594  TB after the first year of PS-1 operations and 750 TB after the
    601595  second year of operations.\VER{INSPECT}{}
    602596
    … …  
    607601\end{enumerate}
    608602
    609 \subsubsection{AP Database}
    610 
    611 %%% Table: AP DB parameters
    612 \begin{table}
    613 \begin{center}
    614 \caption{AP Detection Classes \& Object Parameters\label{APdetections}}
    615 \begin{tabular}{lrrrr}
    616 \hline
    617 \hline
    618 Object Parameter & P2 & P4$\Sigma$ & P4$\Delta$ & SS \\
    619 \hline
    620 PSF x,y, covar, $\alpha,\delta$               & + & + & + & + \\
    621 PSF mag, $\sigma_{\rm mag}$                   & + & + & + & + \\
    622 star/gal sep                                  & + & + & + & + \\
    623 $\sigma_x$, $\sigma_y$, $\theta$              & + & + & + & + \\
    624 local sky data                                & + & + & + & + \\
    625 Petrosian R, M, $R_{50}$, $R_{90}$            & - & + & - & + \\
    626 S\'ersic R, M, AB, $\phi$, $\nu$              & - & + & - & + \\
    627 W.L. $\gamma_1$, $\gamma_2$, pol. terms       & - & - & - & + \\
    628 exp. spaced aps., Poisson noise, variance     & - & - & - & + \\
    629 \hline
    630 \end{tabular}
    631 \end{center}
    632 \end{table}
    633 
    634 %%% Table: AP DB Throughput
    635 \begin{table}
    636 \begin{center}
    637 \caption{AP Data Volume and Throughput Requirements\label{APrates}}
    638 \begin{tabular}{lrrr}
    639 \hline
    640 \hline
    641 Quantity                    & P2                & P4$\Sigma$        & P4$\Delta$        \\
    642 \hline
    643 detection limit             & $20 \sigma$       & $5 \sigma$        & $3 \sigma$        \\
    644 depth (r')                  & 21.8              & 24.0              & 24.5              \\
    645 bytes star$^{-1}$           & 64                & 100               & 64                \\
    646 stars deg$^{-2}$ ($|b|>10$) & $2.0 \times 10^5$ & $8.0 \times 10^5$ & $2.0 \times 10^5$ \\
    647 stars FPA$^{-1}$ ($|b|>10$) & $1.4 \times 10^6$ & $5.6 \times 10^6$ & $1.4 \times 10^6$ \\
    648 stars sec$^{-1}$ ($|b|>10$) & $3.5 \times 10^4$ & $3.5 \times 10^4$ & $8.8 \times 10^3$ \\
    649 MB sec$^{-1}$               & 2.3               & 3.5               & 0.6               \\
    650 AP total TB                 & 7.7               & -                 & -                 \\               
    651 IVP total TB                & 13                & 20                & 3                 \\               
    652 MOPS total TB               & 4                 & 6                 & 1                 \\               
    653 PS-1 total TB               & 25                & 26                & 4                 \\
    654 \hline
    655 \end{tabular}
    656 \end{center}
    657 \end{table}
    658 
    659 %% IPP AP DB Requirements
    660 The IPP AP Database has the following performance requirements:
    661 
    662 \begin{enumerate}
    663 \item The AP Database shall accept new detections at the rate
    664   generated by the telescope from the Phase 2 and Phase 4 analysis.
    665   Except within 10 degrees of the galactic plane, the AP Database
    666   shall keep up with the incoming rates.  The expected rates are
    667   listed in Table~\ref{APrates}, along with the total data volume
    668   required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2,
    669   TLR:3, TLR:22}
    670 
    671 \item The AP Database shall provide access to external Pan-STARRS
    672   clients to the detected objects within \tbr{5 minute} after the
    673   image is obtained.\VER{TEST}{TLR:22}
    674   \label{IPP:DeReq:29c}
    675 \end{enumerate}
    676 
    677603\subsubsection{Metadata Database}
     604
     605%% Metadata DB Requirements
     606
     607The Metadata Database has the following requirements:
     608
     609\begin{enumerate}
     610\item The IPP Metadata Database shall accept metadata from the summit
     611   at a nightly average rate of 1 MB per 40 second.\VER{TEST}{TLR:17,
     612   TLR:21, TLR:25}
     613
     614\item The Metadata Database queries shall have a latency of $< 0.1$
     615  seconds.\VER{TEST}{TLR:17}
     616
     617\item The Metadata Database shall be capable of at least 100 queries
     618  per second.\VER{TEST}{TLR:17}
     619
     620\item The Metadata Database shall be capable of accepting a total data
     621  volume after 2 years of operation of 280 GB. \VER{INSPECT}{TLR:25}
     622
     623\item The Metadata Database shall restrict write access of the
     624  scientific parameters to a different group from the software and
     625  hardware configuration parameters.\VER{TEST}{allocated}
     626\end{enumerate}
    678627
    679628%% Table: Metadata data classes
    … …  
    702651\end{table}
    703652
    704 %% Metadata DB Requirements
    705 
    706 The Metadata Database has the following requirements:
    707 
    708 \begin{enumerate}
    709 \item The IPP Metadata Database shall accept metadata from the summit
    710    at a sustained rate of \tbr{1 MB per 40 second.}\VER{TEST}{TLR:17,
    711    TLR:21, TLR:25}
    712 
    713 \item The Metadata Database queries shall have a latency of $< 0.1$
    714   seconds.\VER{TEST}{TLR:17}
    715 
    716 \item The Metadata Database shall be capable of at least 100 queries
    717   per second.\VER{TEST}{TLR:17}
    718 
    719 \item The Metadata Database shall be capable of accepting a total data
    720   volume after 2 years of operation of 280 GB. \VER{INSPECT}{TLR:25}
    721 
    722 \item The Metadata Database shall restrict write access of the
    723   scientific parameters to a different group from the software and
    724   hardware configuration parameters.\VER{TEST}{allocated}
    725 \end{enumerate}
     653\subsubsection{AP Database}
     654
     655%% IPP AP DB Requirements
     656The IPP AP Database has the following performance requirements:
     657
     658\begin{enumerate}
     659\item The AP Database shall accept new detections at the rate
     660  generated by the telescope from the Phase 2 and Phase 4 analysis.
     661  Except within 10 degrees of the galactic plane, the AP Database
     662  shall keep up with the incoming rates.  The expected rates are
     663  listed in Table~\ref{APrates}, along with the total data volume
     664  required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2,
     665  TLR:3, TLR:22}
     666
     667\item The AP Database shall provide access to external Pan-STARRS
     668  clients to the detected transient objects within 15 minutes after
     669  the image is obtained with an 85\% reliability.\VER{TEST}{TLR:22}
     670  \label{IPP:DeReq:29c}
     671\end{enumerate}
     672
     673%%% Table: AP DB parameters
     674\begin{table}[hb]
     675\begin{center}
     676\caption{AP Detection Classes \& Object Parameters\label{APdetections}}
     677\begin{tabular}{lrrrr}
     678\hline
     679\hline
     680Object Parameter & P2 & P4$\Sigma$ & P4$\Delta$ & SS \\
     681\hline
     682PSF x,y, covar, $\alpha,\delta$               & + & + & + & + \\
     683PSF mag, $\sigma_{\rm mag}$                   & + & + & + & + \\
     684star/gal sep                                  & + & + & + & + \\
     685$\sigma_x$, $\sigma_y$, $\theta$              & + & + & + & + \\
     686local sky data                                & + & + & + & + \\
     687Petrosian R, M, $R_{50}$, $R_{90}$            & - & + & - & + \\
     688S\'ersic R, M, AB, $\phi$, $\nu$              & - & + & - & + \\
     689W.L. $\gamma_1$, $\gamma_2$, pol. terms       & - & - & - & + \\
     690exp. spaced aps., Poisson noise, variance     & - & - & - & + \\
     691\hline
     692\end{tabular}
     693\end{center}
     694\end{table}
     695
     696%%% Table: AP DB Throughput
     697\begin{table}
     698\begin{center}
     699\caption{AP Data Volume and Throughput Requirements\label{APrates}}
     700\begin{tabular}{lrrr}
     701\hline
     702\hline
     703Quantity                    & P2                & P4$\Sigma$        & P4$\Delta$        \\
     704\hline
     705detection limit             & $20 \sigma$       & $5 \sigma$        & $3 \sigma$        \\
     706depth (r')                  & 21.8              & 24.0              & 24.5              \\
     707bytes star$^{-1}$           & 64                & 100               & 64                \\
     708stars deg$^{-2}$ ($|b|>10$) & $2.0 \times 10^5$ & $8.0 \times 10^5$ & $2.0 \times 10^5$ \\
     709stars FPA$^{-1}$ ($|b|>10$) & $1.4 \times 10^6$ & $5.6 \times 10^6$ & $1.4 \times 10^6$ \\
     710stars sec$^{-1}$ ($|b|>10$) & $3.5 \times 10^4$ & $3.5 \times 10^4$ & $8.8 \times 10^3$ \\
     711MB sec$^{-1}$               & 2.3               & 3.5               & 0.6               \\
     712AP total TB                 & 7.7               & -                 & -                 \\               
     713IVP total TB                & 13                & 20                & 3                 \\               
     714MOPS total TB               & 4                 & 6                 & 1                 \\               
     715PS-1 total TB               & 25                & 26                & 4                 \\
     716\hline
     717\end{tabular}
     718\end{center}
     719\end{table}
    726720
    727721\subsubsection{Controller}
    … …  
    810804\begin{enumerate}
    811805\item The IPP Science Analysis shall pre-process the science images
    812  with the master calibration images at a sustained rate of 1 exposure
    813  (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17}
     806 with the master calibration images at a nightly average rate of 1
     807 exposure (2~GB) per 40 seconds.\VER{TEST}{TLR:17}
    814808
    815809\item The IPP Science Analysis shall merge multiple pre-processed
    816810 science images into stacked images with corresponding signal-to-noise
    817  maps at a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17}
     811 maps at a nightly average rate of 1 exposure (2~GB) per 40
     812 seconds.\VER{TEST}{TLR:17}
    818813
    819814\item The IPP Science Analysis shall excise pixels from the input
    … …  
    823818\item The IPP Science Analysis shall merge the cleaned images into the
    824819 static sky image, and update the corresponding exposure (S/N) maps,
    825  at a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17}
     820 at a nightly average rate of 1 exposure (2~GB) per 40
     821 seconds.\VER{TEST}{TLR:17}
    826822
    827823\item The maximum latency between the acquisition of an image and the
    828824  completion of the science image analysis is set by the science
    829825  requirements of the fast transient recovery programs.  The science
    830   image analysis shall process images to detection transients within
    831   \tbr{5 min} of their acquisition.\VER{TEST}{TLR:22}
     826  image analysis shall process images to the detection of transients
     827  within 15 min of their acquisition with an 85\%
     828  reliability.\VER{TEST}{TLR:22}
    832829
    833830\item The science image analysis stages shall processes up to 1000
    … …  
    905902  images which are not undersampled.  \VER{TEST}{TLR:18}
    906903
    907 \item The resulting astrometric solution shall be consistent across the
    908   OTA field to within \tbr{100 milli-arcsec}.\VER{TEST}{TLR:4}
    909904\end{enumerate}
    910905
    … …  
    926921  resulting astrometric solution shall have a residual scatter of $<
    927922  30$ milliarcseconds when calibrated with the AP Survey reference
    928   catalog and $< 100$ milliarcseconds when calibrated with the USNO-B
    929   catalog.\VER{ANALYSIS}{TLR:}
     923  catalog and $< 200$ milliarcseconds when calibrated with the USNO-B
     924  catalog.\VER{ANALYSIS}{TLR:4}
    930925
    931926\item For images obtained under normal observing conditions, the
    932   resulting astrometric solution shall have a precision relative to
    933   ICRS of better than 100 milliarcseconds.\VER{ANALYSIS}{TLR:}
     927  resulting astrometric solution shall have systematic errors relative
     928  to ICRS of $< 100 milliarcseconds$.\VER{ANALYSIS}{TLR:3}
    934929
    935930\item For images obtained under photometric conditions or minimal
    936931  cirrus conditions ($< 0.1$ mag total extinction), the resulting
    937932  photometric calibration shall have a relative accuracy of 5
    938   millimagnitudes.\VER{ANALYSIS}{TLR:}
     933  millimagnitudes.\VER{ANALYSIS}{TLR:1}
    939934
    940935\item For images obtained under photometric conditions or minimal
    … …  
    942937  photometric calibration shall have an absolution photometric
    943938  accuracy of 10 millimagnitudes when calibrated relative to the AP
    944   Survey reference catalog.\VER{ANALYSIS}{TLR:}
     939  Survey reference catalog.\VER{ANALYSIS}{TLR:1}
    945940
    946941\item For images obtained under photometric conditions or minimal
    … …  
    948943  conditions listed in Table~\ref{moonconditions}, the resulting sky
    949944  background subtraction shall leave behind peak-to-peak residuals $<
    950   1$\% of the input sky flux.\VER{ANALYSIS}{TLR:}
     945  1$\% of the input sky flux.\VER{ANALYSIS}{TLR:1}
    951946
    952947\end{enumerate}
    … …  
    968963
    969964\item The sky representation shall degrade the image quality by less
    970   than 10 milliarcseconds added in quadrature to the input image
     965  than 150 milliarcseconds added in quadrature to the input image
    971966  quality.\VER{TEST}{TLR:1}
    972967
    … …  
    975970  time. \VER{TEST}{TLR:17}
    976971
    977 \item \tbd{completeness}
    978 
    979 \item \tbd{contamination}
     972\item The Phase 4 analysis shall have a transient detection
     973  completeness of 99\% for detections with a significance $> 5\sigma$.
     974
     975\item The Phase 4 analysis shall have a false detection rate of $<
     976  5\%$ for transients detections with a significance $> 5\sigma$.
    980977
    981978\end{enumerate}
    … …  
    10251022The required set of Pan-STARRS modules and their functionality is
    10261023specified in the document `Pan-STARRS Image Processing Pipeline Modules
    1027 Supplementary Design Requirements' (PSDC-430-xxx), and details of
     1024Supplementary Design Requirements' (PSDC-430-012), and details of
    10281025specific algorithms are specified in the document `Pan-STARRS Image
    10291026Processing Pipeline Algorithm Design Document' (PSDC-430-006).
    … …  
    10721069\subsubsection{External Catalogs}
    10731070
     1071\begin{table}
     1072\begin{center}
     1073\caption{Astrometric Reference Catalogs\label{AstroRefs}}
     1074\begin{tabular}{lrrrrl}
     1075\hline
     1076\hline
     1077Name       & scatter limit   & proper   & depth      & Nstars     & filters \\
     1078           & (milliarcsec)   & motion   &(mag)       & (millions) &         \\
     1079\hline
     1080Hipparcos  &   1             & 2        &  7.3       &    0.1     & {\em V}       \\
     1081Tycho2     &  10             & 1        & 11.5       &    2.5     & {\em B,V}     \\
     1082UCAC-2     &  20             & 1        & 16.0       &   48.0     & {\em R}       \\
     1083USNO-A2.0  & 250             & N/A      & 19.0       &  526.2     & {\em B,R}     \\
     1084USNO-B1.0  & 200             & 20       & 21.0       & 1042.6     & {\em B,R}     \\
     10852MASS      &  70             & N/A      & 16.0       &  470.0     & {\em J,H,K}   \\
     1086\hline
     1087\end{tabular}
     1088\end{center}
     1089\end{table}
     1090
     1091\begin{table}
     1092\begin{center}
     1093\caption{Photometric Reference Catalogs\label{PhotoRefs}}
     1094\begin{tabular}{lrrr}
     1095\hline
     1096\hline
     1097Name       & scatter  & depth & filters \\
     1098           & mmag     & mag   &         \\
     1099\hline
     1100SDSS       & 15       & 16    & {\em u,g,r,i,z} \\
     1101CFHT-LS    & 10       & 18    & {\em u,g,r,i,z} \\
     1102Landolt    & 10-20    & 15    & {\em U,B,V,R,I} \\
     1103\hline
     1104\end{tabular}
     1105\end{center}
     1106\end{table}
     1107
    10741108The IPP AP Database shall be able to interact with the externally
    10751109provided reference catalogs listed in Table~\ref{AstroRefs} and
    … …  
    10781112\subsubsection{Static Sky Pixel Size}
    10791113
    1080 The IPP static sky shall have a pixel scale of \tbr{0.2\arcsec}.
     1114The IPP static sky shall have a pixel scale of
     11150.2\arcsec.\VER{ANALYSIS}{TLR:16}
    10811116
    10821117\subsection{External Interfaces}
    … …  
    11821217\hline
    11831218\hline
    1184 Raw data           & 200 TB \\
     1219Raw data           & 400 TB \\
    11851220static sky         & 350 TB \\
    11861221calibration frames & 2.8 TB \\
    … …  
    11881223AP db              &  55 TB \\
    11891224\hline
    1190 total              & 610 TB \\
     1225total              & 810 TB \\
    11911226\hline
    11921227\end{tabular}
    … …  
    12041239\begin{enumerate}
    12051240\item The IPP shall store all raw images from the first year from the
    1206   AP and IVP surveys.  This corresponds to 175,000 images, or 175 TB,
    1207   assuming 1 GB per image with compression.  The IPP will require
    1208   space for 200 TB of raw imagery to store the data from these two
    1209   survey components along with raw calibration, test, and short-term
    1210   storage of other raw images not in the AP and IVP
    1211   surveys.\VER{INSPECT}{TLR:23}
     1241  AP and IVP surveys.  This corresponds to 180,000 images, or 360 TB,
     1242  assuming 2 GB per image.  The IPP will require space for 400 TB of
     1243  raw imagery to store the data from these two survey components along
     1244  with raw calibration, test, and short-term storage of other raw
     1245  images not in the AP and IVP surveys.\VER{INSPECT}{TLR:23}
    12121246
    12131247\item The IPP shall store a single copy of the complete static sky in
    … …  
    12261260  represent at most 2 terabytes.  \VER{INSPECT}{TLR:25}
    12271261
    1228 \item The IPP shall have storage capacity for a total of 610 TB of data.
     1262\item The IPP shall have storage capacity for a total of 810 TB of
     1263  data by the end of PS-1.
    12291264\end{enumerate}
    12301265
    … …  
    13531388%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    13541389
    1355 \section{Appendices}
     1390\clearpage
     1391\appendix
    13561392
    13571393\bibliographystyle{plain}
Note: See TracChangeset for help on using the changeset viewer.