Index: /trunk/doc/design/ippSDRS.tex =================================================================== --- /trunk/doc/design/ippSDRS.tex (revision 2543) +++ /trunk/doc/design/ippSDRS.tex (revision 2544) @@ -1,9 +1,9 @@ -%%% $Id: ippSDRS.tex,v 1.14 2004-10-29 22:00:08 eugene Exp $ +%%% $Id: ippSDRS.tex,v 1.15 2004-11-30 23:16:03 eugene Exp $ \documentclass[panstarrs]{panstarrs} % basic document variables -\title{Pan-STARRS Image Processing Pipeline} -\subtitle{Supplementary Design Requirements Specification} -\shorttitle{IPP SDRS} +\title{Pan-STARRS PS-1 Image Processing Pipeline} +\subtitle{System/Subsystem Design Description} +\shorttitle{IPP SSDD} \author{Eugene A. Magnier, Paul A. Price, Josh Hoblitt} \audience{Pan-STARRS PMO} @@ -11,5 +11,5 @@ \project{Pan-STARRS Image Processing Pipeline} \organization{Institute for Astronomy} -\version{DR} +\version{00} \docnumber{PSDC-430-011} @@ -27,13 +27,16 @@ DR.03 & 2004.03.25 & Section reorganization \\ \hline DR.04 & 2004.04.13 & Most sections fleshed out \\ \hline -DR.05 & 2004.04.29 & Reorganisation for consistency \\ \hline +DR.05 & 2004.04.29 & Reorganization for consistency \\ \hline DR.06 & 2004.10.21 & Major revision in prep of PDR \\ \hline \RevisionsEnd +\inserttbd +\inserttbr +\pagebreak + +\tableofcontents +\pagebreak + \listoffigures - -\pagebreak - -\tableofcontents \pagebreak \pagenumbering{arabic} @@ -89,10 +92,10 @@ \subsection{Document Overview} -The Pan-STARRS IPP Software Requirements Specification contains the -complete system requirements of the Pan-STARRS PS-1 IPP in order to -achieve the top-level performance and operational requirements -specified by the SCD. The requirements flow begun in the SGS and -continued in the SCD is further developed in this SRS to provide -additional derived system and subsystem requirements. +The Pan-STARRS IPP System/Subsystem Design Description (SSDD) contains +the complete design description of the Pan-STARRS PS-1 IPP in order to +achieve the requirements specified by the Pan-STARRS PS-1 IPP Software +Requirements Specification (SRS; PSDC-430-005). The requirements flow +begun in the SGS and SCD and continued in the SRS is used to guide the +design presented here. \subsection{Requirements Definitions} @@ -103,8 +106,9 @@ that series is implied. -Open issues (TBDs) in this document are marked \tbd{in bold red}. - -Quantities which should be reviewed (TBRs) are marked \tbr{in bold -blue}. +Open issues (TBDs) in this document are marked {\bf \color{red} in +bold red}. + +Quantities which should be reviewed (TBRs) are marked {\bf +\color{blue} in bold blue}. \subsubsection{``Shall''} When used in this specification, the word @@ -123,8 +127,14 @@ \DocumentsInternalSection -PSDC-130-001 & PS-1 Design Reference Mission \\ \hline +PSDC-230-001 & PS-1 Design Reference Mission \\ \hline +PSDC-230-002 & PS-1 System Concept Definition \\ \hline +PSDC-400-006 & The Pan-STARRS IPP Computational Challenge \\ \hline PSDC-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline -PSDC-430-006 & Pan-STARRS IPP ADD \\ \hline -PSDC-430-007 & Pan-STARRS IPP PSLib SDR \\ \hline +PSDC-430-005 & Pan-STARRS IPP PS-1 Software Requirements Specification \\ \hline +PSDC-430-006 & Pan-STARRS IPP Algorithm Design Document \\ \hline +PSDC-430-007 & Pan-STARRS IPP PSLib Supplementary Design Requirements Specification \\ \hline +PSDC-430-010 & Pan-STARRS IPP Perl Code Conventions \\ \hline +PSDC-430-012 & Pan-STARRS IPP Modules Supplementary Design Requirements Specification \\ \hline +PSDC-430-014 & Pan-STARRS IPP PS-1 Cluster Support \\ \hline \DocumentsExternalSection Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ @@ -140,9 +150,9 @@ Pan-STARRS. It also is responsible for combining all of the science images in a given filter into a single representation of the -non-variable component of the night sky called the ``Static Sky''. To -achieve these goals, the IPP also performs other analysis functions to -generate the calibrations needed in the science image processing and -to occasionally use the derived data to generate improved astrometric -and photometric reference catalogs. It also provides the +non-variable component of the night sky defined as the ``Static Sky''. +To achieve these goals, the IPP also performs other analysis functions +to generate the calibrations needed in the science image processing +and to occasionally use the derived data to generate improved +astrometric and photometric reference catalogs. It also provides the infrastructure needed to store the incoming data and the resulting data products. @@ -166,14 +176,14 @@ transient objects. 3) the Published Science Products Subsystem (PSPS), which will receive all data products of interest to the -outside world, and will act as the long-term archive and publishing -clearinghouse. +community external to the Pan-STARRS data processing systems, and will +act as the long-term archive and publishing clearinghouse. The IPP receives data from two Pan-STARRS subsystems: the Camera, from which it receives the large volume of image data, and OTIS -(Observatory, Telesope and Infrastructure Subsystem), from which it +(Observatory, Telescope and Infrastructure Subsystem), from which it receives metadata describing the images and the environmental conditions. The primary IPP hardware system on which the software -operates will not be located at the summit. Instead, because of -thermal, power, and space constraints, the hardware will likely be +operates will probably not be located at the summit. Instead, because +of thermal, power, and space constraints, the hardware will likely be located in a facility off the mountain. A subset of processing tasks may eventually be assigned to machines at the summit if justified by @@ -186,33 +196,36 @@ This document defines the design requirements of the IPP for the PS-1 -prototype telescope. Even so, much of the IPP design for PS-4 will be +prototype telescope. Much of the IPP design for PS-4 will be identical to or closely based on the PS-1 implementation. The -software organization and the infrastructure systems will be -identical, with minor improvements in details. The range analysis -steps to be performed will be nearly identical, with some additional -details added for PS-4 to improve the accuracy. - -In terms of the IPP, PS-1 differs from the complete PS-4 system in -several important ways. First, with only one telescope and camera, -the data throughput rate is substantially reduce to a maximum of 1 -64-OTA image per 40 seconds rather than 4. Since PS-1 is a prototype -for testing the Pan-STARRS hardware and software subsystems, the -observing strategy is not a fixed quantity. The PS-1 Design Reference -Mission (PSDC-xxx) provides some guidelines for the types of projects -to be performed, including starting an AP Survey which will eventually -cover the entire $3\pi$ steradians of the sky accessible to PS-4. As -a prototype, it is expected that much of the data collected by PS-1 -will be processed multiple times to test and tune the analysis steps. -This difference in approach has implications for the storage required -by PS-1: rather than delete images soon after they have been used, raw -images must be stored for at least the first 18 months of PS-1 -operations. We have used the PS-1 Design Reference Mission as a -baseline for these storage requirements to drive our hardware design. +software organization and the infrastructure systems are expected to +be identical, with minor improvements in details. The type of +analysis steps to be performed will be nearly identical, with some +additional details added for PS-4 to improve the accuracy. + +Although generally very similar, in terms of the IPP PS-1 differs from +the complete PS-4 system in several specific ways. First, with only +one telescope and camera, the data throughput rate is substantially +reduced to a maximum of 1 64-OTA image per 40 seconds rather than 4. +Since PS-1 is a prototype for testing the Pan-STARRS hardware and +software subsystems, the observing strategy is not a fixed quantity. +The PS-1 Design Reference Mission (PSDC-230-001) provides some +guidelines for the types of observing tests which will probably be +performed, including possibly starting an Astrometric and Photometric +Survey which will eventually cover the entire $3\pi$ steradians of the +sky accessible to PS-4. As a prototype, it is expected that much of +the data collected by PS-1 will be processed multiple times to test +and tune the analysis steps. Compare with PS-4, this difference in +approach has implications for the storage required by PS-1: rather +than delete images soon after they have been used, raw images from +demonstration observations must be stored for at least the first two +years of PS-1 operations. The PS-1 Design Reference Mission is used +as an upper limit for these storage requirements to drive the hardware +design. \subsection{System Design Decisions} Since Pan-STARRS is a survey project, all data from the telescopes -will be uniformly analysed by the Pan-STARRS Image Processing Pipeline -(IPP) and the appropriate resulting data products made available to +will be uniformly analyzed by the Pan-STARRS Image Processing Pipeline +(IPP), and the appropriate resulting data products made available to internal and external science analysis systems as they become available. The processing performed by the IPP on the science images @@ -223,17 +236,17 @@ object analysis of the static sky images. In addition, the IPP will produce improved astrometric and photometric reference catalogs on an -occasional basis as needed. The output data products from the IPP -consist of the calibration images, reduced images from the individual -telescopes, combined images, difference images, the static sky image, -object photometry, and reference astrometry and photometry. - -The requirements for the IPP, as identified in the IPP SRS (PSDC-REF) -fall into several broad categories: Data analysis precision, -throughput, system reliability, flexibility, testability, and -traceability. The details of the analysis tasks are specified in +as-needed basis. The output data products from the IPP consist of the +calibration images, reduced images from the individual telescopes, +combined images, difference images, the static sky image, object +photometry, and reference astrometry and photometry. + +The requirements for the IPP, as identified in the PS-1 IPP SRS +(PSDC-430-005) fall into several broad categories: data analysis +precision, throughput, system reliability, flexibility, testability, +and traceability. The details of the analysis tasks are specified in order to achieve the precision. The architectural design as discussed below is motivated by the need for reliability and flexibility. The -hardware organization and the distributed / parallel processing model -is motivated by the throughput requirements. The need for flexibility +hardware organization and the distributed/parallel processing model is +motivated by the throughput requirements. The need for flexibility and testability drives the software organization. The need for simple testing procedures drives both the software organization and the @@ -255,11 +268,11 @@ OTA number 61 from exposure 654321 to produce a specific set of output data products. The analysis stages are discussed in detail in -Section~\ref{IPP:AnalysisStages}. - -Depending on the particular stage, it may process individual images, -collections of images, or derived data products. Because of the -nature of the image data, many of the analysis stages can be run in -parallel. For example, the analysis of a chip in one image does not -depend on the results from another chip. +Section~\ref{sec:AnalysisStages}. + +A particular stage may process individual images, collections of +images, or derived data products. Because of the nature of the image +data, many of the analysis stages can be run in parallel if needed to +increase the processing throughput. For example, the analysis of a +chip in one image does not depend on the results from another chip. \subsection{Architectural Components} @@ -268,13 +281,13 @@ \begin{center} \resizebox{6in}{!}{\includegraphics{pics/IPPoverview}} -\caption{ \label{overview} IPP System Overview} +\caption{ \label{fig:overview} IPP System Overview} \end{center} \end{figure} In order to achieve the required functionality, the IPP provides an -infrastructure within which the Analysis Stages above are exectuted. -In order to facilitate the subsystem testing, we have divided the IPP -software infrastructure into a number of clearly-defined architectural -software units, listed as follows: +infrastructure within which the Analysis Stages described above are +executed. In order to facilitate the subsystem testing, the IPP +software infrastructure has been divided into a number of +clearly-defined architectural software units as follows: \begin{itemize} @@ -288,5 +301,5 @@ restricted to imaging data: it is capable of storing any large data files which are not well-suited for inclusion in a more structured - relational database and for which access needs to be widely + relational database, and for which access needs to be widely available beyond the individual process which created the file. @@ -295,5 +308,9 @@ needed to perform the IPP analyses. The metadata may include the summary weather information for each night, or details about the - filters, camera, telescopes, etc. + filters, camera, telescopes, etc. Note that the IPP Metadata + Database is not required to retain all archival engineering data + from all of Pan-STARRS; other Pan-STARRS subsystems use their own + internal databases to store engineering metadata and only the + necessary subset is transferred to the IPP Metadata Database. \item {\bf Astrometry \& Photometry Database (AP DB):} This component @@ -304,9 +321,9 @@ query and manipulate the objects and detections. -\item {\bf IPP Controller:} In order to perform the analysis stages - required by the IPP, it is necessary to use distributed computing - processes on a large number of computers. The IPP Controller - manages the collection of analysis tasks performed on these - machines. +\item {\bf IPP Controller:} In order to achieve the required + processing throughput for the IPP analysis stages, it is necessary + to use distributed computing processes on a large number of + computers. The IPP Controller manages the collection of analysis + tasks performed on these machines. \item {\bf IPP Scheduler:} This component is a decision-making @@ -318,13 +335,12 @@ The relationship between these software units is shown in -Figure~\ref{overview}. This figure also shows the interactions +Figure~\ref{fig:overview}. This figure also shows the interactions between the IPP and other Pan-STARRS systems. The architectural -components are discussed in detail in -Section~\ref{IPP:ArchComponents}. +components are discussed in detail in Section~\ref{sec:ArchComponents}. \begin{figure} \begin{center} \resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}} -\caption{ \label{hardware} IPP Hardware Organization} +\caption{ \label{fig:hardware} IPP Hardware Organization} \end{center} \end{figure} @@ -332,5 +348,5 @@ \subsection{IPP Hardware Organization} -The IPP needs substantial computer resources, both in terms of +The IPP will utilize substantial computer resources, both in terms of computational power and in terms of data storage and network bandwidth. The IPP requires relatively large amounts of data storage @@ -354,10 +370,10 @@ the static sky storage nodes. -Figure~\ref{hardware} shows our basic concept for the hardware +Figure~\ref{fig:hardware} presents the basic concept for the hardware organization for the IPP. This diagram shows the two types of compute -nodes: OTA-level processing and storage nodes and static sky +nodes: (1) OTA-level processing and storage nodes and (2) Static Sky processing and storage nodes. Also shown are two switches which divide the network into OTA and Static-Sky portions. In such an -organization, the interswitch communication must meet the throughput +organization, the inter-switch communication must meet the throughput needs between these network portions (though a single switch may also be used if its backplane capacity is sufficient). The additional data @@ -367,6 +383,16 @@ \section{System Design : Architectural Components} +\label{sec:ArchComponents} \subsection{IPP Image Server} + +\subsubsection{Corresponding Requirements} + +The Image Server must meet the requirements specified in Section 3.4.1 +of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The specified design +is chosen to meet requirements 3.4.1.3, and 3.4.1.5. The other three +requirements (3.4.1.1, 3.4.1.2, and 3.4.1.4) depend on the volume and +capabilities of the hardware, and are addressed in +Section~\ref{sec:Hardware}. \subsubsection{Image Server Overview} @@ -378,5 +404,5 @@ Server include the raw images, the calibration images, intermediate processing stage images as needed, final processed images, difference -images, image subsections, and any large non-imaging datafiles needed +images, image subsections, and any large non-imaging data files needed by the IPP. The IPP Image Server must retain the files for as long as they are needed by the IPP. @@ -396,12 +422,12 @@ There are three data concepts relevant to the IPP Image Server: \begin{itemize} -\item {\bf storage object} This represents a single, unique data +\item {\bf Storage object:} This represents a single, unique data entity in the Image Server. -\item {\bf instance} A single copy of the storage object in the Image +\item {\bf Instance:} A single copy of the storage object in the Image Server. In general, a given storage object may have several instances in the Image Server, normally on different computer nodes. -\item {\bf file ID} This is the identifier of a particular storage +\item {\bf File ID:} This is the identifier of a particular storage object in the Image Server. The file ID is simply a unique string, equivalent to the filename in a UNIX file system. @@ -421,5 +447,6 @@ on some schedule. -The IPP Image Server consists of the following components: +As shown in Figure~\ref{fig:ImageServer}, the IPP Image Server +consists of the following components: \begin{itemize} @@ -428,5 +455,5 @@ \item Image Server daemon \item Image Server client APIs -\item Image Server maintainence tools +\item Image Server maintenance tools (not shown) \end{itemize} @@ -442,6 +469,6 @@ Clients interact with the IPP Image Server via a small number of C -APIs (Bindings are also provided for Perl and Python and UNIX shell -commands in some cases). The client commands are: +APIs. Bindings are also provided for Perl and Python and UNIX shell +commands in some cases. The client commands are: \begin{itemize} @@ -497,11 +524,11 @@ hardware resources. A {\tt mysql} database engine is used to manage the database table. The database tables defined for the Image Server -are listed in Table~\ref{ImageServerTables}, and their contents are -listed in Appendix A. This database engine need not the same one as -the one used for othe IPP subsystems. +are listed in Table~\ref{tab:ImageServerTables}, and their contents are +listed in Appendix~\ref{sec:ImageServerTableContents}. This database +engine need not be the same one used for other IPP subsystems. % -\begin{table} -\begin{center} -\caption{Image Server Database Tables\label{ImageServerTables}} +\begin{table}[ht] +\begin{center} +\caption{Image Server Database Tables\label{tab:ImageServerTables}} \begin{tabular}{ll} \hline @@ -529,8 +556,8 @@ The IPP Image Server provides a collection of administration tools -which allow for maintainence. These are operations which may be +which allow for maintenance. These are operations which may be automatically scheduled by the IPP or which may be initiated by a -human via a command-shell interface. The maintainence functions -include migrating data between nodes to rebalance the available space +human via a command-shell interface. The maintenance functions +include migrating data between nodes to re-balance the available space (this would only occur for instances which have not been placed on a specific node by the client API). Other functions include checking @@ -542,5 +569,14 @@ \subsection{Metadata Database} -\label{Metadata} +\label{sec:Metadata} + +\subsubsection{Corresponding Requirements} + +The Metadata Database must meet the requirements specified in Section +3.4.2 of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The specified +design is chosen to meet requirements 3.4.2.1, 3.4.2.2, 3.4.2.3, +3.4.2.4, 3.4.2.5. + +\subsubsection{Overview} The IPP Metadata Database acts as a repository for non-pixel data @@ -558,5 +594,5 @@ Metadata Database may be collected and inserted by a separate, dedicated process. Metadata which is large in volume or poorly -structure may also be stored in an appropriate container file (FITS +structured may also be stored in an appropriate container file (FITS Table, FITS Header, XML File) in the Image Server with the Metadata DB providing pointers to these files. @@ -568,5 +604,5 @@ \begin{table}[hb] \begin{center} -\caption{Metadata Database Tables\label{MetadataDBTables}} +\caption{Metadata Database Tables\label{tab:MetadataDBTables}} \begin{tabular}{ll} \hline @@ -597,21 +633,23 @@ \subsubsection{Metadata Tables} -The contents of the Metadata Database will not be completely specified -until the complete collection of data analysis scripts are available. -Even so, we can identify the likely collection of long-term tables, -and some of the temporary processing tables. -Table~\ref{MetadtaDBTables} lists the Metadata tables identified to +Table~\ref{tab:MetadataDBTables} lists the Metadata tables identified to date for the Metadata Database. The contents of these tables are -outlined in Appendix~\ref{MetadataContents}, with examples for the -data entries and thier data types in many cases. +outlined in Appendix~\ref{sec:MetadataTableContents}, with examples for +the data entries and their data types in many cases. Additional +tables will be added as necessary as the data analysis scripts are +fleshed out in detail. The Metadata Database, with a flat data +organization, is flexible enough to add additional information as it +is identified. \subsubsection{Metadata Queries} The IPP provides simple queries to the Metadata Database tables using -autocoded APIs. These queries return a single row or a collection of +auto-coded APIs. These queries return a single row or a collection of rows based on the primary key. The format of the API is identical for all Metadata tables. New tables and APIs can be added to the IPP -system by adding to the autocoding table description files. See -Appendix~\ref{Autocode} for futher information. +system by adding to the auto-code table description files. The +auto-code API includes read and write access permissions to be set +for each table independently. See Appendix~\ref{sec:AutocodeIO} for +further information. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -619,16 +657,26 @@ \subsection{AP Database} +\subsubsection{Corresponding Requirements} + +The AP Database must meet the requirements specified in Section 3.4.3 +of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The specified design +is chosen to meet requirements 3.4.3.1 and 3.4.3.2. In order to meet +the throughput requirements, the AP Database will be distributed +across 10 Nodes independent of the Image Server Nodes. An alternative +organization of the database which will be studied will have the AP +Database co-located with the Image Server Phase 4 Nodes. + \subsubsection{Overview} -The AP (Astrometry \& Photometry) Database is a mechanism to store -data related to astronomical objects derived from various sources with -a variety of associations. The AP Database deals with two related -concepts: {\em objects} and {\em detections}. The objects are -descriptions of astronomical objects while the detections are the -specific measurements of those objects, typically measured from +The AP (Astrometry \& Photometry) Database is a CSCI which stores data +related to astronomical objects derived from various sources with a +variety of associations. The AP Database deals with two related +concepts: {\em objects} and {\em detections}. The {\em objects} are +descriptions of astronomical objects while the {\em detections} are +the specific measurements of those objects, typically measured from astronomical images. A collection of {\em detections} may be used to derive average quantities which describe a particular {\em object}. A -third class of object information which must also be considered are -those supplied by external references. These may be treated as {\em +third class of measurement to be considered are those supplied by +external references. Such measurements may be treated as {\em detections}, with the caveat that access to the raw measurements and metadata are usually unavailable: the reported measurements and errors @@ -637,9 +685,10 @@ The AP Database stores the collections of detections which were derived from specific images from any of the analysis stages. It -provides a mechanism to determine and (in conjunction with the Image -Server) locate the image from which a specific detection was derived. -The AP Database also makes it possible to extract all detections -derived from a specific image and to determine quantities such as the -pixel coordinates of the detection on the image. +provides a mechanism to determine the image from which a specific +detection was derived, and in conjunction with the Image Server locate +the corresponding data file. The AP Database also makes it possible +to extract all detections derived from a specific image and to +determine quantities such as the pixel coordinates of the detection on +the image. The AP Database also has the capability to associate multiple @@ -648,6 +697,6 @@ First, the most distant stars, compact galaxies, and QSOs will have -nearly fixed locations relative to other nearby stars, with only small -deviations for individual measurements. The association between +nearly fixed locations relative to other distant stars, with only +small deviations for individual measurements. The association between multiple detections of such objects is made on the basis of their coincident positions. The AP Database determines the average position @@ -658,5 +707,5 @@ of such objects are linked by their orbits, and depend on both the position and the time of the image. The AP Database does not attempt -to make this link, which is the role of the MOPS system. However, it +to make this link; this is the role of the MOPS system. However, it has the ability to accept identifications made externally with specified detections and to return the identifier of the moving object @@ -667,7 +716,7 @@ moving object detections from other types of queries. -Third, stars in the general vicinity of the solar system fall in +Third, objects in the general vicinity of the solar system fall in between these first two classes of objects. Their proper motion and -parallax response is significant enough ($>1$ arcsec in 1 year) that +parallax response is significant enough ($>0.2$ arcsec in 1 year) that they are not well-described by an average location and a collection of offsets. These objects are described by a distance and a proper @@ -679,15 +728,15 @@ be associated with a specific astronomical object of any of the above classes and are treated as orphans. Most of these will be spurious -(not represent real objects), some will be from solar system objects -for which orbits are not yet determined, some will be from faint stars -near the detection limits, some will be from short-term transients -which have only been detected once. The AP Database maintains these -detections until they have been associated with one of the objects -above. The AP Database provides mechanisms by which individual -detections may be migrated back and forth between the orphan state and -association with an astronomical object. +(not representing real objects), some will be from solar system +objects for which orbits are not yet determined, some will be from +faint stars near the detection limits, and some will be from +short-term transients which have only been detected once. The AP +Database maintains these detections until they have been associated +with one of the objects above. The AP Database provides mechanisms by +which individual detections may be migrated back and forth between the +orphan state and association with an astronomical object. For every object, and all orphaned detections, the AP Database also -provides the capability to determine the images which observed the +provides the capability to determine the images containing the location of the object but for which no detection was made. The minimum set of information which must be carried for these @@ -695,14 +744,13 @@ The AP Database also stores the relationships between various -photometric systems and, in some cases, the evolution of that -relationship. It provides mechanisms to convert between the measured -instrumental magnitude of a detection with a specific filter, -detector, and telescope, and at a particular time and the implied -magnitude in the average Pan-STARRS photometry system, given a -determined set of calibrations. It also provides the capability to -convert magnitudes in one system to the magnitudes in another system; -an example of such a conversion is between the average Pan-STARRS -filter systems and the various reference systems appropriate for those -filters. +photometric systems and the evolution of that relationship. It +provides mechanisms to convert between the measured instrumental +magnitude of a detection with a specific filter, detector, and +telescope, and at a particular time and the implied magnitude in the +average Pan-STARRS photometry system, given a determined set of +calibrations. It also provides the capability to convert magnitudes +in one system to the magnitudes in another system; an example of such +a conversion is between the average Pan-STARRS filter systems and the +various reference systems appropriate for those filters. \begin{figure} @@ -710,5 +758,5 @@ \resizebox{4.5in}{!}{\includegraphics{pics/APDB}} \caption{AP DB components} -\label{fig:APDBRegions} +\label{fig:APDBComponents} \end{center} \end{figure} @@ -726,5 +774,6 @@ time and date of the detection, etc. -The IPP AP Database consists of the following components: +As shown in Figure~\ref{fig:APDBComponents}, the IPP AP Database +consists of the following components: \begin{itemize} @@ -737,32 +786,41 @@ \subsubsection{AP Database Tables} -Table~\ref{APDBTables} lists the tables used by the AP Database. The +Table~\ref{tab:APDBTables} lists the tables used by the AP Database. The contents of these tables are outlined in -Appendix~\ref{APDBTableContents}. Below, we discuss how these tables -are used by the AP Database software. Several of the tables are not -just simple tables in the database but are instead divided into many -subtables, each of which represents a portion of the sky. These -subtables may also be distributed across different computers to -distribute the processing load. +Appendix~\ref{sec:APDBTableContents}. Below, the use of these tables by +the AP Database software is discussed below. Several of the tables +are not just simple tables in the database but are instead table +groups divided into many subtables, each of which represents a portion +of the sky (a {\tt region}). These subtables may also be distributed +across different computers to distribute the processing load. + +\paragraph{Images Table Group} The {\tt Images} table group lists all of the images which provided -the data in the AP Database. These tables are subdivided by region. -In general, the images listed in this table correspond to the Chips. -This group of tables includes sufficient astrometric parameters to -represent the coordinates of the detections to a sufficient accuracy. +the data in the AP Database. These tables are subdivided by region on +the sky. In general, the images listed in this table correspond to +the Chips. This group of tables includes sufficient astrometric +parameters to represent the coordinates of the detections to a +sufficient accuracy. Parallel to the Images table is the Mosaic +table. This table is very similar to the Images table, but defines +the Mosaic which corresponds to a group of Images. The parameters +include the astrometric information needed to define the camera +distortion. + +\paragraph{Image Overlaps Table Group} The specific subtable of {\tt Images} which contains a given image is -the one which contains the center pixel \tbr{or 0,0 pixel} of that -image. An additional table group, {\tt Image Overlaps} (with the same -subtable organization as the {\tt Images} subtables), lists images -which overlap that specific subtable. Thus, given a particular -coordinate, in order to find that images which overlap that -coordinate, it is necessary to search the images in the {\tt Images} -subtable which includes that coordinate, and all images in the {\tt -ImageOverlaps} subtable for that coordinate. +the one which contains the center pixel of that image. An additional +table group, {\tt Image Overlaps} (with the same subtable organization +as the {\tt Images} subtables), lists images which overlap that +specific subtable. Thus, given a particular coordinate, in order to +find that images which overlap that coordinate, it is necessary to +search the images in the {\tt Images} subtable which includes that +coordinate, and all images in the {\tt ImageOverlaps} subtable for +that coordinate. \begin{table}[hb] \begin{center} -\caption{AP Database Tables\label{APDBTables}} +\caption{AP Database Tables\label{tab:APDBTables}} \begin{tabular}{ll} \hline @@ -770,22 +828,24 @@ {\bf Table Name} & {\bf Description} \\ \hline -Images & The images that have objects in the DB. \\ -Image Overlaps & Image regions which are touched by specific images. \\ -Objects & The objects --- average properties of multiple detections of the same object. \\ -Average Magnitudes & Average photometry in multiple filters \\ -Matched Detections & Detections of sources in an image identified with an Object. \\ -Orphaned Detections & Detections of sources in an image not identified with an Object. \\ -Non-detections & Non-detections of objects in an image. \\ -Region Table & spatial distribution of tables \\ -Filters & Filters understood by the system. \\ -Photcodes & Transformations between different photometric systems \\ -Database Machines & computers used to store the tables \\ -% Zero Points & Transformations between different photometric systems \\ -% Distortion Models & Transformations between different photometric systems \\ -% Solar System Objects & Identification of solar system objects \\ +Images & The images that have objects in the DB. \\ +Image Overlaps & Image regions which are touched by specific images. \\ +Objects & The objects --- average properties of multiple detections of the same object. \\ +Average Magnitudes & Average photometry in multiple filters \\ +Solar System Objects & Identification of solar system objects \\ +Matched Detections & Detections of sources in an image identified with an Object. \\ +Orphaned Detections & Detections of sources in an image not identified with an Object. \\ +Non-detections & Non-detections of objects in an image. \\ +Regions & spatial distribution of tables \\ +Filters & Filters understood by the system. \\ +Photcodes & Transformations between different photometric systems \\ +Zero Points & History of Zero-point \& Airmass terms \\ +Distortion Models & History of Optical Distortion terms \\ +Database Hosts & computers used to store the tables \\ \hline \end{tabular} \end{center} \end{table} + +\paragraph{Objects Table Group} The {\tt Objects} table group (also divided by region) stores the @@ -797,9 +857,13 @@ be stored in a separate table. -A related table, also divided in the same regions, is the {\tt Average -Magnitudes} table. In this table, there are multiple rows per average +\paragraph{Average Magnitudes Table Group} + +A related table, also divided into the same regions, is the {\tt +Average Magnitudes} table. In this table, there are multiple rows per object, one for each of the primary filters of interest for which photometric averaging is performed. This organization makes the number of primary (averaged) filters a configurable value. + +\paragraph{Matched Detections Table Group} The {\tt Matched Detections} table stores all of the measurements of @@ -814,9 +878,13 @@ quantities for these types of detections.) +\paragraph{Orphaned Detections Table Group} + The {\tt Orphaned Detections} table stores the detections which have not been correlated with an existing object. This table is only populated for objects below a configuration-specified signal-to-noise -limit (eg 5$\sigma$). Bright orphaned detections are assigned an +limit (e.g., 5$\sigma$). Bright orphaned detections are assigned an object and added to the {\tt Matched Detections} table. + +\paragraph{Non-detections Table Group} The {\tt Non-detections} table stores information about detection @@ -827,8 +895,10 @@ non-detection statistics. +\paragraph{Regions Table} + The {\tt Regions} table is used to subdivide the tables of images, objects, and detections, etc, as discussed above. The AP Database divides the sky into a hierarchy of regions (portions of the sky) each -of which is in turn sub-divided into smaller portions. Since nearly +of which is in turn subdivided into smaller portions. Since nearly all interactions with the AP Database performed by the IPP are limited in spatial coverage, subdividing the tables allows a specific @@ -846,7 +916,8 @@ detection data, the {\tt Regions} table allows for multiple computers to serve the database tables. The region file specifies the machine -which stores the specific table. Figure~\ref{ABDBRegions} illustrates -schematically the subdivision of the sky and the association between -different levels of the hierarchy with different subtables. +which stores the specific table. Figure~\ref{fig:APDBRegions} +illustrates schematically the subdivision of the sky and the +association between different levels of the hierarchy with different +subtables. \begin{figure} @@ -857,4 +928,6 @@ \end{center} \end{figure} + +\paragraph{Other Reference Tables} The {\tt Filters} table identifies all of the physical filters @@ -877,11 +950,10 @@ {\bf Option A:} A client chooses one of the machines and sends its -query or data to be inserted to that machine. The server then uses -the region table to determine which machines contain the relevant -portion of the sky. The data to be inserted is divided into -corresponding region chunks and sent to the appropriate servers. In -the case of queries, the queries are redirected to the appropriate -server(s). The original server may collect the results and return -them to the original client. +query or data to that machine. The server then uses the region table +to determine which machines contain the relevant portion of the sky. +Data to be added to the database is divided into corresponding region +chunks and sent to the appropriate servers. Queries are redirected to +the appropriate server(s). The original server may collect the +results and return them to the original client. {\bf Option B:} The client downloads the region table and performs the @@ -893,5 +965,6 @@ and making each server symmetric. The smaller tables (ie, Region, Filters, etc) could either be downloaded from a single server or -replicated to all AP DB servers. +replicated to all AP DB servers. For these reasons, Option A will be +used for the PS-1 IPP.. \subsubsection{AP Database engine} @@ -922,9 +995,9 @@ to the {\tt Matched Detections} table. Any faint unmatched detections are added to the {\tt Orphaned Detections} table. This division is -important because it lets us automatically associate new detections -with existing bright objects and limits the I/O volume required to -make the detections. In general, there will be many few {\tt Objects} -than {\tt Detections}, and there will be fewer bright orphans than -faint orphans. +important because it allows the automatic association of new +detections with existing bright objects while limiting the I/O volume +required to make the detections. In general, there will be many fewer +{\tt Objects} than {\tt Detections}, and there will be fewer bright +orphans than faint orphans. \paragraph{Insert Reference Objects (addrefs)} @@ -941,6 +1014,6 @@ This operation uses the overlaps of images and multiple observations of the same objects to determine the relative photometry zero-points -for a collection of images. This is a task which would be run much -more infrequently than the object insertion tasks. +for a collection of images. This is a task that wil be run much more +infrequently than the object insertion tasks. \paragraph{Determine Consistent Photometry Zero Points (uniphot)} @@ -951,12 +1024,20 @@ atmospheric stability. -\paragraph{Determine Distortion Model (mosastro)} +\paragraph{Determine Distortion and Static Astrometry Model (mosastro)} This operation uses the reference and image detections to determine an -optical distortion model for the camera. +optical distortion model for the camera and static astrometry model +components. The astrometry model includes: (1) field distortion +introduced by the telescope optics, which is a smoothly-varying +function of the field position relative to the center of the telescope +boresite coordinates. (2) focal plane geometry, which includes the +chip positions and rotations in the focal relative to the boresite, +along with chip-dependent plate-scale modifications needed to +represent tilts or warps of the individual detectors relative to the +ideal flat focal plane. . \begin{table} \begin{center} -\caption{AP Detection Classes \& Object Parameters\label{APdetections}} +\caption{AP Detection Classes \& Object Parameters\label{tab:APdetections}} \begin{tabular}{lrrrr} \hline @@ -978,22 +1059,35 @@ \end{table} -\subsubsection{Notes} - -discuss AP DB throughput issues - -how does the AP Database know about the relationship between a -collection of chips? - -what is astrometry representation in image table? 3rd order polynomial -across the chip? - -does the AP Database know about FPA, Chip, Distortion Model, etc? I -think it probably needs to if it is going to solve for distortion -models. however, this operation may be a combination of AP DB -interaction and MD DB interaction. +\subsubsection{Throughput} + +The AP Database design partly driven by the need to make the +detection-object associations quickly and to processes the incoming +detections at a sufficiently high rate to meet the throughput +requirements. For each upload of the object detections from a +complete FPA, the AP Database must match roughly $1.4 \times 10^{6}$ +detections from an FPA with roughly $6.4 \times 10^{6}$ objects, +including orphaned bright detections. This corresponds to roughly 640 +MB, if each object uses 100 bytes for its descriptive informations +(more than is currently specified in the Object table). With a +throughput of 100 MB/s for reads from a RAID, the AP Database can +perform the data read in a fraction of a second if the data is +distributed across 10 computers. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Controller} +\label{sec:Controller} + +\subsubsection{Corresponding Requirements} + +The Controller must meet the requirements specified in Section 3.4.4 +of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The design must meet +requirements 3.4.4.1 - 3.4.4.7. In particular, the Controller / Node +Agent architecture is chosen to control the I/O flow between the +Controller and the individual processes so that blocking on the I/O +from many remote processes does not saturate the Controller +processing. + +\subsubsection{Overview} \begin{figure} @@ -1027,66 +1121,69 @@ manage background tasks or if the IPP Controller should attempt to send one task per CPU and let the operating system handle the I/O -load. +load. The relationship between the different components of the +Controller is illustrated in Figure~\ref{fig:Controller} and discussed +below. \subsubsection{Nodes} -The Controller maintains a table of processing computers (`Nodes') -available to it and tracks the status of these Nodes. Nodes managed -by the IPP Controller are allowed to be in one of several states, and -the IPP Controller must interact with it in an appropriate way for -each of those states. A computer may be {\tt alive}, {\tt dead} or -{\tt off}. If the computer is {\tt alive}, it responds to commands -from the IPP Controller and may be used for tasks subject to other -constraints. If it is {\tt dead}, the computer is not responsive and -must not be used for executing tasks. The IPP Controller must -identify computers which have died (not responding) and occasionally -test them to see if they are {\tt alive} again. Computers which are -{\tt off} are not available for tasks and must not be tested. -Computers may be set to the {\tt off} or {\tt dead} states by external -subsystems; it is the responsibility of the IPP Controller to return a -computer to the {\tt alive} state if possible. +The Controller maintains a table of available processing computers +(`Nodes') and tracks the status of these Nodes. Nodes managed by the +IPP Controller are allowed to be in one of several states, and the IPP +Controller must interact with it in an appropriate way for each of +those states. A Node may be {\tt alive}, {\tt dead} or {\tt off}. +If the Node is {\tt alive}, it responds to commands from the IPP +Controller and may be used for tasks subject to other constraints. If +it is {\tt dead}, the Node is not responsive and must not be used +for executing tasks. The IPP Controller must identify Nodes which +have died (not responding) and occasionally test them to see if they +are {\tt alive} again. Nodes which are {\tt off} are not +available for tasks and must not be tested. Nodes may be set to +the {\tt off} or {\tt dead} states by external subsystems; it is the +responsibility of the IPP Controller to return a Node to the {\tt +alive} state if possible. The IPP Controller must honor requests (normally from the users) to change the mode of any computing node on demand between {\tt off} and -{\tt dead}. This would normally be done after a computer has been +{\tt dead}. This would normally be done after a Node has been rebooted and is released to the IPP Controller for its use. It must also be able to change the list of allowed tasks as requested by external commands. -Two example scenarios illustrate the transition between these states. -First, imagine a computer crashes. At this point the IPP Controller -should detect that the computer is no longer responsive and mark it -{\tt dead}. It should occasionally try to re-establish communication -with the computer, potentially with longer and longer delays between -attempts. A human could be notified if the computer seems to remain -{\tt dead} for a very long time. In another scenario, a person needs -to work on a computer. They notify the IPP Controller that the -machine is {\tt off}, perhaps with a prior notification that the -machine should be prepared to go off. When work on the machine is -complete, it should be placed in the {\tt dead} state. Only when the -person is done working and testing the machine, and tells the IPP -Controller that the machine is now {\tt dead} can the IPP Controller -attempt to re-start communications and processing on that computer. +Two example scenarios illustrate the transition between these states, +and the basic concept of operations for the IPP Controller. First, +imagine a computer crashes. At this point the IPP Controller should +detect that the Node is no longer responsive and mark it as {\tt +dead}. It should occasionally try to re-establish communication with +the Node, potentially with longer and longer delays between attempts. +A human could be notified if the Node seems to remain {\tt dead} for a +very long time. In another scenario, a person needs to work on a +Node. They notify the IPP Controller that the machine is {\tt off}, +perhaps with a prior notification that the machine should be prepared +to go off. When work on the machine is complete, it should be placed +in the {\tt dead} state. Only when the person is done working and +testing the machine, and tells the IPP Controller that the machine is +now {\tt dead} can the IPP Controller attempt to re-start +communications and re-new processing operations on that Node. \subsubsection{Node Agents} When the Controller starts, it attempts to launch a Node Agent on each -of the available processing Nodes. Modes which are not responsive are -placed marked as {\tt dead} so they may be retried. A Node Agent runs -on each of the individual nodes to execute the tasks as directed by -the Controller. The Node Agents communicate with the Controller via a +of the available processing Nodes. Nodes which are not responsive are +marked as {\tt dead} so they may be re-tried. A Node Agent runs on +each of the individual nodes to execute the tasks as directed by the +Controller. The Node Agents communicate with the Controller via a socket connection. -A Node Agent (which is only on Node in the {\tt alive} state) may be -in one of four modes: {\tt idle}, {\tt busy}, {\tt done}, {\tt crash}. -A Node Agent which is {\tt busy} currently has a task assigned to it -which is executing. The IPP Controller may only assign one task to a -Node at a time. A Node Agent which is in the {\tt idle} state may -have a task assigned to it. When the Node Agent detects that a tasks -has finished, it changes to either the {\tt done} or {\tt crash} -states depending on the outcome of the process execution. The IPP -Controller must also respect a list of task restrictions which may -require specific tasks to run on specific CPUs or exclude specific -tasks from specific CPUs. +A Node Agent (which is only running on a Node in the {\tt alive} +state) may be in one of four modes: {\tt idle}, {\tt busy}, {\tt +done}, {\tt crash}. A Node Agent which is {\tt busy} currently has a +task assigned to it which is executing. The IPP Controller may only +assign one task to a Node at a time. A Node Agent which is in the +{\tt idle} state may have a task assigned to it. When the Node Agent +detects that a tasks has finished, it changes to either the {\tt done} +or {\tt crash} states depending on the outcome of the process +execution. The IPP Controller must also respect a list of task +restrictions which may require specific tasks to run on specific CPUs +or exclude specific tasks from specific CPUs. A task being executed by the Node is run in the UNIX user space as a @@ -1100,5 +1197,5 @@ The Node Agent returns its state ({\tt idle}, {\tt busy}, {\tt done}, -{\tt crash'}) and the exit status of the current processing task, if +{\tt crash}) and the exit status of the current processing task, if available. The reported exit state, if the process has completed without crashing, is the UNIX exit state reported by the task: 0--256 @@ -1120,13 +1217,13 @@ \paragraph{Kill task } -The Node Agent should send a kill signal (signal 9 or 15) to the -current processing task. When the processing task has exited, the -Node Agent should set its state to {\tt crash}. +The Node Agent should send a kill signal (\code{KILL} or \code{TERM}) +to the current processing task. When the processing task has exited, +the Node Agent should set its state to {\tt crash}. \paragraph{Clear task} The Node Agent should set its state {\tt idle}. If a processing stage -is currently running, it should be killed (signal 9 or 15) before the -task is cleared. +is currently running, it should be killed (\code{KILL} or \code{TERM}) +before the task is cleared. \paragraph{Start processing stage} @@ -1145,10 +1242,9 @@ valid resource regardless of the node on which the task is executed. Input and output data resources must be unique where necessary to -avoid conflicts. \tbd{It is the responsibility of the programs to -wait for network lags (ie, NFS delays)}. The IPP Controller gives -each task a unique identifier, which is returned to the requesting -entity. The requestor may then use that ID to obtain status -information on that task or to send control signals to the specific -task. +avoid conflicts. It is the responsibility of the task to wait for +network lags (ie, NFS delays). The IPP Controller gives each task a +unique identifier, which is returned to the requesting entity. The +requestor may then use that ID to obtain status information on that +task or to send control signals to the specific task. Task requests may specify a desired node for the task execution. The @@ -1163,9 +1259,12 @@ Task requests may specify an urgency level. The IPP Controller -determines the priority of the task on the basis of both the priority +determines the priority of the task on the basis of both the urgency and the age of the request. An executing task must be completed on a CPU before any new task is started on that CPU, regardless of -priority. Tasks may be assigned a priority of 0 in which case they -are maintained in the queue and never executed. +priority. The urgency levels range from 0 to 2. Tasks with an +urgency of 1 are scheduled whenever they reach the top of the stack. +Tasks with an urgency of 2 are sent immediately to the top of the +stack. Tasks assigned a priority of 0 are maintained in the queue and +never executed. It may be useful for the Controller to distinguish between tasks @@ -1185,5 +1284,5 @@ completed. -\subsubsection{External Interfaces} +\subsubsection{Controller Interfaces} The IPP Controller must accept commands from other IPP subsystems. @@ -1237,4 +1336,15 @@ \subsection{Scheduler} +\subsubsection{Corresponding Requirements} + +The Scheduler must meet the requirements specified in Section 3.4.5 of +the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The design must meet +requirements 3.4.5.1 - 3.4.5.7. In particular, the Task / Test +division is chosen to prevent the Scheduler from blocking while an +analysis process is performed. Scheduling requirements will be met by +defining appropriate Test periods for the different Tasks. + +\subsubsection{Overview} + The IPP is responsible for a variety of analysis jobs: processing of the science images through several stages; routine assessment of the @@ -1250,9 +1360,10 @@ and initiate the actions. -The IPP Scheduler acts as an intermediary between several components -of the IPP and also between the IPP and external agents such as OTIS -and the users who must monitor the behavior of the IPP. The IPP -Scheduler may be viewed as the central brain of the IPP. -Figure~\ref{Scheduler} illustrates the design of the IPP Scheduler. +The IPP Scheduler acts as an interface between several components of +the IPP and also between the IPP and external agents such as OTIS and +the users who must monitor the behavior of the IPP. The IPP Scheduler +may be viewed as the central brain of the IPP. +Figure~\ref{fig:Scheduler} illustrates the design of the IPP +Scheduler. \subsubsection{Scheduler Tasks and Tests} @@ -1281,5 +1392,5 @@ \begin{center} \resizebox{6in}{!}{\includegraphics{pics/Scheduler}} -\caption{ \label{Scheduler} IPP Scheduler} +\caption{ \label{fig:Scheduler} IPP Scheduler} \end{center} \end{figure} @@ -1288,9 +1399,9 @@ While the IPP Scheduler chooses the tasks to be performed, it is the IPP Controller's responsibility to manage the specific tasks executing -on a given processing node. This division of responsibilites allows -us to isolate and encapsulate the functionality of the IPP Scheduler -and the IPP Controller. With this separation, the IPP Controller does -not need to have any information about the details of the tasks which -it executes, while the IPP Scheduler does not need to monitor the +on a given processing node. This division of responsibilities allows +the different functionalities of the IPP Scheduler and the IPP +Controller to be isolated and encapsulated. With this separation, the +IPP Controller does not information about the details of the tasks it +executes, while the IPP Scheduler does not need to monitor the computer hardware. @@ -1298,7 +1409,7 @@ bi-directional; the IPP Scheduler sends tasks to the IPP Controller, while the IPP Controller informs the IPP Scheduler of the outcome of -those tasks. It is not specified whether the IPP Scheduler and IPP -Controller are components of a single software system or interacting -but distinct software components. +those tasks. For the PS-1 IPP, the IPP Scheduler and the IPP +Controller are distinct, interacting software components. The +interface mechanisms are described in Section~\ref{sec:interfaces}. \subsubsection{Task Rules} @@ -1306,9 +1417,9 @@ The IPP Scheduler takes as input a collection of rules which define the dependency of tasks on certain tests. The IPP Scheduler must -choose between several types of analysis tasks based on those ruls and -on results of the tests. The timescale on which different tasks (and -their related tests) are executed may vary from 10s of seconds to -hours, days, or even week. The list of tasks which the IPP Scheduler -must decide between, and the relevant timescale, follow: +choose between several types of analysis tasks based on those rules +and on results of the tests. The timescale on which different tasks +(and their related tests) are executed may vary from 10s of seconds to +hours, days, or even as long as a week. The list of tasks which the +IPP Scheduler must decide between, and the relevant timescale, follow: \begin{itemize} \item moving data from the Summit pixel server ($\sim 30$ second timescales) @@ -1318,50 +1429,61 @@ \item constructing new detrend images ($\sim$ weekly) \end{itemize} -The scheduler may be viewed as a complex state machine. Our goal is +The scheduler may be viewed as a complex state machine. The goal is to design the scheduler so that rules may be specified independently -from the engine which parses the rules to detemine which specific jobs +from the engine which parses the rules to determine which specific jobs to send to the controller. \subsubsection{User Interface} -The IPP Scheduler provides a user interface which allows a human +The IPP Scheduler shall possess a user interface which allows a human operator, or other processes, to monitor the current state of the Scheduler. Users have the option to specify that a particular task or -set of tasks is of higher or lower priority than the norm, or to -schedule a particular tasks on a different timescale from the basic -rule. - -The IPP Scheduler defines the operating state of the IPP. When the -IPP is in the {\em automatic state}, the IPP Scheduler performs the +set of tasks is of higher or lower urgency (as defined in +Section~\ref{sec:Controller}) than the norm, or to schedule a +particular tasks on a different timescale from the basic rule. + +The IPP Scheduler defines the operating state of the IPP and shares +the same set of states: +\begin{itemize} +\item active state +\item interactive state +\item paused state +\end{itemize} +When the IPP Scheduler is in the {\em active state}, it performs the most appropriate of all possible tasks at a particular time. When the -IPP is in the {\em interactive state}, the IPP Scheduler performs only -the requested action regardless of the outcome of the decision trees. -In addition, in the interactive state, the IPP Scheduler must only -perform the requested actions and not attempt to perform the other -normally-required actions. The only exception to this exclusion is -that, in the interactive state, data is still copied from the summit -system. An additional IPP state is the {\em paused state}, intended -for tests or maintenance, in which case the IPP Scheduler does not -perform even the data copy tasks. Every task is performed on demand -by the user. A user command sets the IPP Scheduler in one of these -three states, {\em automatic}, {\em interactive}, and {\em paused}. +IPP Scheduler is in the {\em interactive state}, it performs only a +specific requested action regardless of the outcome of the decision +trees. In addition, in the interactive state, the IPP Scheduler must +only perform the requested actions and not attempt to perform the +other normally-required actions. The only exception to this exclusion +is that, in the interactive state, data is still copied from the +summit system. An additional IPP state is the {\em paused state}, +intended for tests or maintenance, in which case the IPP Scheduler +does not perform even the data copy tasks. Every task is performed on +demand by the user. A user command sets the IPP Scheduler in one of +these three states, {\em active}, {\em interactive}, and {\em paused}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{System Design : Science Analysis Tasks and Stages} - -In this section, we discuss the design of the science analysis stages -which perform the fundamental image analysis steps of the IPP. The -IPP science image processing stages perform analyses on the night-sky +\label{sec:AnalysisStages} + +This section describes the design of the science analysis stages which +perform the fundamental image analysis steps of the IPP. The IPP +science image processing stages perform analyses on the night-sky science images to extract the science data from these images. These -consist of: Phase 1, the image processing preparation stage; Phase 2, -the image reduction stage; Phase 3, the exposure analysis stage; and -Phase 4, the image combination stage. These analysis tasks must -process the images in a timely manner so that the incoming data stream -will not overload the IPP Image Server. The decision to execute a -specific pipeline for a specific dataset is made by the Scheduler, -which sends the infomation to the Controller. The Controller executes -the pipeline for the data on an appropriate machine and monitors the -success or failure of the processing stage. +consist of: +\begin{itemize} +\item Phase 1, the image processing preparation stage, +\item Phase 2, the image reduction stage +\item Phase 3, the exposure analysis stage +\item Phase 4, the image combination stage. +\end{itemize} +These analysis tasks must process the images in a timely manner so +that the incoming data stream will not overload the IPP Image Server. +The decision to execute a specific pipeline for a specific dataset is +made by the Scheduler, which sends the information to the Controller. +The Controller executes the pipeline for the data on an appropriate +machine and monitors the success or failure of the processing stage. The analysis stages are written as UNIX commands, which may be @@ -1384,6 +1506,6 @@ The recipe is loaded as part of the runtime configuration information -loaded when the analysis script starts. We define four levels of -runtime configuration information. The {\tt site} configuration +loaded when the analysis script starts. Four levels of runtime +configuration information are defined. The {\tt site} configuration defines values specific to the particular installation of the software. For example, the name of the machine which hosts the @@ -1408,5 +1530,5 @@ also be specified on the command line. Examples of the recipe and other runtime configuration options are given in -Appendix~\ref{RuntimeConfig}. +Appendix~\ref{sec:RuntimeConfig}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1422,5 +1544,5 @@ magnification. The guide star coordinates are loaded from the Metadata database. These calculations are performed by comparing the -observed guide star detector coodinates with the known astrometic +observed guide star detector coordinates with the known astrometric positions of these same stars as reported by an external astrometric reference. The accuracy of the resulting astrometric solution is @@ -1440,5 +1562,5 @@ detection to determine the detector coordinates of those bright stars which are not saturated but which are significantly above the -background level. By targetting known locations in the image files, +background level. By targeting known locations in the image files, only a small amount of data will have to be read. @@ -1455,9 +1577,12 @@ phase. It is acceptable for a small number of invalid overlaps to be identified as these will be excluded in Phase 4. Static Sky cells -which do not have sufficient science image overlap \tbr{$< 5\%$} need -not be processed because the few new measured pixels do not add +which do not have sufficient science image overlap ($< 5\%$) need not +be processed because the few new measured pixels do not add significantly to the Static Sky. -\subsubsection{Notes} +\subsubsection{Examples} + +Examples of Phase 1 as called from the command line, with different +types of images: \begin{verbatim} @@ -1505,5 +1630,5 @@ \subsubsection{Load Images} -The Phase 2 analysis must load the science image to be analysed into +The Phase 2 analysis must load the science image to be analyzed into memory, as well as the corresponding metadata (either from the image header and/or from the IPP Metadata Database). It must use the @@ -1520,5 +1645,5 @@ Science images which have been obtained with Orthogonal-Transfer -Guiding have had thier pixel response smoothed by the image correction +Guiding have had their pixel response smoothed by the image correction motion. For these images, some of the detrend images need to be convolved by the same OT kernel, so that they accurately represent the @@ -1539,15 +1664,15 @@ fringe frame(s) by the OT convolution kernel. Specific flags in the static bad pixel mask are also grown by the outline of the OT -convolution kernel (see Section \ref{ap:masks}). +convolution kernel (see Section~\ref{sec:masks}). \subsubsection{Bias Correction / Overscan Subtraction} The image bias must be subtracted. Since different detectors behave in -different ways, several options for modelling the bias are available. +different ways, several options for modeling the bias are available. The bias is measured from the image overscan region. The bias subtraction method must be capable of subtracting a single constant from the complete image, or to subtract a 1-D bias which varies as a function along the overscan. The function used to represent the -overscan region may be a spline or a chebychev polynomial derived from +overscan region may be a spline or a Chebychev polynomial derived from the data values along the overscan. The values used to determine both the single constant or the inputs to the spline and polynomial fits @@ -1562,4 +1687,5 @@ \subparagraph{Flag bad and saturated pixels} +\label{sec:masks} A static bad pixel mask is used to identify pixels which are known to @@ -1567,5 +1693,5 @@ image. Bad pixels which are charge traps are grown by the extent of the OT convolution kernel. Bad pixels above a charge trap (i.e.\ bad -colums) must not be grown, since they were not affected by pixel +columns) must not be grown, since they were not affected by pixel shifting, but only became bad at read-out. @@ -1633,5 +1759,5 @@ artifacts generated by bright stars: bleeding columns, ghosts, or other localized reflection effects. This process also produces a -superbinned image of the background map which may be used as a +super-binned image of the background map which may be used as a debugging diagnostic. @@ -1647,13 +1773,13 @@ \subsubsection{Detect and Measure objects} -After the image have been processed by the preceeding steps, the Phase +After the image have been processed by the preceding steps, the Phase 2 analysis performs a basic object detection analysis. Objects on the flat-fielded object image are found, and general parameters are measured. Object detection is performed at several stages by the IPP, with different object parameters measured in each case. -Table~\ref{APdetections} gives a list of the different detection +Table~\ref{tab:APdetections} gives a list of the different detection stages and the object parameters measured for those stages. For the Phase 2 analysis, the object parameters are: the object centroid and -the position covarience matrix, the instrumental PSF magnitude and +the position covariance matrix, the instrumental PSF magnitude and error, local background level and error, a measurement of the star-galaxy separation, and a measurement of the object shape @@ -1693,5 +1819,5 @@ (either stars with poorly determined proper motion or spurious matches). The resulting astrometric solution is consistent across the -OTA field to within \tbr{0.2 arcsec}. +OTA field to within 1.0 arcsec. \subsubsection{Perform Photometry} @@ -1719,5 +1845,5 @@ %\begin{center} %\resizebox{6in}{!}{\includegraphics{pics/phase2}} -%\caption{ \label{phase2} Phase 2 dataflow - this diagram is old: update} +%\caption{ \label{fig:phase2} Phase 2 dataflow - this diagram is old: update} %\end{center} %\end{figure} @@ -1753,5 +1879,5 @@ center, followed by a rotation to the average rotation of the FPA and adjustment for the central plate scale. The free parameters in this -stage are the boresite coordiates ($R_o, D_o$), the field rotation +stage are the boresite coordinates ($R_o, D_o$), the field rotation ($\theta_o$) and the plate scale ($\rho_o$), and are fitted in Phase 1. These tangent plane coordinates are then distorted by the optical @@ -1778,5 +1904,5 @@ local reference catalog. This analysis may only be performed if a local reference is available. Note that improved relative photometry -calculations may be performed in the absense of a reference catalog on +calculations may be performed in the absence of a reference catalog on the basis of image overlaps in the AP Database {\em after} the detections have been added to the Database. Such a relative @@ -1784,5 +1910,5 @@ performed as an independent analysis process. Given the presence of a local photometry reference, the zero point variations across the field -may be measured, and possibly modelled. If the zero-point variations +may be measured, and possibly modeled. If the zero-point variations are excessive, then the image is marked as non-photometric by the analysis. @@ -1810,5 +1936,5 @@ same number of pixels as an OTA (4k x 4k) and represent a portion of a local tangent plane projection. In order to meet the image -degredation requirements, the pixel scale of the static sky is planned +degradation requirements, the pixel scale of the static sky is planned to be 0.2\arcsec, somewhat smaller than the 0.3\arcsec\ raw image pixel scale. @@ -1849,10 +1975,10 @@ between the input image and the static sky image. This will be done by solving for a best-fit image kernel which minimizes the difference -image using a technique equivalent to the Allard-Lupton method. The -modification we make is that, rather than represent the components of -the image difference kernel as a combination of Gaussians, we will -represent the kernel as a combination of pixels. This method also -automatically determines a photometric match between the static sky -image and the input science image. +image using a technique equivalent to the Allard-Lupton method. One +modification for the IPP is to represent the kernel as a combination +of independent pixels rather than represent the components of the +image difference kernel as a combination of Gaussians. This method +also automatically determines a photometric match between the static +sky image and the input science image. \subsubsection{Object Detection and Measurement} @@ -1860,8 +1986,8 @@ Objects in the difference image are detected and a specific set of object parameters are measured from these detections. -Table~\ref{APdetections} gives a list of the different detection +Table~\ref{tab:APdetections} gives a list of the different detection stages and the object parameters measured for those stages. For the Phase 4 difference image (P4$\Delta$), the measured object parameters -consist of: the object centroid and the position covarience matrix, +consist of: the object centroid and the position covariance matrix, the instrumental PSF magnitude and error, local background level and error, a measurement of the star-galaxy separation, and a measurement @@ -1878,8 +2004,8 @@ Objects in the cleaned, summed image are detected and a specific set of object parameters are measured from these detections. -Table~\ref{APdetections} gives a list of the different detection +Table~\ref{tab:APdetections} gives a list of the different detection stages and the object parameters measured for those stages. For the Phase 4 summed image (P4$\Sigma$), the measured object parameters -consist of: the object centroid and the position covarience matrix, +consist of: the object centroid and the position covariance matrix, the instrumental PSF magnitude and error, local background level and error, a measurement of the star-galaxy separation, a measurement of @@ -1921,5 +2047,5 @@ %\begin{center} %\resizebox{6in}{!}{\includegraphics{pics/phase4}} -%\caption{ \label{phase4} Phase 4 dataflow} +%\caption{ \label{fig:phase4} Phase 4 dataflow} %\end{center} %\end{figure} @@ -1940,7 +2066,7 @@ The Calibration analysis stages may be performed on whatever timescales are appropriate and necessary to maintain the quality and -relevance of the calibration images. Below, we list the specific -calibration data which must be constructed in the calibration analysis -stages. +relevance of the calibration images. The specific calibration data +which must be constructed in the calibration analysis stages is listed +below. The IPP must generate basic calibration images using the raw bias, @@ -2073,5 +2199,5 @@ thin-film interference must also be detected and corrected. Models of this background structure may be a necessary input to the correction -proceedure. The IPP must have the capability of generating image +procedure. The IPP must have the capability of generating image models of the large-scale structure patterns observed with the telescope @@ -2086,5 +2212,5 @@ moved to a variety of locations on the detector in a sequence of images. The flat-field correction frames analysis stage makes use of -targetted observations following a specified dither pattern, and +targeted observations following a specified dither pattern, and extracts the photometered objects from the AP Database to determine the necessary photometric corrections. The resulting image is applied @@ -2092,5 +2218,5 @@ performed by applying the correction to the basic master flat-field image, applying that flat-field image to the dithered photometry -observations, and performing the object detections. Comparion of the +observations, and performing the object detections. Comparison of the photometry of individual stars at different locations on the mosaic will demonstrate the consistency of the flat-field image. @@ -2110,7 +2236,7 @@ \section{System Design : Miscellaneous Tasks} -In this section, we discuss additional operations which are performed -by the IPP but which do not fall under the analysis of the science -images or the creation of the calibration images. +This section discusses additional operations which are performed by +the IPP but which do not fall under the analysis of the science images +or the creation of the calibration images. \subsection{Retrieval} @@ -2128,6 +2254,6 @@ performed in the real-time analysis. The currently envisioned parameters to be measured for every object are listed in -Table~\ref{APdetections}. The parameters include the object centroid -and the position covarience matrix, the instrumental PSF magnitude and +Table~\ref{tab:APdetections}. The parameters include the object centroid +and the position covariance matrix, the instrumental PSF magnitude and error, local background level and error, a measurement of the star-galaxy separation, a measurement of the object shape ($\sigma_x, @@ -2200,23 +2326,31 @@ \subsection{Pan-STARRS Library} -The Pan-STARRS Library will consist of C structures describing the basic -data types needed by the IPP and C functions which perform the basic -data manipulation operations. Note that a subset of the library +The Pan-STARRS Library will consist of C structures describing the +basic data types needed by the IPP and C functions which perform the +basic data manipulation operations. Note that a subset of the library functions will be provided with SWIG interfaces as well to allow for their use in the creation of the processing stages. Examples of the -Pan-STARRS Library are fourier transforms and transforming between pixel -and celestial coordinates. - -\subsection{Modules} +Pan-STARRS Library are Fourier transforms and transforming between +pixel and celestial coordinates. The details of the Pan-STARRS +Library are specified in the document Pan-STARRS IPP PSLib +Supplementary Design Requirements Specification (PSDC-430-007), which +also addresses coding requirements detailed in the IPP PS-1 SRS +(PSDC-430-005), Section 3.3. + +\subsection{IPP Modules} The IPP analysis stages are broken down into modules which represent specific functional operations. The modules will be written in C -using the Pan-STARRS Library functions and will be grouped into a Pan-STARRS -Module Library. The modules will be provided with SWIG interfaces to -all public APIs for their use in processing stages. Examples of -modules are overscan subtraction and image combination. Some modules -(e.g.\ find objects on an image) will be used by multiple stages. - -\subsection{Stages} +using the Pan-STARRS Library functions and will be grouped into a +Pan-STARRS Module Library. The modules will be provided with SWIG +interfaces to all public APIs for their use in processing stages. +Examples of modules are overscan subtraction and image combination. +Some modules (e.g.\ find objects on an image) will be used by multiple +stages. The details of the Pan-STARRS Modules are specified in the +document Pan-STARRS IPP Modules Supplementary Design Requirements +Specification (PSDC-430-012), which also addresses coding requirements +detailed in the IPP PS-1 SRS (PSDC-430-005), Section 3.3. + +\subsection{IPP Stages} The major IPP processing tasks are organized into stages, which @@ -2232,4 +2366,5 @@ \section{Interfaces} +\label{sec:interfaces} \subsection{Internal Interfaces} @@ -2256,6 +2391,6 @@ FITS Tables will be used to store and transport tabular data, -especially large queries from database subsystems. The Autocoding -technique discussed in Appendix~\ref{Autocode} is used to define many +especially large queries from database subsystems. The Auto-coding +technique discussed in Appendix~\ref{sec:AutocodeIO} is used to define many different table interactions. @@ -2269,9 +2404,17 @@ interface to the databases. +Within IPP and Pan-STARRS in general, process-to-process communication +will be defined through auto-coded APIs which support a limited and +validated communication protocol. The APIs will be coded based on a +table which defines the allowed command set and the grammar to be +used. This mechanism will allow a single code block to define +inter-process communication methods for many Pan-STARRS subsystems, +including, within the IPP, the Scheduler-Controller communications. + \subsection{External Interfaces} This subsection describes the interfaces between the IPP and other Pan-STARRS systems and the external clients. The interfaces are -illustrated in Figure~\ref{overview}. +illustrated in Figure~\ref{fig:overview}. \subsubsection{OTIS} @@ -2294,29 +2437,37 @@ \subsubsection{PSPS} -The details of the transfer mechanism have \tbd{not been worked out}. -The data to be transfered include: +Data will be sent to PSPS from the IPP as part of a daily or weekly +analysis process on the Static Sky. The data will be pushed from the +IPP to PSPS when they are available. The data to be transfered +include: \begin{itemize} -\item Static Sky images -\item Postage Stamps -\item Metadata tables -\item Detections \& Object associations. +\item Static Sky images - to be transferred as FITS images or + FITS triangular image regions. +\item Postage Stamps - to be transferred as FITS images. +\item Metadata tables - to be transferred as FITS tables +\item Detections \& Object associations - to be transferred as FITS tables. \end{itemize} \subsubsection{MOPS} -The details of the transfer mechanism have \tbd{not been worked out}. -The data to be transfered include: +Data will be sent to MOPS from the IPP as part of the Phase 4 +analysis. The data will be pushed from the IPP to MOPS when they are +available. The data to be transfered include: \begin{itemize} -\item Image Metadata tables -\item Orphaned Detections +\item Image Metadata tables - to be transferred as FITS tables +\item Orphaned Detections - to be transferred as FITS tables \end{itemize} \subsubsection{Other Preferred Client Science Pipelines} -The details of the transfer mechanism have \tbd{not been worked out}. +These cannot be completely defined until the Clients are defined and +their requirements are specified. The expectation is that the data +products will be the same as for the MOPS. The data will be pushed +from the IPP to the Client Science Pipeline when they are available. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \section{Computer Hardware} +\label{sec:Hardware} \subsection{PS-1 Cluster Design} @@ -2333,125 +2484,159 @@ support the Metadata DB and the AP DB. -The IPP PS-1 SRS (PSDC-xxx) specifies the processing throughput -requirements for the IPP. We have performed benchmark tests of the -processing needs in order to achieve this throughput. The details of -this study are presented in the IPP Hardware Analysis (PSDC-xxx), -which we summarize here. The analysis measures the processing time -(excluding I/O) for both Phase 2 and Phase 4 on an Intel Pentium 4 -processor, and expresses the processing time in GHz-seconds, under the -assumption that a machine with the same architecture and twice the -processor speed with perform the same analysis in half the time. This -is probably a valid assumption within a limited range on hardware -using the same architecture. We independently find that 32-bit Pentium -processors perform somewhat slower (up to a factor of 2) than -equivalently rated 64 bit Opeteron processors. This discrepancy makes -our numbers somewhat conservative, but may only compensate for the -simplistic analysis we have performed. - -Our benchmarks show that the Phase 2 analysis takes 12000 GHz-seconds -for a complete major frame (4 FPAs) while the Phase 4 analysis takes -7800 GHz-seconds for the same major frame. We also examine the total -data I/O required for each processing node both locally to disk and -across the network to other machines. These numbers in turn depend on -whether the data is optimally stored on the OTA nodes (raw images -matched to their calibration images) or if the data are randomized. -There are also differences in the analysis for how many bits and -images are used in the processing. For PS-1, the `minimal' data set -is approrpiate, resulting in a total Phase 2 I/O of 21 GBs per major -frame and a total Phase 4 I/O of 36 GBs. We will use the randomized -numbers as a conservative estimate, and assume the network is the -dominant I/O bottleneck. - -The analysis assumes each CPU is associated with one RAID array -(maximum throughput 110 MB/sec) and one network controller (maximum -throughput 70 MB/s) and that each one is a 2.2 GHz processor. In this -case, given the CPU load and I/O throughput above, the Phase 2 will -require a total of 190 seconds of I/O and 5500 seconds of processing -distributed across the cluster. Likewise, the Phase 4 analysis will -require a total of 330 sec of I/O and 3500 seconds of processing. -Given the 160 seconds available per major frame, these numbers imply a -total of 63 processors are needed to keep up with the processing and -I/O load. - -The other major driver on the IPP PS-1 cluster are the data storage -requirements. We are required to store the entire AP Survey data and -the IVP data, and to have storage enough to represent the Static Sky -by the end of the two year mission. These storage requirements as a -function of time are shown in Figure~\ref{StorageProfile}. Based on -the PS-1 Design Reference Mission (PSDC-xxx), by the end of the -second year, we will have total storage needs of 850 TB for raw images -and the Static Sky, and an additional \tbd{XXX} TB for the AP DB -storage. - -To meet these requirements, we have designed the IPP cluster to use -fat bricks which will be capable of holding 24 disks each. Before -PS-1 goes on line, we will purchase enough disks to fill 1/3 of the -disk slots. After 9 months (2006 Sept), we will purchase the next 1/3 -of the disks, and the remaining disks 9 months after that (2007 June). -We have made conservative estimates of the available disk sizes at -these purchase dates (400 GB, 600 GB, and 900 GB), allowing us to -determine the number of computers needed to meet the storage -specification. We will purchase 80 computers, with the storage -profile shown in the figure, ending at a total capacity (after -discounting volume for RAID overhead and binary vs digital terabytes) -of 950 TBs. The 80 computers will easily meet the processing and I/O -requirements given the above need to 63 processors. - -There are two competing trades we will also want to make. First, we -will want to duplicate data to multiple machines in the network to -protect against catastrophic failures on a single machine. This -double the total data space needed. To compensate, however, we will -also employ compression to data, especially data which is older. -These two factors will tend to cancel each other, so we have ignored -both in out calculations above. - -\tbd{switch information} +The IPP PS-1 SRS (PSDC-430-005) specifies the processing throughput +requirements for the IPP. Benchmark tests of the IPP processing +algorithms have been used to drive the design needed to achieve the +throughput requirements. The details of this study are presented in +the IPP Computational Challenge (PSDC-400-006), summarized here. The +analysis measures the processing time (excluding I/O) for both Phase 2 +and Phase 4 on an Intel Pentium 4 processor, and expresses the +processing time in GHz-seconds, under the assumption that a machine +with the same architecture and twice the processor speed will perform +the same analysis in half the time. This is probably a valid +assumption within a limited range on hardware using the same +architecture. Independent tests show that 32-bit Pentium processors +perform somewhat slower (up to a factor of 2) than equivalently rated +64 bit Opteron processors. This discrepancy makes the measured +numbers somewhat conservative, and compensates for the simplified +analysis performed. The benchmarks show that the Phase 2 analysis +takes 12000 GHz-seconds for a complete major frame (4 FPAs) while the +Phase 4 analysis takes 7800 GHz-seconds for the same major frame. + +The total data I/O required for each processing node, both locally to +disk and across the network to other machines, has also been measured. +These numbers in turn depend on whether the data is optimally stored +on the OTA nodes (raw images matched to their calibration images) or +if the data are randomized across the storage nodes. There are also +differences in the analysis for the number of bits per pixel and the +number of calibration images used in the processing. For PS-1, the +`minimal' data set is appropriate, resulting in a total Phase 2 I/O of +21 GBs per major frame and a total Phase 4 I/O of 36 GBs. The +randomized numbers are used as a conservative estimate, under the +assumption the network, not local disk access, is the dominant I/O +bottleneck. + +The analysis assumes each CPU (rated at 2.2 GHz) is associated with +one RAID array (maximum throughput 110 MB/sec) and one network +controller (maximum throughput 70 MB/s). In this case, given the CPU +load and I/O throughput above, Phase 2 will require a total of 190 +seconds of I/O and 5500 seconds of processing distributed across the +cluster. Likewise, the Phase 4 analysis will require a total of 330 +sec of I/O and 3500 seconds of processing. Given the 160 seconds +available per major frame, these numbers imply a total of 63 +processors are needed to keep up with the processing and I/O load. + +The other major driver on the IPP PS-1 cluster is the data storage +requirements. It is necessary to store the raw images from the entire +AP Survey, the MOPS Verification Program (MVP) and the IPP +Verification Program (IVP), and to have storage enough to represent +the Static Sky by the end of the two year mission. These storage +requirements as a function of time are shown in +Figure~\ref{fig:StorageProfile}. Based on the PS-1 Design Reference +Mission (PSDC-230-001), by the end of the second year, the total +storage requirements for raw images and the Static Sky will be 850 TB, +along with and an additional 55 TB needed for the AP DB storage + +To meet these requirements, the IPP cluster is designed to use fat +bricks which will be capable of holding 24 disks each. The 5U / 24 +disk rack mount computer cases are one of the highest density +solutions currently available. A 4U / 36 disk box is also available +and will be considered. The disk purchases will be staggered in three +waves. Before PS-1 goes on the sky, the first 1/3 of the disks (600 +disks total) will be purchased. Since the lead time for disks is +fairly short, the purchase will be made only when other portions of +Pan-STARRS are clearly on a timeline to success. After 9 months +(tentatively 2006 September), the next 1/3 of the disks will +purchased, and the remaining disks 9 months after that (tentatively +2007 June). Using conservative estimates of the available disk sizes +at these purchase dates (400 GB, 600 GB, and 900 GB), and allocating 1 +of 12 disks to the RAID and 10\% of the volume to file system and +binary Gigabyte overheads, the disk purchases outlined above result in +a total volume after the last purchase of 950 TB. This meets the +requirements with 10\% spare excess. The disk volume profile is also +shown in Figure~\ref{fig:StorageProfile} and shows that the disk space +will be available in the time it is required. + +The total number of computers to be purchased is 80. This provides +the 1800 disk slots and more than enough processors to meet the +processing requirements. This also leaves 5 live spare machines. + +There are two details which are not included in the analysis above: +compression and replication. Compression of the older raw data will +reduce the volume requirements by a factor of roughly two. However, +replication of the data across the network is necessary to ensure the +data against catastrophic failures on a single machine. Replication +doubles the total data space needed. These two factors will tend to +cancel each other, and are ignored in the calculations above. + +The IPP PS-1 clusters will have the following allocations of computers +from this cluster: +\begin{itemize} +\item Phase 2 Nodes: 32 +\item Phase 4 Nodes: 30 +\item AP Database: 10 +\item Metadata Database: 1 +\item Image Server Database: 1 +\item Controller / Scheduler: 1 +\end{itemize} +This distribution meets the projections for computational power for +each of these data systems, and leaves 5 computers as live spares for +redundancy. \subsection{PS-1 Cluster Expected Reliability} -With 80 computers and 1920 disks, we must be cautious about component -failures and their impact on operations and data integrity. There are -several factors which mitigate our exposure to hardware failures. -First, the use of RAID controllers and RAID-5 striping of the data -will protect the data on a single RAID set against the failure of a -single disk in the array. Second, our plan to have duplication across -the cluster will protect us against catastrophic failures. Finally, -the flexibility of the distributed computing plan makes it trivial to -handle the loss of individual machines as the system can automatically -redistribute the load across the cluster. - -The components which are most likely to fail in our experience are, in -order: hard drives, ram, power supplies, and other components. The -hard drive failure rate is by far the dominant concern as it -potentially affects the data integrity. +With 80 computers and 1920 disks, component failures are inevitable. +The cluster design and management must be chosen to minimize their +impact on operations and data integrity. + +There are several factors which reduce the cluster's exposure to +hardware failures. First, the use of RAID controllers and RAID-5 +striping of the data will protect the data on a single RAID set +against the failure of a single disk in the array. Second, +duplication of data across the cluster will protect against +catastrophic failures of the array (loss of two disks, loss of the +array controller card). Finally, the flexibility of the distributed +computing plan minimizes the impact the loss of individual machines +has on operations by making changes in the data and processing +assignments on the cluster a trivial matter. + +The components which are most likely to fail in the experience of our +team are, in order: hard drives, RAID controllers, ram, power +supplies, and other components. The hard drive and RAID controller +failure rates are by far the dominant concerns as they potentially +affects the data integrity. Most sources (REFS: UCSD article, Samsung White Paper) currently imply hard disk failure rates (MTBF) in the range 400,000 hours and 500,000 -hours. We take these as an upper limit, and instead adopt a -conservative value of 100,000 hours. With 1920 disk, this MTBF -implies a failure of one disk every 2.2 days. Since the disks are in -a RAID which reports the disk failures immediately and drops the array -into degraded mode, these failures will not have a huge impact on the -operations, and recovery time is only 10s of minutes. This failure -rate implies that we should be checking for hard disk failures daily. -\tbd{is it necessary to catch failures at night or can the system run -with a degraded disk?}. A catastrophic failure for the array would -require two of the 12 disks to fail before the first failed disk is -replaced. If we assume that maintainence is poor and it is possible -for a disk to take 1 week to be replaced, we calculate a probability -of a catastrophe of 1.8\% each time a disk fails. Combined with the -disk failure rate, we can expect a RAID catastrophe 6 times over the 2 -year operation of PS-1. We can use these numbers as a guideline for -our level of support needed to avoid these RAID failures. Note that -these 6 failures should not cause loss of data since the data is -duplicated across the cluster, but they require over 1 day for -recovery (as the entire array must be replicated across the network). - -\subsection{PS-1 Cluster Support} +hours. These are used as an upper limit, with the more historically +conservative value of 100,000 hours used instead. With 1920 disk, +this MTBF implies a failure of one disk every 2.2 days. Since the +disks are in a RAID which reports the disk failures immediately and +drops the array into degraded mode, these failures will not have a +huge impact on the operations, and recovery time is only 10s of +minutes. This failure rate implies that the maintenance plan must +include checks for hard disk failures on a daily basis, and should +make use of email notification and early warning information (ie, +SMART messages). + +A catastrophic failure for the array would require two of the 12 disks +to fail before the first failed disk is replaced. Assuming that +maintenance is poor and it is possible for a disk to take 1 week to +be replaced, the probability of a catastrophe is 1.8\% each time the +first disk fails. Combined with the disk failure rate, RAID +catastrophes are expected 6 times over the 2 year operation of PS-1. +These numbers can be used as a guideline for the level of support +needed to avoid these RAID failures. Note that these 6 failures +should not cause loss of data since the data is duplicated across the +cluster, but they require over 1 day for recovery (as the entire array +must be replicated across the network). + +A detailed IPP computer cluster commissioning and maintenance plan is +specified in the document `Pan-STARRS PS-1 IPP Cluster Support' +(PSDC-430-014). \begin{figure} \begin{center} \resizebox{6in}{!}{\includegraphics[angle=-90]{pics/ps1_ipp_storage.ps}} -\caption{ \label{StorageProfile} Storage Profile} +\caption{ \label{fig:StorageProfile} Storage Profile} \end{center} \end{figure} @@ -2460,16 +2645,14 @@ \clearpage - -\section{Appendices} - -\subsection{Image Server Database Table Contents} -\label{ImageServerTableContents} - -Tables~\ref{ImageServerTables:SO} - \ref{ImageServerTables:VOL} list +\appendix +\section{Image Server Database Table Contents} +\label{sec:ImageServerTableContents} + +Tables~\ref{tab:ImageServerTables:SO} - \ref{tab:ImageServerTables:VOL} list the basic contents of the Image Server database tables. \begin{table}[bh] \begin{center} -\caption{Storage Object Table Contents\label{ImageServerTables:SO}} +\caption{Storage Object Table Contents\label{tab:ImageServerTables:SO}} \begin{tabular}{lll} \hline @@ -2488,5 +2671,5 @@ \begin{table}[bh] \begin{center} -\caption{Instance Table Contents\label{ImageServerTables:INT}} +\caption{Instance Table Contents\label{tab:ImageServerTables:INT}} \begin{tabular}{lll} \hline @@ -2508,5 +2691,5 @@ \begin{table}[bh] \begin{center} -\caption{Volume Table Contents\label{ImageServerTables:VOL}} +\caption{Volume Table Contents\label{tab:ImageServerTables:VOL}} \begin{tabular}{lll} \hline @@ -2515,5 +2698,5 @@ \hline \code{vol_id} & integer & internal volume identifier \\ -\code{uri} & string & node name? \\ +\code{uri} & string & node name \\ \hline \end{tabular} @@ -2522,13 +2705,13 @@ \clearpage -\subsection{Metadata Database Table Contents} -\label{MetadataTableContents} - -Tables~\ref{WeatherTable} -- \ref{overlaps} list the basic contents of -each of the Metadata Database tables listed in Section~\ref{Metadata}. +\section{Metadata Database Table Contents} +\label{sec:MetadataTableContents} + +Tables~\ref{tab:WeatherTable} -- \ref{tab:overlaps} list the basic contents of +each of the Metadata Database tables listed in Section~\ref{sec:Metadata}. \begin{table}[bh] \begin{center} -\caption{Weather Table: some sample weather points\label{WeatherTable}} +\caption{Weather Table: some sample weather points\label{tab:WeatherTable}} \begin{tabular}{lll} \hline @@ -2549,5 +2732,5 @@ \begin{table}[bh] \begin{center} -\caption{SkyProbe Transparency Table (sample entries)\label{SkyprobeBVTable}} +\caption{SkyProbe Transparency Table (sample entries)\label{tab:SkyprobeBVTable}} \begin{tabular}{lll} \hline @@ -2569,5 +2752,5 @@ \begin{table}[bh] \begin{center} -\caption{Skyprobe Line Absorption Table (sample entries)\label{SkyprobeATable}} +\caption{Skyprobe Line Absorption Table (sample entries)\label{tab:SkyprobeATable}} \begin{tabular}{lll} \hline @@ -2592,5 +2775,5 @@ \begin{table}[bh] \begin{center} -\caption{Skyprobe Line Emission Table (sample entries)\label{SkyprobeETable}} +\caption{Skyprobe Line Emission Table (sample entries)\label{tab:SkyprobeETable}} \begin{tabular}{lll} \hline @@ -2613,5 +2796,5 @@ \begin{table}[bh] \begin{center} -\caption{DIMM Measurements Table\label{DimmTable}} +\caption{DIMM Measurements Table\label{tab:DimmTable}} \begin{tabular}{lll} \hline @@ -2634,5 +2817,5 @@ \begin{table}[bh] \begin{center} -\caption{Near IR Wide-field Camera Results Table\label{NIR-Table}} +\caption{Near IR Wide-field Camera Results Table\label{tab:NIR-Table}} \begin{tabular}{lll} \hline @@ -2653,5 +2836,5 @@ \begin{table}[bh] \begin{center} -\caption{Dome Status Table\label{DomeStatusTable}} +\caption{Dome Status Table\label{tab:DomeStatusTable}} \begin{tabular}{lll} \hline @@ -2671,5 +2854,5 @@ \begin{table}[bh] \begin{center} -\caption{Telescope Status\label{TelescopeStatusTable}} +\caption{Telescope Status\label{tab:TelescopeStatusTable}} \begin{tabular}{lll} \hline @@ -2690,5 +2873,5 @@ \begin{table}[bh] \begin{center} -\caption{Raw FPA Images\label{RawFPAs}} +\caption{Raw FPA Images\label{tab:RawFPAs}} \begin{tabular}{lll} \hline @@ -2720,5 +2903,5 @@ \begin{table}[bh] \begin{center} -\caption{Pending Science Chips\label{PendingChips}} +\caption{Pending Science Chips\label{tab:PendingChips}} \begin{tabular}{lll} \hline @@ -2736,5 +2919,5 @@ \begin{table}[bh] \begin{center} -\caption{Processed Science Chips\label{ProcessedChips}} +\caption{Processed Science Chips\label{tab:ProcessedChips}} \begin{tabular}{lll} \hline @@ -2753,5 +2936,5 @@ \begin{table}[bh] \begin{center} -\caption{Observation Group Information\label{OBS}} +\caption{Observation Group Information\label{tab:OBSGroup}} \begin{tabular}{lll} \hline @@ -2763,5 +2946,5 @@ Type & string & Type of observation. \\ Status & string & Status of the observation group. \\ -\tbd{etc} & \\ +etc & \\ \hline \end{tabular} @@ -2771,5 +2954,5 @@ \begin{table}[bh] \begin{center} -\caption{Observation Frame Information\label{OBS}} +\caption{Observation Frame Information\label{tab:OBSFrame}} \begin{tabular}{lll} \hline @@ -2781,5 +2964,5 @@ Type & string & Type of observation. \\ Status & string & Status of the observation group. \\ -\tbd{etc} & \\ +etc & \\ \hline \end{tabular} @@ -2789,5 +2972,5 @@ \begin{table}[bh] \begin{center} -\caption{Science Processing Stats\label{PSStats}} +\caption{Science Processing Stats\label{tab:PSStats}} \begin{tabular}{lll} \hline @@ -2827,5 +3010,5 @@ \begin{table}[bh] \begin{center} -\caption{Chip / Sky overlaps\label{overlaps}} +\caption{Chip / Sky overlaps\label{tab:overlaps}} \begin{tabular}{lll} \hline @@ -2843,5 +3026,5 @@ \begin{table}[bh] \begin{center} -\caption{Processed Sky-Cell stats\label{ProcessedSky}} +\caption{Processed Sky-Cell stats\label{tab:ProcessedSky}} \begin{tabular}{lll} \hline @@ -2850,5 +3033,5 @@ \hline Input Chips & string & Identification numbers of the chips used to produce the sky cell. \\ -PSF adjustments & string & \tbd{Adjustments to the PSF.} \\ +PSF adjustments & string & Adjustments to the PSF. \\ CR rejection stats & string & Statistics from the CR rejection (number of CRs, distribution, limiting flux). \\ Image comb params & string & Parameters used for the image combination. \\ @@ -2865,13 +3048,321 @@ \clearpage -\subsection{AP Database Table Contents} -\label{APDBTableContents} - -\tbd{Table contents to be defined} +\section{AP Database Table Contents} +\label{sec:APDBTableContents} + +\begin{table}[bh] +\begin{center} +\caption{Images\label{tab:images}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Image ID & & \\ +time/date & & \\ +Exposure Time & & \\ +Nstars & & \\ +NX & & \\ +NY & & \\ +photcode & & \\ +Mcal & & \\ +Mcal error & & \\ +Mcal chisq & & \\ +Airmass & & \\ +Astrometry & & \\ +PSF & & \\ +flags & & \\ +Camera & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Image Overlaps\label{tab:ImageOverlaps}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Image ID & & \\ +Region Table & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Objects\label{tab:Objects}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +ID & & \\ +$\alpha$ & & \\ +$\delta$ & & \\ +$\mu_{\alpha}$ & & \\ +$\mu_{\delta}$ & & \\ +$\sigma_{\alpha}$ & & \\ +$\sigma_{\delta}$ & & \\ +$\chi^2$ position & & \\ +$N_{\rm det}$ & & \\ +$N_{\rm miss}$ & & \\ +flags & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Average Magnitudes\label{tab:AveMags}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +object ID & & \\ +$M_{\rm int}$ & & \\ +$M_{\rm ext}$ & & \\ +$\chi^2_{\rm mag}$& & \\ +$\sigma_{\rm mag}$& & \\ +photcode & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Solar System Objects\label{tab:SSObjs}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +SSO ID & & \\ +$N_{\rm det}$ & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Matched Detections\label{tab:Detections}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +$\alpha$ & & \\ +$\delta$ & & \\ +$\sigma_{\alpha}$ & & \\ +$\sigma_{\delta}$ & & \\ +$M_{\rm inst}$ & & \\ +$M_{\rm cal}$ & & \\ +$\sigma_{\rm mag}$& & \\ +photcode & & \\ +type & & \\ +flags & & \\ +time/date & & \\ +airmass & & \\ +$\sigma_{x}$ & & \\ +$\sigma_{y}$ & & \\ +$\theta$ & & \\ +object ID & & \\ +exptime & & \\ +sky & & \\ +$\sigma_{\rm sky}$& & \\ +etc & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Orphaned Detections\label{tab:Orphans}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +$\alpha$ & & \\ +$\delta$ & & \\ +$\sigma_{\alpha}$ & & \\ +$\sigma_{\delta}$ & & \\ +$M_{\rm inst}$ & & \\ +$M_{\rm cal}$ & & \\ +$\sigma_{\rm mag}$& & \\ +photcode & & \\ +type & & \\ +flags & & \\ +time/date & & \\ +airmass & & \\ +$\sigma_{x}$ & & \\ +$\sigma_{y}$ & & \\ +$\theta$ & & \\ +exptime & & \\ +sky & & \\ +$\sigma_{\rm sky}$& & \\ +etc & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Non-detections\label{tab:NonDetects}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +object ID & & \\ +$N_{\rm non-det}$ & & \\ +last time/date & & \\ +last mag & & \\ +faintest time/date & & \\ +faintest mag & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Regions\label{tab:Regions}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +$\alpha_0$ & & \\ +$\alpha_1$ & & \\ +$\delta_0$ & & \\ +$\delta_1$ & & \\ +Region ID & & \\ +Parent ID & & \\ +Nchildren & & \\ +Images & & \\ +Objects & & \\ +Detections & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Filters\label{tab:Filters}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Filter ID & & \\ +Filter name & & \\ +Photcode & & \\ +$\lambda_0$ & & \\ +$\delta_\lambda$ & & \\ +$\epsilon$ & & \\ +transmission curve& & \\ +time/date & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Photcodes\label{tab:Photcodes}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Photcode & & \\ +Telescope & & \\ +Camera & & \\ +Detector & & \\ +Filter & & \\ +Nominal ZP & & \\ +airmass terms & & \\ +color terms & & \\ +Target & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Zero Point History\label{tab:Zpts}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Photcode & & \\ +start Time/date & & \\ +end Time/date & & \\ +Zero Points & & \\ +airmass & & \\ +color & & \\ +error & & \\ +N measurements & & \\ +N stars & & \\ +photom ref set & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Distortion History\label{tab:Distortions}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Camera & & \\ +Telescope & & \\ +distortion terms & & \\ +time/date & & \\ +residuals / error & & \\ +N stars & & \\ +N images & & \\ +astrom ref set & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Database Hosts\label{tab:APDBHosts}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +machine name & & \\ +machine ID & & \\ +\hline +\end{tabular} +\end{center} +\end{table} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsection{Software Runtime Configuration Issues} -\label{RuntimeConfig} +\section{Software Runtime Configuration Issues} +\label{sec:RuntimeConfig} The IPP Software requires extensive runtime configuration information. @@ -2881,8 +3372,8 @@ Metadata Database or in configuration files available to the user. Both methods are implemented in the current design. In either method, -the necessary parameters are identical. In this section, we discuss -the contents of specific portions of the runtime configuration. - -\subsubsection{Camera Definition Information} +the necessary parameters are identical. This section discusses the +contents of specific portions of the runtime configuration. + +\subsection{Camera Definition Information} Every camera which may be analysed by the IPP has differences in how @@ -2910,10 +3401,10 @@ keywords for the same information at different times (major readout software upgrades, for example, can be accompanied by keyword -revisions). In addition, within Pan-STARRS and the IPP, we would like -the capability to refer to the Metadata database as the authoratative -sources of some of these entries rather than the image headers. Given -this circumstance, it is at least necessary to define the appropriate -source for a given data concept appropriate to data from a specific -camera. +revisions). In addition, within Pan-STARRS and the IPP, it is +necessary to have the capability to refer to the Metadata database as +the authoratative sources of some of these entries rather than the +image headers. Given this circumstance, it is at least necessary to +define the appropriate source for a given data concept appropriate to +data from a specific camera. The second problem arises when actually performing an analysis. In @@ -2933,16 +3424,16 @@ In order to facilitate the operation of the IPP with a variety of cameras, and to allow the software the flexibility to change the -camera defintion dynamically, we define a collection of software -runtime configuration information which defines a given camera. This -information is represented below in the form of the PSLib Metadata -Config file, but may be stored in the Metadata Database or in an -alternate format as appropriate. - -We start by noting that the a single camera is represented as a Focal -Plane Array (FPA), divided into Chips, divided into Cells. For a -single FPA, all imaging data is stored in a FITS file or a collection -of FITS files. Software needs to know where in a given file or set of -files to find a particular Cell, what Cells to expect, what chips to -expect, and the relationships between those entities, etc. +camera defintion dynamically, the IPP includes a collection of +software runtime configuration information which defines a given +camera. This information is represented below in the form of the +PSLib Metadata Config file, but may be stored in the Metadata Database +or in an alternate format as appropriate. + +The a single camera is represented as a Focal Plane Array (FPA), +divided into Chips, divided into Cells. For a single FPA, all imaging +data is stored in a FITS file or a collection of FITS files. Software +needs to know where in a given file or set of files to find a +particular Cell, what Cells to expect, what chips to expect, and the +relationships between those entities, etc. A single camera configuration file (or dataset) represents the @@ -2955,5 +3446,5 @@ NCELL S32 NN NCHIP S32 NN -EXPTIME-SRC STR HD:EXPTIME # need to specify PHU vs EXTNAME? +EXPTIME-SRC STR HD:EXPTIME # need to specify PHU vs EXTNAME EXPTIME-KEY STR EXPTIME DATE-KEY STR DATE-OBS @@ -2965,5 +3456,5 @@ \end{verbatim} -\subsubsection{Analysis Recipe Information} +\subsection{Analysis Recipe Information} In order to maintain flexibility in the analysis details, the IPP uses @@ -2975,9 +3466,9 @@ these may specify a specific value, or they may specify lookup methods (database locations, or header locations). The specifies of each -depends on the context. Below, we provide an example recipe file for -the bias subtraction portion of Phase 2, giving several alternative -options for certain entries. Note that, for example, the overscan -subtraction may be specified as using a particular region given in the -recipe file, or on the basis of a particular header keyword. +depends on the context. Below is an example recipe file for the bias +subtraction portion of Phase 2, giving several alternative options for +certain entries. Note that, for example, the overscan subtraction may +be specified as using a particular region given in the recipe file, or +on the basis of a particular header keyword. \begin{verbatim} @@ -3003,12 +3494,14 @@ \end{verbatim} -\subsection{I/O Code Autogeneration} - -Within IPP, we have a number of data collections which have multiple -representations. We define a tool to automatically generate code to -provide I/O APIs to read and write these data and data structures to -carry them within program. Within the IPP, we will use database -tables (ie, in the Metadata Database), FITS Tables (to exchange bulk -data), and XML (to exchange more complete datasets). +\section{I/O Code Autogeneration} +\label{sec:AutocodeIO} + +The IPP includes a number of data collections which have multiple +representations. A software tool will be used to automatically +generate code to provide I/O APIs to read and write these data and to +define the data structures used to carry them within a program. +Within the IPP, examples of these different data entities include +database tables (ie, in the Metadata Database), FITS Tables (to +exchange bulk data), and XML (to exchange more complete datasets). I/O API Autocode template (example.def): @@ -3046,6 +3539,8 @@ \end{verbatim} -\bibliographystyle{plain} -\bibliography{panstarrs} +%\bibliographystyle{plain} +%\bibliography{panstarrs} + +\input{glossary.tex} \end{document} Index: /trunk/doc/design/ippSRS.tex =================================================================== --- /trunk/doc/design/ippSRS.tex (revision 2543) +++ /trunk/doc/design/ippSRS.tex (revision 2544) @@ -1,7 +1,7 @@ - %%% $Id: ippSRS.tex,v 1.12 2004-10-29 22:00:08 eugene Exp $ + %%% $Id: ippSRS.tex,v 1.13 2004-11-30 23:16:03 eugene Exp $ \documentclass[panstarrs,spec]{panstarrs} % basic document variables -\title{Pan-STARRS Image Processing Pipeline} +\title{Pan-STARRS PS-1 Image Processing Pipeline} \subtitle{Software Requirements Specification} \shorttitle{IPP SRS} @@ -11,5 +11,5 @@ \project{Pan-STARRS Image Processing Pipeline} \organization{Institute for Astronomy} -\version{DR} +\version{01} \docnumber{PSDC-430-005} @@ -34,11 +34,11 @@ \RevisionsStart % version Date Description -DR.01 & 2004.01.01 & First draft \\ \hline -DR.02 & 2004.03.10 & Second draft \\ \hline -DR.03 & 2004.04.13 & Most paragraphs fleshed out \\ \hline -DR.04 & 2004.04.27 & Basic text frozen for internal review \\ \hline -DR.05 & 2004.05.24 & Incorporating comments from internal review \\ \hline -DR.06 & 2004.08.06 & Revisions in prep of SRR \\ \hline -DR.06 & 2004.10.22 & Revisions based on SRR \\ \hline +% DR.01 & 2004.01.01 & First draft \\ \hline +% DR.02 & 2004.03.10 & Second draft \\ \hline +% DR.03 & 2004.04.13 & Most paragraphs fleshed out \\ \hline +% DR.04 & 2004.04.27 & Basic text frozen for internal review \\ \hline +% DR.05 & 2004.05.24 & Incorporating comments from internal review \\ \hline +00 & 2004.08.06 & Revisions in prep of SRR \\ \hline +01 & 2004.10.29 & Revisions based on SRR \\ \hline \RevisionsEnd @@ -121,8 +121,9 @@ that series is implied. -Open issues (TBDs) in this document are marked \tbd{in bold red}. - -Quantities which should be reviewed (TBRs) are marked \tbr{in bold -blue}. +Open issues (TBDs) in this document are marked {\bf \color{red} in +bold red}. + +Quantities which should be reviewed (TBRs) are marked {\bf +\color{blue} in bold blue}. \subsubsection{``Shall''} When used in this specification, the word @@ -141,10 +142,13 @@ \DocumentsInternalSection -PSDC-130-001 & PS-1 Design Reference Mission \\ \hline -PSDC-130-xxx & PS-1 SCD \\ \hline -PSDC-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline -PSDC-430-006 & Pan-STARRS IPP ADD \\ \hline -PSDC-430-006 & Pan-STARRS IPP SDRS \\ \hline -PSDC-430-007 & Pan-STARRS IPP PSLib SDRS \\ \hline +PSDC-230-001 & PS-1 Design Reference Mission \\ \hline +PSDC-230-002 & PS-1 System Concept Definition \\ \hline +PSDC-400-006 & The Pan-STARRS IPP Computational Challenge \\ \hline +PSDC-430-004 & Pan-STARRS PS-1 IPP C Code Conventions \\ \hline +PSDC-430-006 & Pan-STARRS PS-1 IPP Algorithm Design Document \\ \hline +PSDC-430-007 & Pan-STARRS PS-1 IPP PSLib Supplementary Design Requirements Specification \\ \hline +PSDC-430-010 & Pan-STARRS PS-1 IPP Perl Code Conventions \\ \hline +PSDC-430-011 & Pan-STARRS PS-1 IPP System/Subsystem Design Description \\ \hline +PSDC-430-012 & Pan-STARRS PS-1 IPP Modules Supplementary Design Requirements Specification \\ \hline \DocumentsExternalSection Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ @@ -179,5 +183,5 @@ The Pan-STARRS System Concept Definition (SCD) specifies the derived top-level requirements for the IPP, which we reproduce here (with -numbering consistent with this document): +numbering consistent within this document): \begin{enumerate} @@ -261,6 +265,6 @@ \label{TLR:15} -\item The IPP shall degrade the stacked image by no more than \tbr{10 - milliarcseconds (FWHM added in quadrature)} over the theoretical +\item The IPP shall degrade the stacked image by no more than 150 + milliarcseconds (FWHM added in quadrature) over the theoretical limit for the stack under infinite sampling.\VER{ANALYSIS}{SCD:3.5.2} @@ -269,6 +273,6 @@ \item The IPP shall perform the processing of science images to the level of transient detection and static sky inclusion at a rate such - that exposures taken at an \tbr{average cadence of 40 seconds} do - not accumulate in the processing buffer (average throughput + that exposures taken at an average cadence of 40 seconds do not + accumulate in the processing buffer (average throughput requirement).\VER{TEST}{SCD:3.2.2.3} \label{TLR:17} @@ -281,5 +285,5 @@ \item The IPP shall perform transient detection to a completeness of - 99\% at the completeness for transient detections with a significant + 99\% at the completeness for transient detections with a significance $> 5\sigma$.\VER{ANALYSIS}{SCD:xxx} @@ -300,6 +304,7 @@ \label{TLR:21} -\item The IPP shall provide access to preferred Pan-STARRS science clients to the - detected transient objects within \tbr{5 minutes}.\VER{TEST}{SCD:3.5.10} +\item The IPP shall provide access to preferred Pan-STARRS science + clients to the detected transient objects within 15 minutes with at + least 85\% reliability.\VER{TEST}{SCD:3.5.10} \label{TLR:22} @@ -375,6 +380,6 @@ \item Because the delivered code is required to run on UNIX machines, the delivered code shall be in compliance with the language-independent UNIX operating system standard POSIX (Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2004).\VER{INSPECT}{allocated} \item Source code files shall use the UNIX line-break convention (line-feed only). \VER{INSPECT}{allocated} -\item C coding style shall adhere to the standard defined in the document `Pan-STARRS C-coding standard' (PSDC-430-004). \VER{INSPECT}{allocated} -\item Perl coding shall follow the standard defined in the document \tbd{`Pan-STARRS Perl-coding standard' (PSDC-430-0XX)}.\VER{INSPECT}{allocated} +\item C coding style shall adhere to the standard defined in the document `Pan-STARRS IPP C-coding standard' (PSDC-430-004). \VER{INSPECT}{allocated} +\item Perl coding shall follow the standard defined in the document `Pan-STARRS IPP Perl-coding standard' (PSDC-430-010).\VER{INSPECT}{allocated} \end{enumerate} @@ -501,17 +506,7 @@ \subsubsection{Software Configuration} -\paragraph{Version Management} - -The IPP software configuration management system shall ensure that -validated versions of both internal and external software are used -when the software is compiled.\VER{TEST}{allocated} - -\paragraph{Optional Modes} - -The IPP software configuration management system shall provide -optionally selected software version sets under compilation -conditions. For example, compilation of the software for test -purposes with a non-standard FFT tool shall be an -option.\VER{TEST}{allocated} +The IPP software configuration management system shall follow the +processes outlined by the Pan-STARRS IPP Software Configuration +Management Place (PSDC-430-003).\VER{INSPECT}{allocated} \subsection{Architectural Components} @@ -525,12 +520,11 @@ As discussed in the Pan-STARRS System Concept Definition -(PSDC-250-002), the IPP is organized into a number of clearly-defined -software elements. The SCD provides a detailed description of the -roles and responsibilities of these subsystems. In brief, the IPP -consists of: a collection of science analysis programs which perform -the stages of the data analysis; a set of architectural components -which provide the infrastructure needed to run the analysis programs; -and a collection of hardware on which all of the software elements -exist and operate. +(PSDC-230-002), the IPP is organized into a number of clearly-defined +software elements. The SCD provides a detailed description of these +subsystems. In brief, the IPP consists of: a collection of science +analysis programs which perform the stages of the data analysis; a set +of architectural components which provide the infrastructure needed to +run the analysis programs; and a collection of hardware on which all +of the software elements exist and operate. The architectural components consist of: @@ -546,4 +540,10 @@ it is no longer needed by other portions of the IPP. +\item {\bf IPP Metadata Database:} This component is used to store all + other data which are neither image files nor astronomical object + data. The Metadata Database is the authoritative source for all + metadata data, including metadata which may be duplicated elsewhere, + such as in the headers of images in the image database. + \item {\bf Astrometry \& Photometry Database (AP):} This component is used to store and manipulate astronomical objects detected in images @@ -554,10 +554,4 @@ needed to interpret the object data. -\item {\bf IPP Metadata Database:} This component is used to store all - other data which are neither image files nor astronomical object - data. The Metadata Database is the authoritative source for all - metadata data, including metadata which may be duplicated elsewhere, - such as in the headers of images in the image database. - \item {\bf IPP Controller:} In order to perform the analysis stages required by the IPP, it is necessary to use distributed computing @@ -588,5 +582,5 @@ \begin{enumerate} \item The IPP Image Server shall accept raw images from the summit at - a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds.} + a sustained rate of 1 exposure (2~GB) per 40 seconds. \VER{TEST}{TLR:17, TLR:23} @@ -597,6 +591,6 @@ reference to the specified image.\VER{TEST}{allocated} -\item The IPP Image Server shall provide a total data capacity of 300 - TB after the first year of PS-1 operations and 900 TB after the +\item The IPP Image Server shall provide a total data capacity of 400 + TB after the first year of PS-1 operations and 750 TB after the second year of operations.\VER{INSPECT}{} @@ -607,73 +601,28 @@ \end{enumerate} -\subsubsection{AP Database} - -%%% Table: AP DB parameters -\begin{table} -\begin{center} -\caption{AP Detection Classes \& Object Parameters\label{APdetections}} -\begin{tabular}{lrrrr} -\hline -\hline -Object Parameter & P2 & P4$\Sigma$ & P4$\Delta$ & SS \\ -\hline -PSF x,y, covar, $\alpha,\delta$ & + & + & + & + \\ -PSF mag, $\sigma_{\rm mag}$ & + & + & + & + \\ -star/gal sep & + & + & + & + \\ -$\sigma_x$, $\sigma_y$, $\theta$ & + & + & + & + \\ -local sky data & + & + & + & + \\ -Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ -S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ -W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ -exp. spaced aps., Poisson noise, variance & - & - & - & + \\ -\hline -\end{tabular} -\end{center} -\end{table} - -%%% Table: AP DB Throughput -\begin{table} -\begin{center} -\caption{AP Data Volume and Throughput Requirements\label{APrates}} -\begin{tabular}{lrrr} -\hline -\hline -Quantity & P2 & P4$\Sigma$ & P4$\Delta$ \\ -\hline -detection limit & $20 \sigma$ & $5 \sigma$ & $3 \sigma$ \\ -depth (r') & 21.8 & 24.0 & 24.5 \\ -bytes star$^{-1}$ & 64 & 100 & 64 \\ -stars deg$^{-2}$ ($|b|>10$) & $2.0 \times 10^5$ & $8.0 \times 10^5$ & $2.0 \times 10^5$ \\ -stars FPA$^{-1}$ ($|b|>10$) & $1.4 \times 10^6$ & $5.6 \times 10^6$ & $1.4 \times 10^6$ \\ -stars sec$^{-1}$ ($|b|>10$) & $3.5 \times 10^4$ & $3.5 \times 10^4$ & $8.8 \times 10^3$ \\ -MB sec$^{-1}$ & 2.3 & 3.5 & 0.6 \\ -AP total TB & 7.7 & - & - \\ -IVP total TB & 13 & 20 & 3 \\ -MOPS total TB & 4 & 6 & 1 \\ -PS-1 total TB & 25 & 26 & 4 \\ -\hline -\end{tabular} -\end{center} -\end{table} - -%% IPP AP DB Requirements -The IPP AP Database has the following performance requirements: - -\begin{enumerate} -\item The AP Database shall accept new detections at the rate - generated by the telescope from the Phase 2 and Phase 4 analysis. - Except within 10 degrees of the galactic plane, the AP Database - shall keep up with the incoming rates. The expected rates are - listed in Table~\ref{APrates}, along with the total data volume - required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2, - TLR:3, TLR:22} - -\item The AP Database shall provide access to external Pan-STARRS - clients to the detected objects within \tbr{5 minute} after the - image is obtained.\VER{TEST}{TLR:22} - \label{IPP:DeReq:29c} -\end{enumerate} - \subsubsection{Metadata Database} + +%% Metadata DB Requirements + +The Metadata Database has the following requirements: + +\begin{enumerate} +\item The IPP Metadata Database shall accept metadata from the summit + at a nightly average rate of 1 MB per 40 second.\VER{TEST}{TLR:17, + TLR:21, TLR:25} + +\item The Metadata Database queries shall have a latency of $< 0.1$ + seconds.\VER{TEST}{TLR:17} + +\item The Metadata Database shall be capable of at least 100 queries + per second.\VER{TEST}{TLR:17} + +\item The Metadata Database shall be capable of accepting a total data + volume after 2 years of operation of 280 GB. \VER{INSPECT}{TLR:25} + +\item The Metadata Database shall restrict write access of the + scientific parameters to a different group from the software and + hardware configuration parameters.\VER{TEST}{allocated} +\end{enumerate} %% Table: Metadata data classes @@ -702,26 +651,71 @@ \end{table} -%% Metadata DB Requirements - -The Metadata Database has the following requirements: - -\begin{enumerate} -\item The IPP Metadata Database shall accept metadata from the summit - at a sustained rate of \tbr{1 MB per 40 second.}\VER{TEST}{TLR:17, - TLR:21, TLR:25} - -\item The Metadata Database queries shall have a latency of $< 0.1$ - seconds.\VER{TEST}{TLR:17} - -\item The Metadata Database shall be capable of at least 100 queries - per second.\VER{TEST}{TLR:17} - -\item The Metadata Database shall be capable of accepting a total data - volume after 2 years of operation of 280 GB. \VER{INSPECT}{TLR:25} - -\item The Metadata Database shall restrict write access of the - scientific parameters to a different group from the software and - hardware configuration parameters.\VER{TEST}{allocated} -\end{enumerate} +\subsubsection{AP Database} + +%% IPP AP DB Requirements +The IPP AP Database has the following performance requirements: + +\begin{enumerate} +\item The AP Database shall accept new detections at the rate + generated by the telescope from the Phase 2 and Phase 4 analysis. + Except within 10 degrees of the galactic plane, the AP Database + shall keep up with the incoming rates. The expected rates are + listed in Table~\ref{APrates}, along with the total data volume + required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2, + TLR:3, TLR:22} + +\item The AP Database shall provide access to external Pan-STARRS + clients to the detected transient objects within 15 minutes after + the image is obtained with an 85\% reliability.\VER{TEST}{TLR:22} + \label{IPP:DeReq:29c} +\end{enumerate} + +%%% Table: AP DB parameters +\begin{table}[hb] +\begin{center} +\caption{AP Detection Classes \& Object Parameters\label{APdetections}} +\begin{tabular}{lrrrr} +\hline +\hline +Object Parameter & P2 & P4$\Sigma$ & P4$\Delta$ & SS \\ +\hline +PSF x,y, covar, $\alpha,\delta$ & + & + & + & + \\ +PSF mag, $\sigma_{\rm mag}$ & + & + & + & + \\ +star/gal sep & + & + & + & + \\ +$\sigma_x$, $\sigma_y$, $\theta$ & + & + & + & + \\ +local sky data & + & + & + & + \\ +Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ +S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ +W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ +exp. spaced aps., Poisson noise, variance & - & - & - & + \\ +\hline +\end{tabular} +\end{center} +\end{table} + +%%% Table: AP DB Throughput +\begin{table} +\begin{center} +\caption{AP Data Volume and Throughput Requirements\label{APrates}} +\begin{tabular}{lrrr} +\hline +\hline +Quantity & P2 & P4$\Sigma$ & P4$\Delta$ \\ +\hline +detection limit & $20 \sigma$ & $5 \sigma$ & $3 \sigma$ \\ +depth (r') & 21.8 & 24.0 & 24.5 \\ +bytes star$^{-1}$ & 64 & 100 & 64 \\ +stars deg$^{-2}$ ($|b|>10$) & $2.0 \times 10^5$ & $8.0 \times 10^5$ & $2.0 \times 10^5$ \\ +stars FPA$^{-1}$ ($|b|>10$) & $1.4 \times 10^6$ & $5.6 \times 10^6$ & $1.4 \times 10^6$ \\ +stars sec$^{-1}$ ($|b|>10$) & $3.5 \times 10^4$ & $3.5 \times 10^4$ & $8.8 \times 10^3$ \\ +MB sec$^{-1}$ & 2.3 & 3.5 & 0.6 \\ +AP total TB & 7.7 & - & - \\ +IVP total TB & 13 & 20 & 3 \\ +MOPS total TB & 4 & 6 & 1 \\ +PS-1 total TB & 25 & 26 & 4 \\ +\hline +\end{tabular} +\end{center} +\end{table} \subsubsection{Controller} @@ -810,10 +804,11 @@ \begin{enumerate} \item The IPP Science Analysis shall pre-process the science images - with the master calibration images at a sustained rate of 1 exposure - (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17} + with the master calibration images at a nightly average rate of 1 + exposure (2~GB) per 40 seconds.\VER{TEST}{TLR:17} \item The IPP Science Analysis shall merge multiple pre-processed science images into stacked images with corresponding signal-to-noise - maps at a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17} + maps at a nightly average rate of 1 exposure (2~GB) per 40 + seconds.\VER{TEST}{TLR:17} \item The IPP Science Analysis shall excise pixels from the input @@ -823,11 +818,13 @@ \item The IPP Science Analysis shall merge the cleaned images into the static sky image, and update the corresponding exposure (S/N) maps, - at a sustained rate of 1 exposure (2~GB) per \tbr{40 seconds}.\VER{TEST}{TLR:17} + at a nightly average rate of 1 exposure (2~GB) per 40 + seconds.\VER{TEST}{TLR:17} \item The maximum latency between the acquisition of an image and the completion of the science image analysis is set by the science requirements of the fast transient recovery programs. The science - image analysis shall process images to detection transients within - \tbr{5 min} of their acquisition.\VER{TEST}{TLR:22} + image analysis shall process images to the detection of transients + within 15 min of their acquisition with an 85\% + reliability.\VER{TEST}{TLR:22} \item The science image analysis stages shall processes up to 1000 @@ -905,6 +902,4 @@ images which are not undersampled. \VER{TEST}{TLR:18} -\item The resulting astrometric solution shall be consistent across the - OTA field to within \tbr{100 milli-arcsec}.\VER{TEST}{TLR:4} \end{enumerate} @@ -926,15 +921,15 @@ resulting astrometric solution shall have a residual scatter of $< 30$ milliarcseconds when calibrated with the AP Survey reference - catalog and $< 100$ milliarcseconds when calibrated with the USNO-B - catalog.\VER{ANALYSIS}{TLR:} + catalog and $< 200$ milliarcseconds when calibrated with the USNO-B + catalog.\VER{ANALYSIS}{TLR:4} \item For images obtained under normal observing conditions, the - resulting astrometric solution shall have a precision relative to - ICRS of better than 100 milliarcseconds.\VER{ANALYSIS}{TLR:} + resulting astrometric solution shall have systematic errors relative + to ICRS of $< 100 milliarcseconds$.\VER{ANALYSIS}{TLR:3} \item For images obtained under photometric conditions or minimal cirrus conditions ($< 0.1$ mag total extinction), the resulting photometric calibration shall have a relative accuracy of 5 - millimagnitudes.\VER{ANALYSIS}{TLR:} + millimagnitudes.\VER{ANALYSIS}{TLR:1} \item For images obtained under photometric conditions or minimal @@ -942,5 +937,5 @@ photometric calibration shall have an absolution photometric accuracy of 10 millimagnitudes when calibrated relative to the AP - Survey reference catalog.\VER{ANALYSIS}{TLR:} + Survey reference catalog.\VER{ANALYSIS}{TLR:1} \item For images obtained under photometric conditions or minimal @@ -948,5 +943,5 @@ conditions listed in Table~\ref{moonconditions}, the resulting sky background subtraction shall leave behind peak-to-peak residuals $< - 1$\% of the input sky flux.\VER{ANALYSIS}{TLR:} + 1$\% of the input sky flux.\VER{ANALYSIS}{TLR:1} \end{enumerate} @@ -968,5 +963,5 @@ \item The sky representation shall degrade the image quality by less - than 10 milliarcseconds added in quadrature to the input image + than 150 milliarcseconds added in quadrature to the input image quality.\VER{TEST}{TLR:1} @@ -975,7 +970,9 @@ time. \VER{TEST}{TLR:17} -\item \tbd{completeness} - -\item \tbd{contamination} +\item The Phase 4 analysis shall have a transient detection + completeness of 99\% for detections with a significance $> 5\sigma$. + +\item The Phase 4 analysis shall have a false detection rate of $< + 5\%$ for transients detections with a significance $> 5\sigma$. \end{enumerate} @@ -1025,5 +1022,5 @@ The required set of Pan-STARRS modules and their functionality is specified in the document `Pan-STARRS Image Processing Pipeline Modules -Supplementary Design Requirements' (PSDC-430-xxx), and details of +Supplementary Design Requirements' (PSDC-430-012), and details of specific algorithms are specified in the document `Pan-STARRS Image Processing Pipeline Algorithm Design Document' (PSDC-430-006). @@ -1072,4 +1069,41 @@ \subsubsection{External Catalogs} +\begin{table} +\begin{center} +\caption{Astrometric Reference Catalogs\label{AstroRefs}} +\begin{tabular}{lrrrrl} +\hline +\hline +Name & scatter limit & proper & depth & Nstars & filters \\ + & (milliarcsec) & motion &(mag) & (millions) & \\ +\hline +Hipparcos & 1 & 2 & 7.3 & 0.1 & {\em V} \\ +Tycho2 & 10 & 1 & 11.5 & 2.5 & {\em B,V} \\ +UCAC-2 & 20 & 1 & 16.0 & 48.0 & {\em R} \\ +USNO-A2.0 & 250 & N/A & 19.0 & 526.2 & {\em B,R} \\ +USNO-B1.0 & 200 & 20 & 21.0 & 1042.6 & {\em B,R} \\ +2MASS & 70 & N/A & 16.0 & 470.0 & {\em J,H,K} \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table} +\begin{center} +\caption{Photometric Reference Catalogs\label{PhotoRefs}} +\begin{tabular}{lrrr} +\hline +\hline +Name & scatter & depth & filters \\ + & mmag & mag & \\ +\hline +SDSS & 15 & 16 & {\em u,g,r,i,z} \\ +CFHT-LS & 10 & 18 & {\em u,g,r,i,z} \\ +Landolt & 10-20 & 15 & {\em U,B,V,R,I} \\ +\hline +\end{tabular} +\end{center} +\end{table} + The IPP AP Database shall be able to interact with the externally provided reference catalogs listed in Table~\ref{AstroRefs} and @@ -1078,5 +1112,6 @@ \subsubsection{Static Sky Pixel Size} -The IPP static sky shall have a pixel scale of \tbr{0.2\arcsec}. +The IPP static sky shall have a pixel scale of +0.2\arcsec.\VER{ANALYSIS}{TLR:16} \subsection{External Interfaces} @@ -1182,5 +1217,5 @@ \hline \hline -Raw data & 200 TB \\ +Raw data & 400 TB \\ static sky & 350 TB \\ calibration frames & 2.8 TB \\ @@ -1188,5 +1223,5 @@ AP db & 55 TB \\ \hline -total & 610 TB \\ +total & 810 TB \\ \hline \end{tabular} @@ -1204,10 +1239,9 @@ \begin{enumerate} \item The IPP shall store all raw images from the first year from the - AP and IVP surveys. This corresponds to 175,000 images, or 175 TB, - assuming 1 GB per image with compression. The IPP will require - space for 200 TB of raw imagery to store the data from these two - survey components along with raw calibration, test, and short-term - storage of other raw images not in the AP and IVP - surveys.\VER{INSPECT}{TLR:23} + AP and IVP surveys. This corresponds to 180,000 images, or 360 TB, + assuming 2 GB per image. The IPP will require space for 400 TB of + raw imagery to store the data from these two survey components along + with raw calibration, test, and short-term storage of other raw + images not in the AP and IVP surveys.\VER{INSPECT}{TLR:23} \item The IPP shall store a single copy of the complete static sky in @@ -1226,5 +1260,6 @@ represent at most 2 terabytes. \VER{INSPECT}{TLR:25} -\item The IPP shall have storage capacity for a total of 610 TB of data. +\item The IPP shall have storage capacity for a total of 810 TB of + data by the end of PS-1. \end{enumerate} @@ -1353,5 +1388,6 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\section{Appendices} +\clearpage +\appendix \bibliographystyle{plain}