IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 2192


Ignore:
Timestamp:
Oct 21, 2004, 6:43:35 PM (22 years ago)
Author:
eugene
Message:

finished major revisions

Location:
trunk/doc/design
Files:
2 edited

Legend:

Unmodified
Added
Removed

  • trunk/doc/design/ippSDRS.tex

    r2186 r2192  
    1 %%% $Id: ippSDRS.tex,v 1.10 2004-10-21 03:55:59 eugene Exp $
     1%%% $Id: ippSDRS.tex,v 1.11 2004-10-22 04:43:35 eugene Exp $
    22\documentclass[panstarrs]{panstarrs}
    33
    … …  
    66\subtitle{Supplementary Design Requirements Specification}
    77\shorttitle{IPP SDRS}
     8\audience{Pan-STARRS PMO}
    89\author{Eugene Magnier, Paul Price, Josh Hoblitt}
    910\group{Pan-STARRS Algorithm Group}
    … …  
    3031
    3132\listoffigures
    32 
    33 \begin{verbatim}
    34 TODOs
    35 - add hardware org diagram to section 3
    36 - clean 3.4 system ifs: describe types of interactions, which are push which are pull?
    37 - 3.5: move to 3.1?  summary of requirements?
    38 - add Image Server figure
    39 - discuss AP DB operations: addstar, delstar, relphot, etc
    40 - discuss AP DB throughput issues
    41 - unify controller discussion
    42 - scheduler: distinguish states
    43 \end{verbatim}
    4433
    4534\pagebreak
    … …  
    169158
    170159The users of the IPP output are all systems internal to the Pan-STARRS
    171 project.  They consist of the Transient Science Client, which will
    172 receive the detections of transient objects on short time-scales; the
    173 Moving Object Processing System (MOPS), which will receive the
    174 detections of non-stationary transient objects on day-to-week
    175 timescales; and the Published Science Products Subsystem (PSPS), which
    176 will receive all data products of interest to the outside world, and
    177 will act as the long-term archive and publishing clearinghouse.
    178 
    179 The primary IPP hardware system on which the software operates will
    180 not be located at the summit.  Instead, because of thermal, power, and
    181 space constraints, the hardware will likely be located in a facility
    182 off the mountain.  A subset of processing tasks may eventually be
    183 assigned to machines at the summit if justified by the savings in data
    184 transfer time and cost.
     160project.  They consist of: 1) the Preferred Science Clients, which
     161receive specified data products on short timescales.  2) the Moving
     162Object Processing System (MOPS), which is one of the Preferred Science
     163Clients, but has the distinction of being a component funded by
     164Pan-STARRS.  It will receive the detections of non-stationary
     165transient objects.  3) the Published Science Products Subsystem
     166(PSPS), which will receive all data products of interest to the
     167outside world, and will act as the long-term archive and publishing
     168clearinghouse.
     169
     170The IPP receives data from two Pan-STARRS subsystems: the Camera, from
     171which it receives the large volume of image data, and OTIS
     172(Observatory, Telesope and Infrastructure Subsystem), from which it
     173receives metadata describing the images and the environmental
     174conditions.  The primary IPP hardware system on which the software
     175operates will not be located at the summit.  Instead, because of
     176thermal, power, and space constraints, the hardware will likely be
     177located in a facility off the mountain.  A subset of processing tasks
     178may eventually be assigned to machines at the summit if justified by
     179the savings in data transfer time and cost.
     180
     181The Pan-STARRS camera produces images consisting of multiple chips
     182(Orthogonal Transfer Arrays or OTAs), each consisting of multiple
     183cells (continuous set of pixels).  The baseline design for the
     184Pan-STARRS camera contains 64 chips each with 64 cells.
    185185
    186186This document defines the design requirements of the IPP for the PS-1
    … …  
    195195several important ways.  First, with only one telescope and camera,
    196196the data throughput rate is substantially reduce to a maximum of 1
    197 64-OTA image per 40 seconds rather than 4.  In addition, much of the
    198 PS-1 mission will be devoted to calibration and testing which will
    199 imply a different level of processing.  For a significant fraction of
    200 PS-1, data will be obtained for the AP Survey covering the entire
    201 $3\pi$ steradians of the sky accessible to PS-4.  These images will
    202 not initially be analysed to the level of having multiple images
    203 combined.  Rather, the analysis will only be performed for individual
    204 focal plane array images.  Only after the AP Survey is done, the
    205 analysis process has been validated, and the complete AP Survey
    206 reference catalog has been generated will it be possible to generate
    207 the first epoch static sky image, rougly 18 months into the PS-1
    208 mission.  This difference in approach has implications for the storage
    209 required by PS-1: rather than delete images soon after they have been
    210 used, raw images must be stored for at least the first 18 months of
    211 PS-1 operations.
     19764-OTA image per 40 seconds rather than 4.  Since PS-1 is a prototype
     198for testing the Pan-STARRS hardware and software subsystems, the
     199observing strategy is not a fixed quantity.  The PS-1 Design Reference
     200Mission (PSDC-xxx) provides some guidelines for the types of projects
     201to be performed, including starting an AP Survey which will eventually
     202cover the entire $3\pi$ steradians of the sky accessible to PS-4.  As
     203a prototype, it is expected that much of the data collected by PS-1
     204will be processed multiple times to test and tune the analysis steps.
     205This difference in approach has implications for the storage required
     206by PS-1: rather than delete images soon after they have been used, raw
     207images must be stored for at least the first 18 months of PS-1
     208operations.  We have used the PS-1 Design Reference Mission as a
     209baseline for these storage requirements to drive our hardware design.
    212210
    213211\subsection{System Design Decisions}
    214212
    215 Since Pan-STARRS is a survey project, all data from the telescopes will be
    216 uniformly analysed by the Pan-STARRS Image Processing Pipeline (IPP) and
    217 the appropriate resulting data products made available to internal and
    218 external science analysis systems as they become available.  The
    219 processing performed by the IPP on the science images will consist of
    220 detrending and object detection for the individual images, combination
    221 of multiple overlapping images and further object detection,
    222 subtraction of a reference (static-sky) image and detection of
    223 residual objects, update of the static sky images, and detailed object
    224 analysis of the static sky images.  In addition, the IPP will produce
    225 improved astrometric and photometric reference catalogs on an
     213Since Pan-STARRS is a survey project, all data from the telescopes
     214will be uniformly analysed by the Pan-STARRS Image Processing Pipeline
     215(IPP) and the appropriate resulting data products made available to
     216internal and external science analysis systems as they become
     217available.  The processing performed by the IPP on the science images
     218will consist of detrending and object detection for the individual
     219images, combination of multiple overlapping images and further object
     220detection, subtraction of a reference (static-sky) image and detection
     221of residual objects, update of the static sky images, and detailed
     222object analysis of the static sky images.  In addition, the IPP will
     223produce improved astrometric and photometric reference catalogs on an
    226224occasional basis as needed.  The output data products from the IPP
    227225consist of the calibration images, reduced images from the individual
    228226telescopes, combined images, difference images, the static sky image,
    229227object photometry, and reference astrometry and photometry.
    230 
    231 The IPP interacts closely with other Pan-STARRS systems responsible for
    232 other aspects of the Pan-STARRS operation, including the summit systems
    233 (OATS), the science object database, the Moving Object Processing
    234 System (MOPS), and potentially other client science pipelines.
    235228
    236229The requirements for the IPP, as identified in the IPP SRS (PSDC-REF)
    … …  
    264257
    265258Depending on the particular stage, it may process individual images,
    266 collections of images, or on derived data products.  Because of the
     259collections of images, or derived data products.  Because of the
    267260nature of the image data, many of the analysis stages can be run in
    268 parallel because, for example, the analysis of a chip in one image
    269 does not depend on the results from another chip.
     261parallel.  For example, the analysis of a chip in one image does not
     262depend on the results from another chip.
    270263
    271264\subsection{Architectural Components}
     265
     266\begin{figure}
     267\begin{center}
     268\resizebox{6in}{!}{\includegraphics{pics/IPPoverview}}
     269\caption{ \label{overview} IPP System Overview}
     270\end{center}
     271\end{figure}
    272272
    273273In order to achieve the required functionality, the IPP provides an
    … …  
    324324\begin{figure}
    325325\begin{center}
    326 \resizebox{6in}{!}{\includegraphics{pics/IPPoverview}}
    327 \caption{ \label{overview} IPP System Overview}
    328 \end{center}
    329 \end{figure}
    330 
    331 \subsection{IPP Hardware Organization}
    332 
    333 \begin{figure}
    334 \begin{center}
    335326\resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}}
    336327\caption{ \label{hardware} IPP Hardware Organization}
    337328\end{center}
    338329\end{figure}
     330
     331\subsection{IPP Hardware Organization}
    339332
    340333The IPP needs substantial computer resources, both in terms of
    … …  
    356349those that provide the storage for the other data systems, the
    357350Metadata Database and the AP Database.  In addition, the computational
    358 tasks related to Phase 2 take place on the per-OTA storage nodes and
    359 the Phase 4 computation takes place on the static sky storage nodes.
     351tasks related to the individual images take place on the per-OTA
     352storage nodes and the processing of stacks of images takes place on
     353the static sky storage nodes.
    360354
    361355Figure~\ref{hardware} shows our basic concept for the hardware
    362356organization for the IPP.  This diagram shows the two types of compute
    363 nodes: OTA-level processing and storage nodes (dominated by Phase 2)
    364 and static sky processing and storage nodes (mostly Phase 4).  Also
    365 shown are two switches which divide the network into OTA and
    366 Static-Sky portions.  In such an organization, the interswitch
    367 communication must meet the throughput needs between these network
    368 portions.  The additional data systems (Metadata Database and AP
    369 Database) are also shown.
     357nodes: OTA-level processing and storage nodes and static sky
     358processing and storage nodes.  Also shown are two switches which
     359divide the network into OTA and Static-Sky portions.  In such an
     360organization, the interswitch communication must meet the throughput
     361needs between these network portions (though a single switch may also
     362be used if its backplane capacity is sufficient).  The additional data
     363systems (Metadata Database and AP Database) are also shown.
    370364
    371365%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    … …  
    390384across a collection of computer nodes, each with their own data
    391385storage resources.  Any single file is stored on only a single
    392 computer and storage system.  In order to achieve the data throughput
     386computer and storage device.  In order to achieve the data throughput
    393387requirements, the IPP Image Server may distribute the images across
    394388the processor nodes in an organized fashion, i.e., associating
    … …  
    405399
    406400\item {\bf instance} A single copy of the storage object in the Image
    407   Server.  In general, given storage object may have several instances
     401  Server.  In general, a given storage object may have several instances
    408402  in the Image Server, normally on different computer nodes.
    409403
    … …  
    413407\end{itemize}
    414408
    415 The Image Server provides file pointers (in C), handles (in Perl), or
    416 file names corresponding to the instances of the storage objects.  The
    417 Image Server provides the data organization but does not define a file
    418 system; it assumes the existence of an appropriate file system which
    419 provides makes the files visible as local files.  This may be done
    420 over many machines with a network file system such as NFS or GFS.
     409The Image Server provides file pointers (in C), handles (in Perl or
     410Python), or file names corresponding to the instances of the storage
     411objects.  The Image Server provides the data organization but does not
     412define a file system; it assumes the existence of an appropriate file
     413system which makes the files visible as local files.  This
     414may be done over many machines with a network file system such as NFS
     415or GFS.
    421416
    422417The IPP Image Server provides the storage and access mechanisms, but
    … …  
    432427\item Image Server daemon
    433428\item Image Server client APIs
     429\item Image Server maintainence tools
    434430\end{itemize}
    435431
    … …  
    445441
    446442Clients interact with the IPP Image Server via a small number of C
    447 APIs (Bindings are also provided for Perl and Python).  The client
    448 commands are:
     443APIs (Bindings are also provided for Perl and Python and UNIX shell
     444commands in some cases).  The client commands are:
    449445
    450446\begin{itemize}
    … …  
    455451  node name on which the new storage object must be located.  If this
    456452  target is not given, the Image Server places the new storage object
    457   on an appropriate machine from the pool (least filled?  most data?
    458   randomized?  the details need to be decided).
     453  on an appropriate machine from the pool, though the details need to
     454  be specified.
    459455
    460456\item {\tt open object}: open an instance of an existing storage
    … …  
    474470  specified storage object, including the number of instances of the object.
    475471
    476 \item {\tt increment object count}: adds a new instance of the given
    477   storage object.  The target node may be optionally specified,
    478   otherwise an appropriate node is selected.
    479 
    480 \item {\tt decrement object count}: removes one of the instances of
    481   the storage object.  The input parameters may optionally specify the
    482   target machine to delete.
     472\item {\tt replicate object}:a new instance of the given storage
     473  object.  The target node may be optionally specified, otherwise an
     474  appropriate node is selected.
     475
     476\item {\tt cull object}: removes one of the instances of the storage
     477  object.  The input parameters may optionally specify the target
     478  machine to delete.
    483479
    484480\item {\tt delete object}: deletes all instances of the storage object
    … …  
    499495about the data storage objects, their instances, and the available
    500496hardware resources.  A {\tt mysql} database engine is used to manage
    501 the database.  The database tables defined for the Image Server are
    502 listed in Table~\ref{ImageServerTables}, and their current contents
    503 are listed in Appendix A.  This database engine need not the same one
    504 as the one used for othe IPP subsystems.
     497the database table.  The database tables defined for the Image Server
     498are listed in Table~\ref{ImageServerTables}, and their contents are
     499listed in Appendix A.  This database engine need not the same one as
     500the one used for othe IPP subsystems.
    505501%
    506502\begin{table}
    … …  
    533529The IPP Image Server provides a collection of administration tools
    534530which allow for maintainence.  These are operations which may be
    535 automatically scheduled for the IPP or which may be initiated by a
     531automatically scheduled by the IPP or which may be initiated by a
    536532human via a command-shell interface.  The maintainence functions
    537533include migrating data between nodes to rebalance the available space
    … …  
    540536for file corruption, which involves sweeping all files on a data
    541537volume and comparing the calculated file checksum to the currently
    542 recorded value. 
     538recorded value.
    543539
    544540%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    545541
    546542\subsection{Metadata Database}
    547 
    548 The IPP Metadata Database acts as a repository for all non-pixel data
     543\label{Metadata}
     544
     545The IPP Metadata Database acts as a repository for non-pixel data
    549546needed by the IPP subsystems.  This includes the image metadata, the
    550547environmental data, system configuration data and system reference
    551548data.  The Metadata Database is required to save the non-ephemeral
    552549data for the lifetime of the project for future reference and
    553 additional analysis.  The Metadata Database may potentially be used in
    554 close coupling with the analysis pipelines to store temporary data
    555 either within or between stages of the analysis.  In this scenario,
    556 the analysis pipeline will interact directly with the database.
    557 However, database latency may make this scenario impractical, in which
    558 case the database may be used for long-term storage only.  In this
    559 scenario, the data produced by analysis pipelines which is destined
    560 for the Metadata Database may be collected and inserted by a separate,
    561 dedicated process or analysis pipeline collection of processes.
    562 Metadata which is large in volume or poorly structure may also be
    563 stored in an appropriate container file (FITS Table, FITS Header, XML
    564 File) in the Image Server with the Metadata DB providing pointers to
    565 these files.
     550additional analysis.  The Metadata Database may be used in close
     551coupling with the analysis pipelines to store temporary data either
     552within or between stages of the analysis.  In this scenario, the
     553analysis pipeline will interact directly with the database.  However,
     554database latency may make this scenario impractical, in which case the
     555database may be used for long-term storage only.  In this scenario,
     556the data produced by analysis pipelines which is destined for the
     557Metadata Database may be collected and inserted by a separate,
     558dedicated process.  Metadata which is large in volume or poorly
     559structure may also be stored in an appropriate container file (FITS
     560Table, FITS Header, XML File) in the Image Server with the Metadata DB
     561providing pointers to these files.
    566562
    567563The IPP Metadata Database is a simple database system, consisting of a
    … …  
    569565\code{mysql} database engine will be used to drive the database.
    570566
    571 \subsubsection{Metadata Tables}
    572 \label{Metadata}
    573 
    574 The complete contents of the Metadata Database will not be completely
    575 specified until the complete collection of data analysis scripts are
    576 available.  Even so, we can make a good first pass at the likely
    577 collection of long-term tables, and some of the temporary processing
    578 tables.  Table~\ref{MetadtaDBTables} lists the Metadata tables
    579 identified to date for the Metadata Database.  The contents of these
    580 tables are outlined in Appendix~\ref{MetadataContents}, with examples
    581 for the data entries and thier data types in many cases.
    582 
    583 \subsubsection{Metadata Queries}
    584 
    585 The IPP provides simple queries to the Metadata Database tables using
    586 autocoded APIs.  These queries allow for a single row or a simple
    587 collection of rows based on the primary key.  The format of the API is
    588 identical for all Metadata tables.  New tables and APIs can be added
    589 to the IPP system by adding to the autocoding table description
    590 files.  See Appendix~\ref{Autocode} for futher information. 
    591 
    592 \begin{table}
     567\begin{table}[hb]
    593568\begin{center}
    594569\caption{Metadata Database Tables\label{MetadataDBTables}}
    … …  
    619594\end{table}
    620595
     596\subsubsection{Metadata Tables}
     597
     598The contents of the Metadata Database will not be completely specified
     599until the complete collection of data analysis scripts are available.
     600Even so, we can identify the likely collection of long-term tables,
     601and some of the temporary processing tables.
     602Table~\ref{MetadtaDBTables} lists the Metadata tables identified to
     603date for the Metadata Database.  The contents of these tables are
     604outlined in Appendix~\ref{MetadataContents}, with examples for the
     605data entries and thier data types in many cases.
     606
     607\subsubsection{Metadata Queries}
     608
     609The IPP provides simple queries to the Metadata Database tables using
     610autocoded APIs.  These queries return a single row or a collection of
     611rows based on the primary key.  The format of the API is identical for
     612all Metadata tables.  New tables and APIs can be added to the IPP
     613system by adding to the autocoding table description files.  See
     614Appendix~\ref{Autocode} for futher information.
     615
    621616%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    622617
    … …  
    636631those supplied by external references.  These may be treated as {\em
    637632detections}, with the caveat that access to the raw measurements and
    638 metadata are usually unavailable; the reported measurements and errors
     633metadata are usually unavailable: the reported measurements and errors
    639634must be accepted as they are reported.
    640635
    … …  
    645640The AP Database also makes it possible to extract all detections
    646641derived from a specific image and to determine quantities such as the
    647 coordinates of the detection in pixel coordinates on the image.
     642pixel coordinates of the detection on the image.
    648643
    649644The AP Database also has the capability to associate multiple
    … …  
    710705filters.
    711706
     707\begin{figure}
     708\begin{center}
     709\resizebox{4.5in}{!}{\includegraphics{pics/APDB}}
     710\caption{AP DB components}
     711\label{fig:APDBRegions}
     712\end{center}
     713\end{figure}
     714
    712715The AP Database provides interfaces to extract lists of objects and
    713716detections based on various query parameters.  It provides the
    … …  
    749752
    750753The specific subtable of {\tt Images} which contains a given image is
    751 that one which contains the center pixel \tbr{or 0,0 pixel} of that
     754the one which contains the center pixel \tbr{or 0,0 pixel} of that
    752755image.  An additional table group, {\tt Image Overlaps} (with the same
    753756subtable organization as the {\tt Images} subtables), lists images
    754757which overlap that specific subtable.  Thus, given a particular
    755758coordinate, in order to find that images which overlap that
    756 coordinate, it is necessary to load the images in the {\tt Images}
     759coordinate, it is necessary to search the images in the {\tt Images}
    757760subtable which includes that coordinate, and all images in the {\tt
    758 ImageOverlaps} table for that coordinate.
     761ImageOverlaps} subtable for that coordinate.
     762
     763\begin{table}[hb]
     764\begin{center}
     765\caption{AP Database Tables\label{APDBTables}}
     766\begin{tabular}{ll}
     767\hline
     768\hline
     769{\bf Table Name} & {\bf Description} \\
     770\hline
     771Images              & The images that have objects in the DB. \\
     772Image Overlaps      & Image regions which are touched by specific images. \\
     773Objects             & The objects --- average properties of multiple detections of the same object. \\
     774Average Magnitudes  & Average photometry in multiple filters \\
     775Matched Detections  & Detections of sources in an image identified with an Object. \\
     776Orphaned Detections & Detections of sources in an image not identified with an Object. \\
     777Non-detections      & Non-detections of objects in an image. \\
     778Region Table        & spatial distribution of tables \\
     779Filters             & Filters understood by the system. \\
     780Photcodes           & Transformations between different photometric systems \\
     781Database Machines   & computers used to store the tables \\
     782% Zero Points       & Transformations between different photometric systems \\
     783% Distortion Models & Transformations between different photometric systems \\
     784% Solar System Objects & Identification of solar system objects \\
     785\hline
     786\end{tabular}
     787\end{center}
     788\end{table}
    759789
    760790The {\tt Objects} table group (also divided by region) stores the
    … …  
    776806detections associated with the average {\tt Objects}.  As discussed
    777807below, bright objects (above a configuration-specified signal-to-noise
    778 level) are assigned an object even if only one detection has been
    779 found at that position.  Faint orphaned objects are not added to this
    780 list or the list of objects.  The different types of detections (P2,
     808level) are defined object even if only one detection has been found at
     809that position.  Faint orphaned objects are not added to this list or
     810the list of objects.  The different types of detections (P2,
    781811P4$\Delta$, P4$\Sigma$) are distinguished by their photometry codes.
    782 \tbr{This is only valid if the AP Database does not store different
    783 quantities for these types of detections}
     812(This is only valid if the AP Database does not store different
     813quantities for these types of detections.)
    784814
    785815The {\tt Orphaned Detections} table stores the detections which have
    786816not been correlated with an existing object.  This table is only
    787817populated for objects below a configuration-specified signal-to-noise
    788 limit.  Otherwise, even orphaned detections are assigned an object and
    789 added to the {\tt Matched Detections} table.
     818limit (eg 5$\sigma$).  Bright orphaned detections are assigned an
     819object and added to the {\tt Matched Detections} table.
    790820
    791821The {\tt Non-detections} table stores information about detection
    … …  
    816846to serve the database tables.  The region file specifies the machine
    817847which stores the specific table.  Figure~\ref{ABDBRegions} illustrates
    818 schamatically the subdivision of the sky and the association between
     848schematically the subdivision of the sky and the association between
    819849different levels of the hierarchy with different subtables.
    820 
    821 The {\tt Filters} table identifies all of the physical filters
    822 (specific, named pieces of glass) known to the system.  A related
    823 table, {\tt Photcodes}, defines relationships between specific
    824 photometry systems.  A system may consist of a detector, telescope,
    825 and specific filter, or it may be a derived photometry system.  The
    826 {\tt Database Machines} table identifies all of the computers
    827 available to the AP Database.
    828 
    829 \begin{table}
    830 \begin{center}
    831 \caption{AP Database Tables\label{APDBTables}}
    832 \begin{tabular}{ll}
    833 \hline
    834 \hline
    835 {\bf Table Name} & {\bf Description} \\
    836 \hline
    837 Images              & The images that have objects in the DB. \\
    838 Image Overlaps      & Image regions which are touched by specific images. \\
    839 Objects             & The objects --- average properties of multiple detections of the same object. \\
    840 Average Magnitudes  & Average photometry in multiple filters \\
    841 Matched Detections  & Detections of sources in an image identified with an Object. \\
    842 Orphaned Detections & Detections of sources in an image not identified with an Object. \\
    843 Non-detections      & Non-detections of objects in an image. \\
    844 Region Table        & spatial distribution of tables \\
    845 Filters             & Filters understood by the system. \\
    846 Photcodes           & Transformations between different photometric systems \\
    847 Database Machines   & computers used to store the tables \\
    848 % Zero Points       & Transformations between different photometric systems \\
    849 % Distortion Models & Transformations between different photometric systems \\
    850 \hline
    851 \end{tabular}
    852 \end{center}
    853 \end{table}
    854850
    855851\begin{figure}
    … …  
    861857\end{figure}
    862858
    863 \begin{figure}
    864 \begin{center}
    865 \resizebox{4.5in}{!}{\includegraphics{pics/APDB}}
    866 \caption{AP DB components}
    867 \label{fig:APDBRegions}
    868 \end{center}
    869 \end{figure}
     859The {\tt Filters} table identifies all of the physical filters
     860(specific pieces of glass) known to the system.  A related table, {\tt
     861Photcodes}, defines relationships between photometry systems.  A
     862photometry system may consist of a detector, telescope, and specific
     863filter, or it may be a derived photometry system.  The {\tt Database
     864Machines} table identifies all of the computers available to the AP
     865Database.
    870866
    871867\subsubsection{AP Database servers}
    … …  
    902898The backend database engine for the AP Database stores the tables and
    903899provides them to the servers on demand.  The AP Database will use a
    904 \code{mysql} database engine for the function.
     900\code{mysql} database engine for this function.
    905901
    906902\subsubsection{AP DB Client operations}
    … …  
    908904The AP Database client interactions consist of a collection of basic
    909905queries of the database, along with more complex operations to perform
    910 particular tasks.  \tbd{queries are not yet listed; provide list from
    911   DVO}.  The complex operations are listed below.
     906particular tasks.  The complex operations are listed below.
    912907
    913908\paragraph{Insert Image \& Detection Set (addstar)}
    … …  
    958953
    959954This operation uses the reference and image detections to determine an
    960 optical distortion model for the camera.  ñ
     955optical distortion model for the camera.
    961956
    962957\begin{table}
    … …  
    984979\subsubsection{Notes}
    985980
     981discuss AP DB throughput issues
     982
    986983how does the AP Database know about the relationship between a
    987984collection of chips? 
    … …  
    10211018Controller receives a variety of inputs from other subsystems,
    10221019described below, and initiates actions such as adding a new process to
    1023 its queue.  The IPP Controller also provides information to other
    1024 subsystems on demand about its processing history and current state.
    1025 Each physical computer may have multiple processors; since the IPP
    1026 Controller is managing processing tasks, it treats each processor
    1027 independently.  It is up to the system configuration if each computer
    1028 needs to reserve one of its CPUs to manage background tasks or if the
    1029 IPP Controller should attempt to send one task per CPU and let the
    1030 kernel handle the I/O load.
    1031 
    1032 \subsubsection{Controller Nodes}
    1033 
    1034 Computers managed by the IPP Controller are allowed to be in one of
    1035 several states, and the IPP Controller must interact with it in an
    1036 appropriate way for each of those states.  A computer may be {\tt
    1037 alive}, {\tt dead} or {\tt off}.  If the computer is {\tt alive}, it
    1038 responds to commands from the IPP Controller and may be used for tasks
    1039 subject to other constraints.  If it is {\tt dead}, the computer is
    1040 not responsive and must not be used for executing tasks.  The IPP
    1041 Controller must identify computers which have died and occasionally
     1020the queue of pending tasks.  The IPP Controller also provides
     1021information to other subsystems on demand about its processing history
     1022and current state.  Each physical computer may have multiple
     1023processors; since the IPP Controller is managing processing tasks, it
     1024treats each processor independently.  It is up to the system
     1025configuration if each computer needs to reserve one of its CPUs to
     1026manage background tasks or if the IPP Controller should attempt to
     1027send one task per CPU and let the operating system handle the I/O
     1028load.
     1029
     1030\subsubsection{Nodes}
     1031
     1032The Controller maintains a table of processing computers (`Nodes')
     1033available to it and tracks the status of these Nodes.  Nodes managed
     1034by the IPP Controller are allowed to be in one of several states, and
     1035the IPP Controller must interact with it in an appropriate way for
     1036each of those states.  A computer may be {\tt alive}, {\tt dead} or
     1037{\tt off}.  If the computer is {\tt alive}, it responds to commands
     1038from the IPP Controller and may be used for tasks subject to other
     1039constraints.  If it is {\tt dead}, the computer is not responsive and
     1040must not be used for executing tasks.  The IPP Controller must
     1041identify computers which have died (not responding) and occasionally
    10421042test them to see if they are {\tt alive} again.  Computers which are
    1043 {\tt off} are not available for tests and must not be tested.
     1043{\tt off} are not available for tasks and must not be tested.
    10441044Computers may be set to the {\tt off} or {\tt dead} states by external
    10451045subsystems; it is the responsibility of the IPP Controller to return a
    1046 computer to the {\tt alive} state if possible. 
     1046computer to the {\tt alive} state if possible.
    10471047
    10481048The IPP Controller must honor requests (normally from the users) to
    10491049change the mode of any computing node on demand between {\tt off} and
    10501050{\tt dead}.  This would normally be done after a computer has been
    1051 rebooted and is release to the IPP Controller for its use.  It must
     1051rebooted and is released to the IPP Controller for its use.  It must
    10521052also be able to change the list of allowed tasks as requested by
    10531053external commands.
    … …  
    10611061{\tt dead} for a very long time.  In another scenario, a person needs
    10621062to work on a computer.  They notify the IPP Controller that the
    1063 machine is off, perhaps with a prior notification that the machine
    1064 should be prepared to go off.  When work on the machine is complete,
    1065 it should be placed in the {\tt dead} state.  Only when the person is
    1066 done working and testing the machine, and tells the IPP Controller
    1067 that the machine is now {\tt dead} can the IPP Controller attempt to
    1068 re-start communications and processing on that computer.
    1069 
    1070 CPUs on computers which are in the {\tt alive} state may be in one of
    1071 two modes: {\tt busy} and {\tt free}.  A CPU which is {\tt busy}
    1072 currently has a task assigned to it.  The IPP Controller may only
    1073 assign one task to one CPU at a time.  A CPU which is in the {\tt
    1074 free} state may have tasks assigned to it.  The IPP Controller must
    1075 also respect a list of task restrictions which may require specific
    1076 tasks to run on specific CPUs or exclude specific tasks from specific
    1077 CPUs.
    1078 
    1079 The Controller maintains a table of processing nodes available to it
    1080 and the status of these Nodes.  When the Controller starts, it
    1081 attempts to launch a Node Agent on each of the available processing
    1082 nodes.  Modes which are not responsive are placed into an inactive
    1083 state and retried occasionally.
    1084 
    1085 \subsubsection{Controller Node Agents}
    1086 
    1087 A Node Agent runs on each of the individual nodes to perform the tasks
    1088 as directed by the Controller.  The Node Agents communicate with the
    1089 Controller via a socket connection.
    1090 
    1091 A processing stage is executed in the UNIX user space, and is run as a
    1092 fork by the Node Agent.  The Node Agent must monitor the standard
    1093 error and standard output of the processing stage and save them in
    1094 separate buffers.  If the process dies, the Node Agent must detect the
    1095 crash.  The Node Agent must respond to various commands from the
    1096 Controller, as follows:
     1063machine is {\tt off}, perhaps with a prior notification that the
     1064machine should be prepared to go off.  When work on the machine is
     1065complete, it should be placed in the {\tt dead} state.  Only when the
     1066person is done working and testing the machine, and tells the IPP
     1067Controller that the machine is now {\tt dead} can the IPP Controller
     1068attempt to re-start communications and processing on that computer.
     1069
     1070\subsubsection{Node Agents}
     1071
     1072When the Controller starts, it attempts to launch a Node Agent on each
     1073of the available processing Nodes.  Modes which are not responsive are
     1074placed marked as {\tt dead} so they may be retried.  A Node Agent runs
     1075on each of the individual nodes to execute the tasks as directed by
     1076the Controller.  The Node Agents communicate with the Controller via a
     1077socket connection.
     1078
     1079A Node Agent (which is only on Node in the {\tt alive} state) may be
     1080in one of four modes: {\tt idle}, {\tt busy}, {\tt done}, {\tt crash}.
     1081A Node Agent which is {\tt busy} currently has a task assigned to it
     1082which is executing.  The IPP Controller may only assign one task to a
     1083Node at a time.  A Node Agent which is in the {\tt idle} state may
     1084have a task assigned to it.  When the Node Agent detects that a tasks
     1085has finished, it changes to either the {\tt done} or {\tt crash}
     1086states depending on the outcome of the process execution.  The IPP
     1087Controller must also respect a list of task restrictions which may
     1088require specific tasks to run on specific CPUs or exclude specific
     1089tasks from specific CPUs.
     1090
     1091A task being executed by the Node is run in the UNIX user space as a
     1092forked process.  The Node Agent must monitor the standard error and
     1093standard output of the executing task and save them in separate
     1094buffers.  If the process exits or dies, the Node Agent must detect
     1095this result and change state appropriately.  The Node Agent must
     1096respond to various commands from the Controller, as follows:
    10971097
    10981098\paragraph{Report status}
    10991099
    1100 The Node Agent returns the state of the Node (idle, busy, done), the
    1101 state of the current processing stage (`none', `busy', `crash',
    1102 `done'), and the exit status of the current processing stage, if
    1103 available.
    1104 
    1105 The four possible states of the Node indicate that the client has no
    1106 current processing stage (`idle'), that it has a processing stage
    1107 which is still running (`busy'), or that it has a processing stage
    1108 which has completed.  The last two states indicate if the current
    1109 processing stage has crashed (`crash'), or if the current processing
    1110 stage has exited gracefully (`done').  The reported exit state, if the
    1111 process has completed without crashing, is the UNIX exit state
    1112 reported by the processing stage: 0--256 with 0 indicating a
    1113 successful completion.
     1100The Node Agent returns its state ({\tt idle}, {\tt busy}, {\tt done},
     1101{\tt crash'}) and the exit status of the current processing task, if
     1102available.  The reported exit state, if the process has completed
     1103without crashing, is the UNIX exit state reported by the task: 0--256
     1104with 0 indicating a successful completion.
    11141105
    11151106\paragraph{Report stdout}
    … …  
    11181109the complete contents of the stdout buffer via a buffered write and
    11191110flush the buffer when it is finished.  The Node Agent will not accept
    1120 more data on the stdout buffer from the current processing stage until
     1111more data on the stdout buffer from the current processing task until
    11211112the send is complete and the buffer is flushed.  The daemon must
    11221113accept all of the buffer output.
    … …  
    11261117Identical to `report stdout', but for stderr.
    11271118
    1128 \paragraph{Kill processing stage}
    1129 
    1130 The Node Agent should send a kill signal to the current processing
    1131 stage.  When the processing stage has exited, the Node Agent should
    1132 set the processing stage status to `crash' and the Node status to
    1133 `done'.
    1134 
    1135 \paragraph{Clear processing stage}
    1136 
    1137 The Node Agent should set the current processing stage state to `none'
    1138 and the Node state to `idle'.  If a processing stage is currently
    1139 running, it should be killed (signal 9 or 15) before the processing
    1140 stage is cleared.
     1119\paragraph{Kill task }
     1120
     1121The Node Agent should send a kill signal (signal 9 or 15) to the
     1122current processing task.  When the processing task has exited, the
     1123Node Agent should set its state to {\tt crash}.
     1124
     1125\paragraph{Clear task}
     1126
     1127The Node Agent should set its state {\tt idle}.  If a processing stage
     1128is currently running, it should be killed (signal 9 or 15) before the
     1129task is cleared.
    11411130
    11421131\paragraph{Start processing stage}
    … …  
    11441133The Node Agent forks a specified command.  The command should be a
    11451134standard UNIX command without command line redirection or
    1146 backgrounding.  For this reason, the Node Agent must provide a layer
    1147 of security, for example, by employing SSL authentication.
     1135backgrounding.  The task is run with the same user ID as the Node
     1136Agent, which is also the same user ID as the Controller.
    11481137
    11491138\subsubsection{Tasks}
    … …  
    11521141requests include the specific command to be executed and are in the
    11531142form of a UNIX command which could be performed on any of the
    1154 computing nodes.  Any input or output data structures in the commands
    1155 must be a valid resource regardless of the node on which the task is
    1156 executed.  Input and output data resources must be unique where
    1157 necessary to avoid conflicts.  The IPP Controller gives each task a
    1158 unique identifier, which is returned to the requesting entity.  The
    1159 requestor may then use that ID to obtain status information on that
    1160 task or to send control signals to the specific task.
     1143computing nodes.  Any input or output data in the commands must be a
     1144valid resource regardless of the node on which the task is executed.
     1145Input and output data resources must be unique where necessary to
     1146avoid conflicts.  \tbd{It is the responsibility of the programs to
     1147wait for network lags (ie, NFS delays)}.  The IPP Controller gives
     1148each task a unique identifier, which is returned to the requesting
     1149entity.  The requestor may then use that ID to obtain status
     1150information on that task or to send control signals to the specific
     1151task.
    11611152
    11621153Task requests may specify a desired node for the task execution.  The
    … …  
    12451236\subsection{Scheduler}
    12461237
    1247 \begin{figure}
    1248 \begin{center}
    1249 \resizebox{6in}{!}{\includegraphics{pics/Scheduler}}
    1250 \caption{ \label{Scheduler} IPP Scheduler}
    1251 \end{center}
    1252 \end{figure}
    1253 
    1254 The IPP is responsible for a variety of analysis tasks: processing of
     1238The IPP is responsible for a variety of analysis jobs: processing of
    12551239the science images through several stages; routine assessment of the
    12561240detrend (instrumental calibration) images used in processing the
    … …  
    12701254Scheduler may be viewed as the central brain of the IPP.
    12711255Figure~\ref{Scheduler} illustrates the design of the IPP Scheduler.
     1256
     1257\subsubsection{Scheduler Tasks and Tests}
    12721258
    12731259The IPP Scheduler performs two types of actions.  'Tasks' are
    … …  
    12891275Database or other subsystems.  Based on the successful completion (or
    12901276not!) of the tasks, and the new state of entries in the Metadata
    1291 Database, the Scheduler can define new tasks.
    1292 
    1293 The IPP Scheduler sends commands to the IPP Controller for execution.
     1277Database, the Scheduler can define new tasks.
     1278
     1279\begin{figure}
     1280\begin{center}
     1281\resizebox{6in}{!}{\includegraphics{pics/Scheduler}}
     1282\caption{ \label{Scheduler} IPP Scheduler}
     1283\end{center}
     1284\end{figure}
     1285
     1286The IPP Scheduler sends tasks to the IPP Controller for execution.
    12941287While the IPP Scheduler chooses the tasks to be performed, it is the
    12951288IPP Controller's responsibility to manage the specific tasks executing
    1296 on a given processing node.  Examples of these tasks include the
    1297 process of copying or moving data from the Summit data systems to the
    1298 IPP Image Server; image processing analysis stages performed on the
    1299 science images by the appropriate processing nodes; and the analysis
    1300 of the data in the AP Database.  This division of responsibilites
    1301 allows us to isolate and encapsulate the functionality of the IPP
    1302 Scheduler and the IPP Controller.  With this separation, the IPP
    1303 Controller does not need to have any information about the details of
    1304 the tasks which it executes, while the IPP Scheduler does not need to
    1305 monitor the computer hardware.
     1289on a given processing node.  This division of responsibilites allows
     1290us to isolate and encapsulate the functionality of the IPP Scheduler
     1291and the IPP Controller.  With this separation, the IPP Controller does
     1292not need to have any information about the details of the tasks which
     1293it executes, while the IPP Scheduler does not need to monitor the
     1294computer hardware.
    13061295
    13071296Communication between the IPP Scheduler and the IPP Controller is
    … …  
    13121301but distinct software components.
    13131302
    1314 The IPP Scheduler takes as input the current list of pending images,
    1315 both science and calibration, and a description of the current
    1316 observing plan or strategy on some time-scale.  The IPP Scheduler also
    1317 takes input from humans who manage the IPP.
    1318 
    1319 The IPP Scheduler must choose between several types of analysis tasks
    1320 based on the contents of those lists and on the requirements of the
    1321 users.  The list of tasks which the IPP Scheduler must decide between
    1322 includes:
     1303\subsubsection{Task Rules}
     1304
     1305The IPP Scheduler takes as input a collection of rules which define
     1306the dependency of tasks on certain tests.  The IPP Scheduler must
     1307choose between several types of analysis tasks based on those ruls and
     1308on results of the tests.  The timescale on which different tasks (and
     1309their related tests) are executed may vary from 10s of seconds to
     1310hours, days, or even week.  The list of tasks which the IPP Scheduler
     1311must decide between, and the relevant timescale, follow:
    13231312\begin{itemize}
    13241313\item moving data from the Summit pixel server ($\sim 30$ second timescales)
    … …  
    13271316  nightly)
    13281317\item constructing new detrend images ($\sim$ weekly)
    1329 \item updating and improving the photometric and astrometric reference
    1330   catalogs ($\sim$ yearly).
    13311318\end{itemize}
    1332 
    1333 The IPP Scheduler chooses between tasks which are relevant on several
    1334 different time-scales.  The time-scales range from 2 times per minute
    1335 to once or twice a year, as noted in the list above.  The IPP
    1336 Scheduler must also make use of user input in managing such choices.
    1337 Users have the option to specify that a particular task or set of
    1338 tasks is of higher or lower priority than the norm.
    1339 
    13401319The scheduler may be viewed as a complex state machine.  Our goal is
    1341 to design the rules independently from the engine which parses the
    1342 rules to detemine which specific jobs to send to the controller.
    1343 
    1344 \subsubsection{Scheduler User Interface}
     1320to design the scheduler so that rules may be specified independently
     1321from the engine which parses the rules to detemine which specific jobs
     1322to send to the controller.
     1323
     1324\subsubsection{User Interface}
    13451325
    13461326The IPP Scheduler provides a user interface which allows a human
    13471327operator, or other processes, to monitor the current state of the
    1348 Scheduler. 
     1328Scheduler.  Users have the option to specify that a particular task or
     1329set of tasks is of higher or lower priority than the norm, or to
     1330schedule a particular tasks on a different timescale from the basic
     1331rule.
    13491332
    13501333The IPP Scheduler defines the operating state of the IPP.  When the
    … …  
    13601343for tests or maintenance, in which case the IPP Scheduler does not
    13611344perform even the data copy tasks.  Every task is performed on demand
    1362 by the user.  The user command sets the IPP Scheduler in one of these
     1345by the user.  A user command sets the IPP Scheduler in one of these
    13631346three states, {\em automatic}, {\em interactive}, and {\em paused}.
    13641347
    … …  
    14111394installation running on the Pan-STARRS cluster.  The {\tt base}
    14121395configuration defines general data sources which may be needed by any
    1413 portion of the IPP.  The list of known telescope or filters might be
     1396portion of the IPP.  The list of known telescopes or filters might be
    14141397an example.  The {\tt camera} configuration consists of information
    1415 which defines the parameters relevant to the camera known by the IPP.
     1398which defines the parameters relevant to the cameras known by the IPP.
    14161399For example, the default layout of the detectors or the names of
    14171400specific header keyword values would be defined for each camera in a
    … …  
    14781461
    14791462\begin{verbatim}
    1480 possible command forms:
    1481 
    1482 P1 filename.fits [FPA is single fits file]
    1483 P1 filename.list [FPA is collection of files]
    1484 P1 FPA IA        [FPA info from metadata db]
    1485 
    1486 sources for the input data:
    1487 
    1488 distortion model:
    1489   metadata table
    1490   XML file
    1491   FITS table
    1492   metadata -> image server
    1493   user provided on command line
    1494   recipe provided
    1495 
    1496 camera layout:
    1497   metadata table
    1498   XML file
    1499   FITS table
    1500   metadata -> image server
    1501   user provided on command line
    1502   recipe provided
    1503 
    1504 boresite coordinates guess:
    1505   image header (keywords from recipe)
    1506   metadata table
    1507 
    1508 guide stars
    1509   collection of video streams
    1510   collection of centroid time histories
    1511   list of centroids, coordinates
     1463Phase1 -file filename.fits [FPA is single fits file]
     1464Phase1 -list filename.list [FPA is collection of files]
     1465Phase1 -imdb ID            [FPA is single file in image server]
     1466Phase1 -FPA  ID            [FPA identifier from metadata db]
    15121467\end{verbatim}
    15131468
    … …  
    16421597with the choice a user-configurable option.
    16431598
    1644 The input science and mask frames are trimmed by the extent of the OT
    1645 convolution kernel in each direction ($+x$, $-x$, $+y$, $-y$).  Within
    1646 the PSLib image handling functions, the trim function is a virtual
    1647 operation which simply marks the boundaries of the trimmed image but
    1648 does not remove the corresponding memory.  This allows the later
    1649 corrections to work with untrimmed correction images and still apply
    1650 the correct pixels.  At the end of Phase 2, the only the trimmed
    1651 portions of the output images are written out to disk.
     1599The input science and mask frames are additionally trimmed by the
     1600extent of the OT convolution kernel in each direction ($+x$, $-x$,
     1601$+y$, $-y$).  Within the PSLib image handling functions, the trim
     1602function is a virtual operation which simply marks the boundaries of
     1603the trimmed image but does not remove the corresponding memory.  This
     1604allows the later corrections to work with untrimmed correction images
     1605and still apply the correct pixels.  At the end of Phase 2, the only
     1606the trimmed portions of the output images are written out to disk.
    16521607
    16531608\subsubsection{Non-Linearity Correction}
    … …  
    16611616\subsubsection{Flat field Correction}
    16621617
    1663 The object image (after bias correction and non-linearity correction)
     1618The science image (after bias correction and non-linearity correction)
    16641619must be corrected for sensitivity variations as a function of
    16651620position, dividing by a flat-field image.  The mask is also modified
    … …  
    17601715are sent to the IPP Pixel Server.
    17611716
    1762 \begin{figure}
    1763 \begin{center}
    1764 \resizebox{6in}{!}{\includegraphics{pics/phase2}}
    1765 \caption{ \label{phase2} Phase 2 dataflow}
    1766 \end{center}
    1767 \end{figure}
     1717%\begin{figure}
     1718%\begin{center}
     1719%\resizebox{6in}{!}{\includegraphics{pics/phase2}}
     1720%\caption{ \label{phase2} Phase 2 dataflow - this diagram is old: update}
     1721%\end{center}
     1722%\end{figure}
    17681723
    17691724%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    … …  
    18391794Phase 2.
    18401795
    1841 \begin{figure}
    1842 \begin{center}
    1843 \resizebox{4.5in}{!}{\includegraphics{pics/phase3}}
    1844 \caption{ \label{phase3} Phase 3 dataflow}
    1845 \end{center}
    1846 \end{figure}
    1847 
    18481796%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    18491797
    … …  
    18601808The working concept is that the static sky cells contain roughly the
    18611809same number of pixels as an OTA (4k x 4k) and represent a portion of a
    1862 local tangent plane projection.  As mentioned above
    1863 (Section~\ref{IPP:ImageServer}), the pixel scale of the static sky is
    1864 planned to be 0.2\arcsec, somewhat smaller than the 0.3\arcsec\ raw
    1865 image pixel scale.
     1810local tangent plane projection.  In order to meet the image
     1811degredation requirements, the pixel scale of the static sky is planned
     1812to be 0.2\arcsec, somewhat smaller than the 0.3\arcsec\ raw image
     1813pixel scale.
    18661814
    18671815For each sky cell, the corresponding pixels are extracted from the
    … …  
    18961844\subsubsection{Static Sky Subtraction}
    18971845
    1898 \tbd{add some details about the static-sky subtraction issues.
    1899   Allard-Lupton-Price method}.
     1846The corresponding static sky image is subtracted from the combined
     1847image stack.  In this step, it is necessary to match the image kernel
     1848between the input image and the static sky image.  This will be done
     1849by solving for a best-fit image kernel which minimizes the difference
     1850image using a technique equivalent to the Allard-Lupton method.  The
     1851modification we make is that, rather than represent the components of
     1852the image difference kernel as a combination of Gaussians, we will
     1853represent the kernel as a combination of pixels.  This method also
     1854automatically determines a photometric match between the static sky
     1855image and the input science image.
    19001856
    19011857\subsubsection{Object Detection and Measurement}
    … …  
    19441900adding these objects to the database, the transients which are
    19451901correlated with previous detections of an object (and those which are
    1946 not) will automatically be determined.  An independent process will
    1947 query the AP Database for such transient objects of interest which are
    1948 to be sent, along with their associated metadata, to the MOPS and
    1949 other science client pipelines.  This step must be performed at least
    1950 once per night.
     1902not) will automatically be determined.  A subset of these transient
     1903objects are sent, along with their associated metadata, to the MOPS
     1904and other preferred science client pipelines. 
    19511905
    19521906\subsubsection{Static Sky Update}
    … …  
    19631917a time when the computing infrastructure is not under significant load.
    19641918
    1965 \begin{figure}
    1966 \begin{center}
    1967 \resizebox{6in}{!}{\includegraphics{pics/phase4}}
    1968 \caption{ \label{phase4} Phase 4 dataflow}
    1969 \end{center}
    1970 \end{figure}
     1919%\begin{figure}
     1920%\begin{center}
     1921%\resizebox{6in}{!}{\includegraphics{pics/phase4}}
     1922%\caption{ \label{phase4} Phase 4 dataflow}
     1923%\end{center}
     1924%\end{figure}
    19711925
    19721926%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    … …  
    21552109\section{System Design : Miscellaneous Tasks}
    21562110
    2157 In this section, we discuss the design of the science analysis stages
    2158 which perform the fundamental image analysis steps of the IPP.
     2111In this section, we discuss additional operations which are performed
     2112by the IPP but which do not fall under the analysis of the science
     2113images or the creation of the calibration images. 
    21592114
    21602115\subsection{Retrieval}
    21612116
    2162 The retrieval stages simply retrieve pixel data from an external
    2163 source (ordinarily OATS at the Summit, but it could conceivably be
    2164 some other external source) and store it on the nodes.
     2117The retrieval stage simply retrieves images from an external source
     2118(ordinarily OTIS at the Summit, but it could conceivably be some other
     2119external source) and store it in the Image Server. 
    21652120
    21662121\subsection{Static Sky Analysis}
    … …  
    22252180program in C or through the use of a high-level language such as Perl,
    22262181Python, or Tcl employing the SWIG interfaces.  For the high-level
    2227 functions in the operational system, the IPP will make use of
    2228 \tbd{Python} as the scripting language to provide the required
    2229 flow-control to tie the modules together.
     2182functions in the operational system, the IPP will make use of Perl as
     2183the scripting language to provide the required flow-control to tie the
     2184modules together.
    22302185
    22312186This approach satisfies the requirement that complicated low-level
    … …  
    22692224therefore represents the maximum amount of effort which can be
    22702225performed in serial without interaction between parallel threads.  The
    2271 stages will be written in \tbd{Python}, linking the modules together.
     2226stages will be written in Perl, linking the modules together.
    22722227Examples of stages are Phase 2 (detrend images) and Phase 4 (combine
    22732228images from multiple telescopes and search for transients).
    … …  
    22902245the Controller, and determines the next action based on the contents
    22912246of the Metadata Database.  The various subsystems specify the API for
    2292 client / server interactions with them.  Commands will be sent using
    2293 either text-based commands, SOAP or an equivalent protocol.  The
    2294 format of the exchanged data may be in one of several forms discussed
    2295 below.
     2247client / server interactions, and are discussed in their individual
     2248section.  Commands will be sent using either text-based commands, SOAP
     2249or an equivalent protocol.  The format of the exchanged data may be in
     2250one of several forms discussed below.
    22962251
    22972252FITS Images will be used to transport images between the components of
    … …  
    23182273Pan-STARRS systems and the external clients.  The interfaces are
    23192274illustrated in Figure~\ref{overview}. 
    2320 
    2321 Incoming data is received by
    2322 either the IPS (pixels), the IMD (metadata), or the IOD (objects).
    2323 Requests for data by external clients are also made to these three
    2324 databases.  Requests for data made by the IPP are generated by the IPP
    2325 Scheduler or the science processing pipelines.
    23262275
    23272276\subsubsection{OTIS}
    … …  
    24482397requirements given the above need to 63 processors. 
    24492398
     2399There are two competing trades we will also want to make.  First, we
     2400will want to duplicate data to multiple machines in the network to
     2401protect against catastrophic failures on a single machine.  This
     2402double the total data space needed.  To compensate, however, we will
     2403also employ compression to data, especially data which is older.
     2404These two factors will tend to cancel each other, so we have ignored
     2405both in out calculations above.
     2406
    24502407\tbd{switch information}
    24512408
    2452 \tbd{RAID and compression / duplication plan}
    2453 
    24542409\subsection{PS-1 Cluster Expected Reliability}
    2455  
     2410
     2411With 80 computers and 1920 disks, we must be cautious about component
     2412failures and their impact on operations and data integrity.  There are
     2413several factors which mitigate our exposure to hardware failures.
     2414First, the use of RAID controllers and RAID-5 striping of the data
     2415will protect the data on a single RAID set against the failure of a
     2416single disk in the array.  Second, our plan to have duplication across
     2417the cluster will protect us against catastrophic failures.  Finally,
     2418the flexibility of the distributed computing plan makes it trivial to
     2419handle the loss of individual machines as the system can automatically
     2420redistribute the load across the cluster.
     2421
     2422The components which are most likely to fail in our experience are, in
     2423order: hard drives, ram, power supplies, and other components.  The
     2424hard drive failure rate is by far the dominant concern as it
     2425potentially affects the data integrity. 
     2426
     2427Most sources (REFS: UCSD article, Samsung White Paper) currently imply
     2428hard disk failure rates (MTBF) in the range 400,000 hours and 500,000
     2429hours.  We take these as an upper limit, and instead adopt a
     2430conservative value of 100,000 hours.  With 1920 disk, this MTBF
     2431implies a failure of one disk every 2.2 days.  Since the disks are in
     2432a RAID which reports the disk failures immediately and drops the array
     2433into degraded mode, these failures will not have a huge impact on the
     2434operations, and recovery time is only 10s of minutes.  This failure
     2435rate implies that we should be checking for hard disk failures daily.
     2436\tbd{is it necessary to catch failures at night or can the system run
     2437with a degraded disk?}.  A catastrophic failure for the array would
     2438require two of the 12 disks to fail before the first failed disk is
     2439replaced.  If we assume that maintainence is poor and it is possible
     2440for a disk to take 1 week to be replaced, we calculate a probability
     2441of a catastrophe of 1.8\% each time a disk fails.  Combined with the
     2442disk failure rate, we can expect a RAID catastrophe 6 times over the 2
     2443year operation of PS-1.  We can use these numbers as a guideline for
     2444our level of support needed to avoid these RAID failures.  Note that
     2445these 6 failures should not cause loss of data since the data is
     2446duplicated across the cluster, but they require over 1 day for
     2447recovery (as the entire array must be replicated across the network).
     2448
    24562449\subsection{PS-1 Cluster Support}
    24572450
    24582451\begin{figure}
    24592452\begin{center}
    2460 \resizebox{6in}{!}{\includegraphics{pics/ps1_ipp_storage.ps}}
     2453\resizebox{6in}{!}{\includegraphics[angle=-90]{pics/ps1_ipp_storage.ps}}
    24612454\caption{ \label{StorageProfile} Storage Profile}
    24622455\end{center}
    … …  
    24702463
    24712464\subsection{Image Server Database Table Contents}
    2472 \ref{ImageServerTableContents}
    2473 
    2474 \begin{table}
     2465\label{ImageServerTableContents}
     2466
     2467Tables~\ref{ImageServerTables:SO} - \ref{ImageServerTables:VOL} list
     2468the basic contents of the Image Server database tables. 
     2469
     2470\begin{table}[bh]
    24752471\begin{center}
    24762472\caption{Storage Object Table Contents\label{ImageServerTables:SO}}
    … …  
    24892485\end{table}
    24902486
    2491 \begin{table}
     2487\begin{table}[bh]
    24922488\begin{center}
    24932489\caption{Instance Table Contents\label{ImageServerTables:INT}}
    … …  
    25092505\end{table}
    25102506
    2511 \begin{table}
     2507\begin{table}[bh]
    25122508\begin{center}
    25132509\caption{Volume Table Contents\label{ImageServerTables:VOL}}
    … …  
    25262522
    25272523\subsection{Metadata Database Table Contents}
    2528 \ref{MetadataTableContents}
    2529 
    2530 Tables \tbd{NN} -- \tbd{NN} list the basic contents of each of the
    2531 Metadata Database tables listed in Section~\ref{Metadata}.
    2532 
    2533 \begin{table}
     2524\label{MetadataTableContents}
     2525
     2526Tables~\ref{WeatherTable} -- \ref{overlaps} list the basic contents of
     2527each of the Metadata Database tables listed in Section~\ref{Metadata}.
     2528
     2529\begin{table}[bh]
    25342530\begin{center}
    25352531\caption{Weather Table: some sample weather points\label{WeatherTable}}
    … …  
    25502546\end{table}
    25512547
    2552 \begin{table}
     2548\begin{table}[bh]
    25532549\begin{center}
    25542550\caption{SkyProbe Transparency Table (sample entries)\label{SkyprobeBVTable}}
    … …  
    25702566\end{table}
    25712567
    2572 \begin{table}
     2568\begin{table}[bh]
    25732569\begin{center}
    25742570\caption{Skyprobe Line Absorption Table (sample entries)\label{SkyprobeATable}}
    … …  
    25932589\end{table}
    25942590
    2595 \begin{table}
     2591\begin{table}[bh]
    25962592\begin{center}
    25972593\caption{Skyprobe Line Emission Table (sample entries)\label{SkyprobeETable}}
    … …  
    26142610\end{table}
    26152611
    2616 \begin{table}
     2612\begin{table}[bh]
    26172613\begin{center}
    26182614\caption{DIMM Measurements Table\label{DimmTable}}
    … …  
    26352631\end{table}
    26362632
    2637 \begin{table}
     2633\begin{table}[bh]
    26382634\begin{center}
    26392635\caption{Near IR Wide-field Camera Results Table\label{NIR-Table}}
    … …  
    26542650\end{table}
    26552651
    2656 \begin{table}
     2652\begin{table}[bh]
    26572653\begin{center}
    26582654\caption{Dome Status Table\label{DomeStatusTable}}
    … …  
    26722668\end{table}
    26732669
    2674 \begin{table}
     2670\begin{table}[bh]
    26752671\begin{center}
    26762672\caption{Telescope Status\label{TelescopeStatusTable}}
    … …  
    26912687\end{table}
    26922688
    2693 \begin{table}
     2689\begin{table}[bh]
    26942690\begin{center}
    26952691\caption{Raw FPA Images\label{RawFPAs}}
    … …  
    27212717\end{table}
    27222718
    2723 \begin{table}
     2719\begin{table}[bh]
    27242720\begin{center}
    27252721\caption{Pending Science Chips\label{PendingChips}}
    … …  
    27372733\end{table}
    27382734
    2739 \begin{table}
     2735\begin{table}[bh]
    27402736\begin{center}
    27412737\caption{Processed Science Chips\label{ProcessedChips}}
    … …  
    27542750\end{table}
    27552751
    2756 \begin{table}
     2752\begin{table}[bh]
    27572753\begin{center}
    27582754\caption{Observation Group Information\label{OBS}}
    … …  
    27722768\end{table}
    27732769
    2774 \begin{table}
     2770\begin{table}[bh]
    27752771\begin{center}
    27762772\caption{Observation Frame Information\label{OBS}}
    … …  
    27902786\end{table}
    27912787
    2792 \begin{table}
     2788\begin{table}[bh]
    27932789\begin{center}
    27942790\caption{Science Processing Stats\label{PSStats}}
    … …  
    28282824\end{table}
    28292825
    2830 \begin{table}
     2826\begin{table}[bh]
    28312827\begin{center}
    28322828\caption{Chip / Sky overlaps\label{overlaps}}
    … …  
    28442840\end{table}
    28452841
    2846 \begin{table}
     2842\begin{table}[bh]
    28472843\begin{center}
    2848 \caption{Processed Sky-Cell stats\label{}}
     2844\caption{Processed Sky-Cell stats\label{ProcessedSky}}
    28492845\begin{tabular}{lll}
    28502846\hline
    … …  
    28582854Diff image params  & string        & Parameters used for the image differencing. \\
    28592855Average weight     & string        & The weight of the reference image \\
    2860 P4D object stats   & string        & Summary statistics of the object detection (number of objects, depth, other input parameters). \\
    2861 P4S object stats   & string        & Summary statistics of the object detection (number of objects, depth, other input parameters). \\
     2856P4D object stats   & string        & Summary statistics of the object detection \\
     2857P4S object stats   & string        & Summary statistics of the object detection \\
    28622858Software versions  & string        & Software versions of modules used in the sky cell processing. \\
    28632859Processing stats   & string        & Summary statistics of the processing (CPU time, etc). \\
    … …  
    28692865
    28702866\subsection{AP Database Table Contents}
    2871 \ref{APDBTableContents}
    2872 
     2867\label{APDBTableContents}
     2868
     2869\tbd{Table contents to be defined}
    28732870
    28742871%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

  • trunk/doc/design/ippSRS.tex

    r2186 r2192  
    1  %%% $Id: ippSRS.tex,v 1.10 2004-10-21 03:55:59 eugene Exp $
     1 %%% $Id: ippSRS.tex,v 1.11 2004-10-22 04:43:35 eugene Exp $
    22\documentclass[panstarrs,spec]{panstarrs}
    33
    … …  
    180180
    181181\begin{enumerate}
    182 \item For images obtained in photometric weather, produce reduced
    183   science images for each full camera exposure with photometric
    184   zero-point scatter less than 1\% across the full
    185   field. \VER{ANALYSIS}{SCD:3.2.2.5}
     182\item For images obtained in photometric weather with normal detector
     183  characteristics and providing appropriate flat-field images and
     184  correction data have been obtained, the IPP shall produce reduced
     185  science images for each full camera exposure with relative
     186  photometric zero-point scatter less than 1\% ($1 \sigma$) across the
     187  full field. \VER{ANALYSIS}{SCD:3.2.2.5}
    186188  \label{TLR:1}
    187189
    188 \item For images obtained in photometric weather, produce reduced
    189   science images for each full camera exposure which are
    190   photometrically calibrated with respect to the Pan-STARRS filter
    191   system with a 1$\sigma$ accuracy of 1\%.\VER{ANALYSIS}{SCD:3.2.2.5}
     190\item For images of reference fields calibrated for the IPP filter set
     191  and obtained in photometric weather with normal detector
     192  characteristics and providing appropriate flat-field images and
     193  correction data have been obtained, the IPP shall determine and
     194  track zero-points for these exposures with a 1$\sigma$ accuracy of
     195  1\%.\VER{ANALYSIS}{SCD:3.2.2.5}
    192196  \label{TLR:2}
    193197
    194198\item For images obtained under normal seeing conditions and optical
    195   distortion, produce reduced science images for each full camera
    196   exposure with an astrometric calibration providing $< 30$
    197   milliarcsecond scatter (1$\sigma$) for sequential images of the same
    198   location.\VER{ANALYSIS}{SCD:3.2.2.7}
     199  distortion, the IPP shall produce reduced science images for each
     200  full camera exposure with an astrometric calibration providing $<
     201  30$ milliarcsecond scatter (1$\sigma$) for sequential images of the
     202  same location.\VER{ANALYSIS}{SCD:3.2.2.7}
    199203  \label{TLR:4}
    200204
    201205\item For images obtained under normal seeing conditions and optical
    202   distortion, produce reduced science images for each full camera
    203   exposure with an astrometric calibration providing $< 100$
    204   milliarcsecond scatter (1$\sigma$) relative to the ICRS reference
    205   system.\VER{ANALYSIS}{SCD:3.2.2.6}
     206  distortion, the IPP shall produce reduced science images for each
     207  full camera exposure with an astrometric calibration providing $<
     208  100$ milliarcsecond scatter (1$\sigma$) relative to the ICRS
     209  reference system.\VER{ANALYSIS}{SCD:3.2.2.6}
    206210  \label{TLR:3}
    207211
    208212\item In photometric weather and under moon conditions listed in
    209   Table~\ref{moonconditions}, produce reduced science images for each
    210   full camera exposure which have background variations of less than
    211   1\% in regions free of large ($> 30$ pixels diameter) astronomical
    212   structures.\VER{ANALYSIS}{SCD:3.5.12}
     213  Table~\ref{moonconditions}, the IPP shall produce reduced science
     214  images for each full camera exposure which have background
     215  variations of less than 1\% in regions free of large ($> 30$ pixels
     216  diameter) astronomical structures.\VER{ANALYSIS}{SCD:3.5.12}
    213217  \label{TLR:5}
    214218
    215 \item In photometric weather, produce reduced science images for each
    216   full camera exposure which have background deviations from the
    217   static sky in the same filter of less than 1\% for the median in
    218   large ($> 30$ pixels diameter) regions.\VER{ANALYSIS}{SCD:3.5.12}
     219\item In photometric weather, the IPP shall produce reduced science
     220  images for each full camera exposure which have background
     221  deviations from the static sky in the same filter of less than 1\%
     222  for the median in large ($> 30$ pixels diameter)
     223  regions.\VER{ANALYSIS}{SCD:3.5.12}
    219224  \label{TLR:5a}
    220225
    221 \item Merge all $g$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
     226\item The IPP shall merge all $g$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
    222227  \label{TLR:6}
    223228
    224 \item Merge all $r$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
     229\item The IPP shall merge all $r$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
    225230  \label{TLR:7}
    226231
    227 \item Merge all $i$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
     232\item The IPP shall merge all $i$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
    228233  \label{TLR:8}
    229234
    230 \item Merge all $z$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
     235\item The IPP shall merge all $z$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
    231236  \label{TLR:9}
    232237
    233 \item Merge all $y$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
     238\item The IPP shall merge all $y$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
    234239  \label{TLR:10}
    235240
    236 \item Merge all $w$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
     241\item The IPP shall merge all $w$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
    237242  \label{TLR:11}
    238243
    239 \item Detect and classify objects on the individual processed science
     244\item The IPP shall detect and classify objects on the individual processed science
    240245  images.\VER{TASK}{SCD:3.2.2.16}
    241246  \label{TLR:12}
    242247
    243 \item Detect and classify objects on the stacked groups of science
    244   images.\VER{TASK}{SCD:3.2.2.16}
     248\item The IPP shall detect and classify objects on the stacked groups
     249  of science images.\VER{TASK}{SCD:3.2.2.16}
    245250  \label{TLR:13}
    246251
    247 \item Detect and classify objects on the static sky image.\VER{TASK}{SCD:3.2.2.16}
     252\item The IPP shall detect and classify objects on the static sky
     253  image.\VER{TASK}{SCD:3.2.2.16}
    248254  \label{TLR:14}
    249255
    250 \item Detect transients with significance $>3\sigma$ in the individual
    251   science images relative to the static sky
     256\item The IPP shall detect transients with significance $>3\sigma$ in
     257  the individual science images relative to the static sky
    252258  image.\VER{ANALYSIS}{SCD:3.2.2.16}
    253259  \label{TLR:15}
    254260
    255 \item Degrade the stacked image by no more than \tbr{10
     261\item The IPP shall degrade the stacked image by no more than \tbr{10
    256262  milliarcseconds (FWHM added in quadrature)} over the theoretical
    257263  limit for the stack under infinite
    … …  
    259265  \label{TLR:16}
    260266
    261 \item Perform the processing of science images to the level of
    262   transient detection and static sky inclusion at a rate such that
    263   exposures taken at an \tbr{average cadence of 40 seconds} do not
    264   accumulate in the processing buffer (average throughput
     267\item The IPP shall perform the processing of science images to the
     268  level of transient detection and static sky inclusion at a rate such
     269  that exposures taken at an \tbr{average cadence of 40 seconds} do
     270  not accumulate in the processing buffer (average throughput
    265271  requirement).\VER{TEST}{SCD:3.2.2.3}
    266272  \label{TLR:17}
    267273
    268 \item Limit the \tbr{false alarm rate (FAR) to less than 5\%} for
    269   transient detections $> 5\sigma$ sent to the preferred client
     274\item The IPP shall limit the false alarm rate (FAR) to less than 5\%
     275  for transient detections $> 5\sigma$ sent to the preferred client
    270276  science pipelines.\footnote{note difference with PS-4: 1\%}
    271277  \VER{ANALYSIS}{SCD:3.2.2.13}
    272278 \label{TLR:18}
    273279
    274 \item Perform \tbr{transient detection to a completeness of 99\%} at
    275   the completeness for transient detections with a significant $>
    276   5\sigma$.\VER{ANALYSIS}{SCD:xxx}
    277 
    278 \item Publish the static sky images to the Pan-STARRS Published
    279   Science Products Subsystem (PSPS) once per \tbr{6
    280   months}.\VER{TASK}{SCD:3.2.2.18}
     280\item The IPP shall perform transient detection to a completeness of
     281  99\% at the completeness for transient detections with a significant
     282  $> 5\sigma$.\VER{ANALYSIS}{SCD:xxx}
     283
     284\item The IPP shall publish the static sky images to the Pan-STARRS
     285  Published Science Products Subsystem (PSPS) at a rate so the full
     286  sky is transmitted once per year.\VER{TASK}{SCD:3.2.2.18}
    281287  \label{TLR:19}
    282288
    283 \item Publish the detected objects to the Pan-STARRS Published Science
    284   Products Subsystem (PSPS) once per month.\VER{TASK}{SCD:3.2.2.18}
     289\item The IPP shall publish the detected objects to the Pan-STARRS
     290  Published Science Products Subsystem (PSPS) at a rate such that the
     291  objects from the full sky are transmitted once per
     292  year.\VER{TASK}{SCD:3.2.2.18}
    285293  \label{TLR:20}
    286294
    287 \item Send the IPP metadata and received OTIS metadata to the
    288   Pan-STARRS Published Science Products Subsystem (PSPS)
     295\item The IPP shall send the IPP metadata and received OTIS metadata
     296  to the Pan-STARRS Published Science Products Subsystem (PSPS)
    289297  weekly.\VER{TASK}{SCD:3.2.2.18}
    290298  \label{TLR:21}
    291299
    292 \item Provide access to preferred Pan-STARRS science clients to the
     300\item The IPP shall provide access to preferred Pan-STARRS science clients to the
    293301  detected transient objects within \tbr{5 minutes}.\VER{TEST}{SCD:3.5.10}
    294302  \label{TLR:22}
    295303
    296 \item Provide sufficent storage volume for raw images from the AP and
     304\item The IPP shall provide sufficent storage volume for raw images from the AP and
    297305  IVP Surveys and the \grizy\ Static Sky.\footnote{note difference with
    298306  PS-4: 1 month of raw images} \VER{INSPECT}{allocated}
    299307  \label{TLR:23}
    300308
    301 \item Provide sufficient storage volume for all detections from the
     309\item The IPP shall provide sufficient storage volume for all detections from the
    302310  AP, IVP, and MVP Surveys.\footnote{note difference with PS-4: 1 year
    303311  of detections}\VER{INSPECT}{allocated}
    304312  \label{TLR:24}
    305313
    306 \item Provide sufficient storage volume for 2 years of
     314\item The IPP shall provide sufficient storage volume for 2 years of
    307315 metadata.\footnote{note difference with PS-4: 10 years of
    308316 metadata}\VER{INSPECT}{allocated}
    … …  
    787795\item The AP Database shall accept new detections at the rate
    788796  generated by the telescope from the Phase 2 and Phase 4 analysis.
    789   \tbr{Except within 10 degrees of the galactic plane, the AP Database
    790   shall keep up with the incoming rates.}  The expected rates are
     797  Except within 10 degrees of the galactic plane, the AP Database
     798  shall keep up with the incoming rates.  The expected rates are
    791799  listed in Table~\ref{APrates}, along with the total data volume
    792   required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2, TLR:3, TLR:22}
     800  required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2,
     801  TLR:3, TLR:22}
    793802
    794803\item The AP Database shall provide access to external Pan-STARRS
    … …  
    940949  computers.\VER{TEST}{TLR:17}
    941950
    942 \item The IPP Controller shall limit command latency to \tbr{$< 0.1$} seconds.\VER{TEST}{TLR:17}
    943 
    944 \item The IPP Controller shall be capable of performing up to \tbr{10 tasks per second}.\VER{TEST}{TLR:17}
    945 
    946 \item The IPP Controller shall be capable of buffering up to a total of \tbr{64 MB} of messages.\VER{TEST}{TLR:17}
    947 
    948 \item The IPP Controller shall be capable of executing up to \tbr{6 million tasks per month}.\VER{TEST}{TLR:17}
    949 
    950 \item The IPP Controller shall be capable of interacting with up to \tbr{256} client processes.\VER{TEST}{TLR:17}
     951\item The IPP Controller shall limit command latency to $< 0.1$ seconds.\VER{TEST}{TLR:17}
     952
     953\item The IPP Controller shall be capable of performing up to 10 tasks per second.\VER{TEST}{TLR:17}
     954
     955\item The IPP Controller shall be capable of buffering up to a total of 64 MB of messages.\VER{TEST}{TLR:17}
     956
     957\item The IPP Controller shall be capable of executing up to 6 million tasks per month.\VER{TEST}{TLR:17}
     958
     959\item The IPP Controller shall be capable of interacting with up to 256 client processes.\VER{TEST}{TLR:17}
    951960
    952961\item The IPP Controller shall be capable of accepting up to 2 non-client (external) requests per second.\VER{TEST}{TLR:17}
    … …  
    987996\begin{enumerate}
    988997\item The IPP Scheduler shall publish the static sky images to the
    989   Pan-STARRS PSPS on a time-scale of \tbr{6 month}.\VER{TEST}{TLR:19}
     998  Pan-STARRS PSPS at a rate so that the full sky is transmitted once
     999  per year.\VER{TEST}{TLR:19}
    9901000
    9911001\item The IPP Scheduler shall query the Databases on a regular basis
    9921002  to check for new input information.  These queries shall take place
    993   at least once every \tbr{1 seconds}.\VER{TEST}{TLR:17}
     1003  at least once every second.\VER{TEST}{TLR:17}
    9941004
    9951005\item The IPP Scheduler shall accept new User input in real-time:
    … …  
    9971007
    9981008\item The IPP Scheduler shall publish the detected objects to the
    999   Pan-STARRS PSPS on a time-scale of \tbr{1 month}.\VER{TEST}{TLR:20}
     1009  Pan-STARRS PSPS at a rate so that the objects from the full sky are
     1010  transmitted once per year.\VER{TEST}{TLR:20}
    10001011
    10011012\item The IPP Scheduler shall publish the IPP and OTIS metadata to the
    1002   Pan-STARRS PSPS on a time-scale of \tbr{1 week}.\VER{TEST}{TLR:21}
     1013  Pan-STARRS PSPS on a time-scale of 1 week.\VER{TEST}{TLR:21}
    10031014
    10041015\item The IPP Scheduler shall send the detected single-occurance
    … …  
    10851096
    10861097\item Calculate the Image cell / Sky cell overlaps for each image.
    1087   Sky cells which do not have sufficient science image overlap \tbr{$<
    1088   5\%$} are excluded from the overlap table.
     1098  Sky cells which do not have sufficient science image overlap $< 5\%$
     1099  are excluded from the overlap table.
    10891100
    10901101\end{itemize}
    … …  
    11011112
    11021113\item Bright-star extraction from the image data shall be performed in
    1103   less than \tbr{1 second}.\VER{TEST}{TLR:17}
     1114  less than 1 second.\VER{TEST}{TLR:17}
    11041115 
    11051116\item In order for blind astrometry of an image to succeed, it is
    11061117  necessary that approximate image coordinates be known.  The Phase 1
    11071118  analysis shall succeed despite initial coordinate errors as large as
    1108   \tbr{20\arcsec}.\VER{TEST}{TLR:3}
     1119  20\arcsec.\VER{TEST}{TLR:3}
    11091120 
    11101121\end{enumerate}
    … …  
    11281139
    11291140\item Mask ghosts of bright stars which introduce residual feature
    1130   more significant than \tbr{1\%} of the background.
     1141  more significant than 1\% of the background.
    11311142
    11321143\item Bias subtract the image.
    … …  
    11881199  time. \VER{TEST}{TLR:17}
    11891200
    1190 \item The bias subtraction shall leave no residuals greater than
    1191   \tbr{1 DN} peak-to-peak for images within the normal range of bias
     1201\item The bias subtraction shall leave no residuals greater than 1 DN
     1202  peak-to-peak for images within the normal range of bias
    11921203  variations.\VER{TEST}{TLR:1}
    11931204
    … …  
    12061217
    12071218\item The background residuals shall have peak-to-peak variations of
    1208   less than \tbr{1\%} of the input background amplitude.\VER{ANALYSIS}{TLR:5}
    1209 
    1210 \item The background residuals shall have a scatter of less than
    1211   \tbr{1\%} of the input background amplitude for apertures of less
    1212   than \tbr{10 arcsec}.\VER{ANALYSIS}{TLR:1}
     1219  less than 1\% of the input background
     1220  amplitude.\VER{ANALYSIS}{TLR:5}
     1221
     1222\item The background residuals shall have a scatter of less than 1\%
     1223  of the input background amplitude for apertures of less than 10
     1224  arcsec.\VER{ANALYSIS}{TLR:1}
    12131225
    12141226\item The Phase 2 analysis shall detect cosmic rays with flux $> 5\sigma$ by
    … …  
    12351247  photometric zero point and zero-point variations across the field.
    12361248
    1237 \item If zero-point variations are significant (\tbr{$> 0.01$ mag
    1238   peak-to-peak}), the zero-point variations are modeled with a
     1249\item If zero-point variations are significant ($> 0.01$ mag
     1250  peak-to-peak), the zero-point variations are modeled with a
    12391251  polynomial correction of order 3 or less.
    12401252
    … …  
    14371449
    14381450\begin{enumerate}
    1439 \item The IPP Calibration Analysis shall produce master calibration images
    1440 from the raw calibration images in less \tbr{2 hours}.\VER{TEST}{TLR:17, TLR:22}
     1451\item The IPP Calibration Analysis shall produce master calibration
     1452  images from the raw calibration images in less 2
     1453  hours.\VER{TEST}{TLR:17, TLR:22}
    14411454
    14421455\item Master calibration images shall not introduce systematic
    1443  uncertainties in the photometry greater than \tbr{0.2\%}.\VER{TEST}{TLR:1}
     1456 uncertainties in the photometry greater than 0.2\%.\VER{TEST}{TLR:1}
    14441457
    14451458\end{enumerate}
    … …  
    14771490
    14781491\item The Dark calibration stage raises an error if the input images
    1479   have exposure times which deviate by more than
    1480   \tbr{2\%}.
     1492  have exposure times which deviate by more than 2\%.
    14811493
    14821494\item The Dark calibration stage corrects the input dark images for
    … …  
    15381550
    15391551\item The Mask calibration stage masks the pixels which are
    1540   inconsistent in the input flats by more than \tbr{1\%}, given
    1541   sufficient signal-to-noise in the input flats.
     1552  inconsistent in the input flats by more than 1\%, given sufficient
     1553  signal-to-noise in the input flats.
    15421554
    15431555\item The Mask calibration stage mask the pixels which are
    15441556  consistently low or high in the input flats by more than a factor of
    1545   \tbr{3} beyond the typical pixel.
     1557  3 beyond the typical pixel.
    15461558
    15471559\item The Mask calibration stage masks the pixels identified in a
    … …  
    16601672\item The IPP Calibration system monitors changes in the telescope
    16611673  astrometry parameters and issue a warning if the parameters change
    1662   by more than \tbr{2\%}.
     1674  by more than 2\%.
    16631675\end{itemize}
    16641676
Note: See TracChangeset for help on using the changeset viewer.