Index: /trunk/doc/design/ippSDRS.tex
===================================================================
--- /trunk/doc/design/ippSDRS.tex	(revision 2191)
+++ /trunk/doc/design/ippSDRS.tex	(revision 2192)
@@ -1,3 +1,3 @@
-%%% $Id: ippSDRS.tex,v 1.10 2004-10-21 03:55:59 eugene Exp $
+%%% $Id: ippSDRS.tex,v 1.11 2004-10-22 04:43:35 eugene Exp $
 \documentclass[panstarrs]{panstarrs}
 
@@ -6,4 +6,5 @@
 \subtitle{Supplementary Design Requirements Specification}
 \shorttitle{IPP SDRS}
+\audience{Pan-STARRS PMO}
 \author{Eugene Magnier, Paul Price, Josh Hoblitt}
 \group{Pan-STARRS Algorithm Group}
@@ -30,16 +31,4 @@
 
 \listoffigures
-
-\begin{verbatim}
-TODOs
-- add hardware org diagram to section 3
-- clean 3.4 system ifs: describe types of interactions, which are push which are pull?
-- 3.5: move to 3.1?  summary of requirements?
-- add Image Server figure
-- discuss AP DB operations: addstar, delstar, relphot, etc
-- discuss AP DB throughput issues
-- unify controller discussion 
-- scheduler: distinguish states
-\end{verbatim}
 
 \pagebreak
@@ -169,18 +158,29 @@
 
 The users of the IPP output are all systems internal to the Pan-STARRS
-project.  They consist of the Transient Science Client, which will
-receive the detections of transient objects on short time-scales; the
-Moving Object Processing System (MOPS), which will receive the
-detections of non-stationary transient objects on day-to-week
-timescales; and the Published Science Products Subsystem (PSPS), which
-will receive all data products of interest to the outside world, and
-will act as the long-term archive and publishing clearinghouse.
-
-The primary IPP hardware system on which the software operates will
-not be located at the summit.  Instead, because of thermal, power, and
-space constraints, the hardware will likely be located in a facility
-off the mountain.  A subset of processing tasks may eventually be
-assigned to machines at the summit if justified by the savings in data
-transfer time and cost.
+project.  They consist of: 1) the Preferred Science Clients, which
+receive specified data products on short timescales.  2) the Moving
+Object Processing System (MOPS), which is one of the Preferred Science
+Clients, but has the distinction of being a component funded by
+Pan-STARRS.  It will receive the detections of non-stationary
+transient objects.  3) the Published Science Products Subsystem
+(PSPS), which will receive all data products of interest to the
+outside world, and will act as the long-term archive and publishing
+clearinghouse.
+
+The IPP receives data from two Pan-STARRS subsystems: the Camera, from
+which it receives the large volume of image data, and OTIS
+(Observatory, Telesope and Infrastructure Subsystem), from which it
+receives metadata describing the images and the environmental
+conditions.  The primary IPP hardware system on which the software
+operates will not be located at the summit.  Instead, because of
+thermal, power, and space constraints, the hardware will likely be
+located in a facility off the mountain.  A subset of processing tasks
+may eventually be assigned to machines at the summit if justified by
+the savings in data transfer time and cost.
+
+The Pan-STARRS camera produces images consisting of multiple chips
+(Orthogonal Transfer Arrays or OTAs), each consisting of multiple
+cells (continuous set of pixels).  The baseline design for the
+Pan-STARRS camera contains 64 chips each with 64 cells.
 
 This document defines the design requirements of the IPP for the PS-1
@@ -195,42 +195,35 @@
 several important ways.  First, with only one telescope and camera,
 the data throughput rate is substantially reduce to a maximum of 1
-64-OTA image per 40 seconds rather than 4.  In addition, much of the
-PS-1 mission will be devoted to calibration and testing which will
-imply a different level of processing.  For a significant fraction of
-PS-1, data will be obtained for the AP Survey covering the entire
-$3\pi$ steradians of the sky accessible to PS-4.  These images will
-not initially be analysed to the level of having multiple images
-combined.  Rather, the analysis will only be performed for individual
-focal plane array images.  Only after the AP Survey is done, the
-analysis process has been validated, and the complete AP Survey
-reference catalog has been generated will it be possible to generate
-the first epoch static sky image, rougly 18 months into the PS-1
-mission.  This difference in approach has implications for the storage
-required by PS-1: rather than delete images soon after they have been
-used, raw images must be stored for at least the first 18 months of
-PS-1 operations.
+64-OTA image per 40 seconds rather than 4.  Since PS-1 is a prototype
+for testing the Pan-STARRS hardware and software subsystems, the
+observing strategy is not a fixed quantity.  The PS-1 Design Reference
+Mission (PSDC-xxx) provides some guidelines for the types of projects
+to be performed, including starting an AP Survey which will eventually
+cover the entire $3\pi$ steradians of the sky accessible to PS-4.  As
+a prototype, it is expected that much of the data collected by PS-1
+will be processed multiple times to test and tune the analysis steps.
+This difference in approach has implications for the storage required
+by PS-1: rather than delete images soon after they have been used, raw
+images must be stored for at least the first 18 months of PS-1
+operations.  We have used the PS-1 Design Reference Mission as a
+baseline for these storage requirements to drive our hardware design.
 
 \subsection{System Design Decisions}
 
-Since Pan-STARRS is a survey project, all data from the telescopes will be
-uniformly analysed by the Pan-STARRS Image Processing Pipeline (IPP) and
-the appropriate resulting data products made available to internal and
-external science analysis systems as they become available.  The
-processing performed by the IPP on the science images will consist of
-detrending and object detection for the individual images, combination
-of multiple overlapping images and further object detection,
-subtraction of a reference (static-sky) image and detection of
-residual objects, update of the static sky images, and detailed object
-analysis of the static sky images.  In addition, the IPP will produce
-improved astrometric and photometric reference catalogs on an
+Since Pan-STARRS is a survey project, all data from the telescopes
+will be uniformly analysed by the Pan-STARRS Image Processing Pipeline
+(IPP) and the appropriate resulting data products made available to
+internal and external science analysis systems as they become
+available.  The processing performed by the IPP on the science images
+will consist of detrending and object detection for the individual
+images, combination of multiple overlapping images and further object
+detection, subtraction of a reference (static-sky) image and detection
+of residual objects, update of the static sky images, and detailed
+object analysis of the static sky images.  In addition, the IPP will
+produce improved astrometric and photometric reference catalogs on an
 occasional basis as needed.  The output data products from the IPP
 consist of the calibration images, reduced images from the individual
 telescopes, combined images, difference images, the static sky image,
 object photometry, and reference astrometry and photometry.
-
-The IPP interacts closely with other Pan-STARRS systems responsible for
-other aspects of the Pan-STARRS operation, including the summit systems
-(OATS), the science object database, the Moving Object Processing
-System (MOPS), and potentially other client science pipelines.
 
 The requirements for the IPP, as identified in the IPP SRS (PSDC-REF)
@@ -264,10 +257,17 @@
 
 Depending on the particular stage, it may process individual images,
-collections of images, or on derived data products.  Because of the
+collections of images, or derived data products.  Because of the
 nature of the image data, many of the analysis stages can be run in
-parallel because, for example, the analysis of a chip in one image
-does not depend on the results from another chip.
+parallel.  For example, the analysis of a chip in one image does not
+depend on the results from another chip.
 
 \subsection{Architectural Components}
+
+\begin{figure}
+\begin{center}
+\resizebox{6in}{!}{\includegraphics{pics/IPPoverview}}
+\caption{ \label{overview} IPP System Overview}
+\end{center}
+\end{figure}
 
 In order to achieve the required functionality, the IPP provides an
@@ -324,17 +324,10 @@
 \begin{figure}
 \begin{center}
-\resizebox{6in}{!}{\includegraphics{pics/IPPoverview}}
-\caption{ \label{overview} IPP System Overview}
-\end{center}
-\end{figure}
-
-\subsection{IPP Hardware Organization}
-
-\begin{figure}
-\begin{center}
 \resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}}
 \caption{ \label{hardware} IPP Hardware Organization}
 \end{center}
 \end{figure}
+
+\subsection{IPP Hardware Organization}
 
 The IPP needs substantial computer resources, both in terms of
@@ -356,16 +349,17 @@
 those that provide the storage for the other data systems, the
 Metadata Database and the AP Database.  In addition, the computational
-tasks related to Phase 2 take place on the per-OTA storage nodes and
-the Phase 4 computation takes place on the static sky storage nodes.
+tasks related to the individual images take place on the per-OTA
+storage nodes and the processing of stacks of images takes place on
+the static sky storage nodes.
 
 Figure~\ref{hardware} shows our basic concept for the hardware
 organization for the IPP.  This diagram shows the two types of compute
-nodes: OTA-level processing and storage nodes (dominated by Phase 2)
-and static sky processing and storage nodes (mostly Phase 4).  Also
-shown are two switches which divide the network into OTA and
-Static-Sky portions.  In such an organization, the interswitch
-communication must meet the throughput needs between these network
-portions.  The additional data systems (Metadata Database and AP
-Database) are also shown.
+nodes: OTA-level processing and storage nodes and static sky
+processing and storage nodes.  Also shown are two switches which
+divide the network into OTA and Static-Sky portions.  In such an
+organization, the interswitch communication must meet the throughput
+needs between these network portions (though a single switch may also
+be used if its backplane capacity is sufficient).  The additional data
+systems (Metadata Database and AP Database) are also shown.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -390,5 +384,5 @@
 across a collection of computer nodes, each with their own data
 storage resources.  Any single file is stored on only a single
-computer and storage system.  In order to achieve the data throughput
+computer and storage device.  In order to achieve the data throughput
 requirements, the IPP Image Server may distribute the images across
 the processor nodes in an organized fashion, i.e., associating
@@ -405,5 +399,5 @@
 
 \item {\bf instance} A single copy of the storage object in the Image
-  Server.  In general, given storage object may have several instances
+  Server.  In general, a given storage object may have several instances
   in the Image Server, normally on different computer nodes.
 
@@ -413,10 +407,11 @@
 \end{itemize}
 
-The Image Server provides file pointers (in C), handles (in Perl), or
-file names corresponding to the instances of the storage objects.  The
-Image Server provides the data organization but does not define a file
-system; it assumes the existence of an appropriate file system which
-provides makes the files visible as local files.  This may be done
-over many machines with a network file system such as NFS or GFS.
+The Image Server provides file pointers (in C), handles (in Perl or
+Python), or file names corresponding to the instances of the storage
+objects.  The Image Server provides the data organization but does not
+define a file system; it assumes the existence of an appropriate file
+system which makes the files visible as local files.  This
+may be done over many machines with a network file system such as NFS
+or GFS.
 
 The IPP Image Server provides the storage and access mechanisms, but
@@ -432,4 +427,5 @@
 \item Image Server daemon
 \item Image Server client APIs
+\item Image Server maintainence tools
 \end{itemize}
 
@@ -445,6 +441,6 @@
 
 Clients interact with the IPP Image Server via a small number of C
-APIs (Bindings are also provided for Perl and Python).  The client
-commands are:
+APIs (Bindings are also provided for Perl and Python and UNIX shell
+commands in some cases).  The client commands are:
 
 \begin{itemize}
@@ -455,6 +451,6 @@
   node name on which the new storage object must be located.  If this
   target is not given, the Image Server places the new storage object
-  on an appropriate machine from the pool (least filled?  most data?
-  randomized?  the details need to be decided).
+  on an appropriate machine from the pool, though the details need to
+  be specified.
 
 \item {\tt open object}: open an instance of an existing storage
@@ -474,11 +470,11 @@
   specified storage object, including the number of instances of the object.
 
-\item {\tt increment object count}: adds a new instance of the given
-  storage object.  The target node may be optionally specified,
-  otherwise an appropriate node is selected. 
-
-\item {\tt decrement object count}: removes one of the instances of
-  the storage object.  The input parameters may optionally specify the
-  target machine to delete. 
+\item {\tt replicate object}:a new instance of the given storage
+  object.  The target node may be optionally specified, otherwise an
+  appropriate node is selected.
+
+\item {\tt cull object}: removes one of the instances of the storage
+  object.  The input parameters may optionally specify the target
+  machine to delete.
 
 \item {\tt delete object}: deletes all instances of the storage object
@@ -499,8 +495,8 @@
 about the data storage objects, their instances, and the available
 hardware resources.  A {\tt mysql} database engine is used to manage
-the database.  The database tables defined for the Image Server are
-listed in Table~\ref{ImageServerTables}, and their current contents
-are listed in Appendix A.  This database engine need not the same one
-as the one used for othe IPP subsystems.
+the database table.  The database tables defined for the Image Server
+are listed in Table~\ref{ImageServerTables}, and their contents are
+listed in Appendix A.  This database engine need not the same one as
+the one used for othe IPP subsystems.
 %
 \begin{table}
@@ -533,5 +529,5 @@
 The IPP Image Server provides a collection of administration tools
 which allow for maintainence.  These are operations which may be
-automatically scheduled for the IPP or which may be initiated by a
+automatically scheduled by the IPP or which may be initiated by a
 human via a command-shell interface.  The maintainence functions
 include migrating data between nodes to rebalance the available space
@@ -540,28 +536,28 @@
 for file corruption, which involves sweeping all files on a data
 volume and comparing the calculated file checksum to the currently
-recorded value.  
+recorded value.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
 \subsection{Metadata Database}
-
-The IPP Metadata Database acts as a repository for all non-pixel data
+\label{Metadata}
+
+The IPP Metadata Database acts as a repository for non-pixel data
 needed by the IPP subsystems.  This includes the image metadata, the
 environmental data, system configuration data and system reference
 data.  The Metadata Database is required to save the non-ephemeral
 data for the lifetime of the project for future reference and
-additional analysis.  The Metadata Database may potentially be used in
-close coupling with the analysis pipelines to store temporary data
-either within or between stages of the analysis.  In this scenario,
-the analysis pipeline will interact directly with the database.
-However, database latency may make this scenario impractical, in which
-case the database may be used for long-term storage only.  In this
-scenario, the data produced by analysis pipelines which is destined
-for the Metadata Database may be collected and inserted by a separate,
-dedicated process or analysis pipeline collection of processes.
-Metadata which is large in volume or poorly structure may also be
-stored in an appropriate container file (FITS Table, FITS Header, XML
-File) in the Image Server with the Metadata DB providing pointers to
-these files.
+additional analysis.  The Metadata Database may be used in close
+coupling with the analysis pipelines to store temporary data either
+within or between stages of the analysis.  In this scenario, the
+analysis pipeline will interact directly with the database.  However,
+database latency may make this scenario impractical, in which case the
+database may be used for long-term storage only.  In this scenario,
+the data produced by analysis pipelines which is destined for the
+Metadata Database may be collected and inserted by a separate,
+dedicated process.  Metadata which is large in volume or poorly
+structure may also be stored in an appropriate container file (FITS
+Table, FITS Header, XML File) in the Image Server with the Metadata DB
+providing pointers to these files.
 
 The IPP Metadata Database is a simple database system, consisting of a
@@ -569,26 +565,5 @@
 \code{mysql} database engine will be used to drive the database.
 
-\subsubsection{Metadata Tables}
-\label{Metadata}
-
-The complete contents of the Metadata Database will not be completely
-specified until the complete collection of data analysis scripts are
-available.  Even so, we can make a good first pass at the likely
-collection of long-term tables, and some of the temporary processing
-tables.  Table~\ref{MetadtaDBTables} lists the Metadata tables
-identified to date for the Metadata Database.  The contents of these
-tables are outlined in Appendix~\ref{MetadataContents}, with examples
-for the data entries and thier data types in many cases.
-
-\subsubsection{Metadata Queries}
-
-The IPP provides simple queries to the Metadata Database tables using
-autocoded APIs.  These queries allow for a single row or a simple
-collection of rows based on the primary key.  The format of the API is
-identical for all Metadata tables.  New tables and APIs can be added
-to the IPP system by adding to the autocoding table description
-files.  See Appendix~\ref{Autocode} for futher information.  
-
-\begin{table}
+\begin{table}[hb]
 \begin{center}
 \caption{Metadata Database Tables\label{MetadataDBTables}}
@@ -619,4 +594,24 @@
 \end{table}
 
+\subsubsection{Metadata Tables}
+
+The contents of the Metadata Database will not be completely specified
+until the complete collection of data analysis scripts are available.
+Even so, we can identify the likely collection of long-term tables,
+and some of the temporary processing tables.
+Table~\ref{MetadtaDBTables} lists the Metadata tables identified to
+date for the Metadata Database.  The contents of these tables are
+outlined in Appendix~\ref{MetadataContents}, with examples for the
+data entries and thier data types in many cases.
+
+\subsubsection{Metadata Queries}
+
+The IPP provides simple queries to the Metadata Database tables using
+autocoded APIs.  These queries return a single row or a collection of
+rows based on the primary key.  The format of the API is identical for
+all Metadata tables.  New tables and APIs can be added to the IPP
+system by adding to the autocoding table description files.  See
+Appendix~\ref{Autocode} for futher information.
+
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
@@ -636,5 +631,5 @@
 those supplied by external references.  These may be treated as {\em
 detections}, with the caveat that access to the raw measurements and
-metadata are usually unavailable; the reported measurements and errors
+metadata are usually unavailable: the reported measurements and errors
 must be accepted as they are reported.
 
@@ -645,5 +640,5 @@
 The AP Database also makes it possible to extract all detections
 derived from a specific image and to determine quantities such as the
-coordinates of the detection in pixel coordinates on the image.
+pixel coordinates of the detection on the image.
 
 The AP Database also has the capability to associate multiple
@@ -710,4 +705,12 @@
 filters.
 
+\begin{figure}
+\begin{center}
+\resizebox{4.5in}{!}{\includegraphics{pics/APDB}}
+\caption{AP DB components}
+\label{fig:APDBRegions}
+\end{center}
+\end{figure}
+
 The AP Database provides interfaces to extract lists of objects and
 detections based on various query parameters.  It provides the
@@ -749,12 +752,39 @@
 
 The specific subtable of {\tt Images} which contains a given image is
-that one which contains the center pixel \tbr{or 0,0 pixel} of that
+the one which contains the center pixel \tbr{or 0,0 pixel} of that
 image.  An additional table group, {\tt Image Overlaps} (with the same
 subtable organization as the {\tt Images} subtables), lists images
 which overlap that specific subtable.  Thus, given a particular
 coordinate, in order to find that images which overlap that
-coordinate, it is necessary to load the images in the {\tt Images}
+coordinate, it is necessary to search the images in the {\tt Images}
 subtable which includes that coordinate, and all images in the {\tt
-ImageOverlaps} table for that coordinate.
+ImageOverlaps} subtable for that coordinate.
+
+\begin{table}[hb]
+\begin{center}
+\caption{AP Database Tables\label{APDBTables}}
+\begin{tabular}{ll}
+\hline
+\hline
+{\bf Table Name} & {\bf Description} \\
+\hline
+Images              & The images that have objects in the DB. \\
+Image Overlaps      & Image regions which are touched by specific images. \\
+Objects             & The objects --- average properties of multiple detections of the same object. \\
+Average Magnitudes  & Average photometry in multiple filters \\
+Matched Detections  & Detections of sources in an image identified with an Object. \\
+Orphaned Detections & Detections of sources in an image not identified with an Object. \\
+Non-detections      & Non-detections of objects in an image. \\
+Region Table        & spatial distribution of tables \\
+Filters             & Filters understood by the system. \\
+Photcodes           & Transformations between different photometric systems \\
+Database Machines   & computers used to store the tables \\
+% Zero Points       & Transformations between different photometric systems \\
+% Distortion Models & Transformations between different photometric systems \\
+% Solar System Objects & Identification of solar system objects \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
 
 The {\tt Objects} table group (also divided by region) stores the
@@ -776,16 +806,16 @@
 detections associated with the average {\tt Objects}.  As discussed
 below, bright objects (above a configuration-specified signal-to-noise
-level) are assigned an object even if only one detection has been
-found at that position.  Faint orphaned objects are not added to this
-list or the list of objects.  The different types of detections (P2,
+level) are defined object even if only one detection has been found at
+that position.  Faint orphaned objects are not added to this list or
+the list of objects.  The different types of detections (P2,
 P4$\Delta$, P4$\Sigma$) are distinguished by their photometry codes.
-\tbr{This is only valid if the AP Database does not store different
-quantities for these types of detections}
+(This is only valid if the AP Database does not store different
+quantities for these types of detections.)
 
 The {\tt Orphaned Detections} table stores the detections which have
 not been correlated with an existing object.  This table is only
 populated for objects below a configuration-specified signal-to-noise
-limit.  Otherwise, even orphaned detections are assigned an object and
-added to the {\tt Matched Detections} table.
+limit (eg 5$\sigma$).  Bright orphaned detections are assigned an
+object and added to the {\tt Matched Detections} table.
 
 The {\tt Non-detections} table stores information about detection
@@ -816,40 +846,6 @@
 to serve the database tables.  The region file specifies the machine
 which stores the specific table.  Figure~\ref{ABDBRegions} illustrates
-schamatically the subdivision of the sky and the association between
+schematically the subdivision of the sky and the association between
 different levels of the hierarchy with different subtables.
-
-The {\tt Filters} table identifies all of the physical filters
-(specific, named pieces of glass) known to the system.  A related
-table, {\tt Photcodes}, defines relationships between specific
-photometry systems.  A system may consist of a detector, telescope,
-and specific filter, or it may be a derived photometry system.  The
-{\tt Database Machines} table identifies all of the computers
-available to the AP Database.
-
-\begin{table}
-\begin{center}
-\caption{AP Database Tables\label{APDBTables}}
-\begin{tabular}{ll}
-\hline
-\hline
-{\bf Table Name} & {\bf Description} \\
-\hline
-Images              & The images that have objects in the DB. \\
-Image Overlaps      & Image regions which are touched by specific images. \\
-Objects             & The objects --- average properties of multiple detections of the same object. \\
-Average Magnitudes  & Average photometry in multiple filters \\
-Matched Detections  & Detections of sources in an image identified with an Object. \\
-Orphaned Detections & Detections of sources in an image not identified with an Object. \\
-Non-detections      & Non-detections of objects in an image. \\
-Region Table        & spatial distribution of tables \\
-Filters             & Filters understood by the system. \\
-Photcodes           & Transformations between different photometric systems \\
-Database Machines   & computers used to store the tables \\
-% Zero Points       & Transformations between different photometric systems \\
-% Distortion Models & Transformations between different photometric systems \\
-\hline
-\end{tabular}
-\end{center}
-\end{table}
 
 \begin{figure}
@@ -861,11 +857,11 @@
 \end{figure}
 
-\begin{figure}
-\begin{center}
-\resizebox{4.5in}{!}{\includegraphics{pics/APDB}}
-\caption{AP DB components}
-\label{fig:APDBRegions}
-\end{center}
-\end{figure}
+The {\tt Filters} table identifies all of the physical filters
+(specific pieces of glass) known to the system.  A related table, {\tt
+Photcodes}, defines relationships between photometry systems.  A
+photometry system may consist of a detector, telescope, and specific
+filter, or it may be a derived photometry system.  The {\tt Database
+Machines} table identifies all of the computers available to the AP
+Database.
 
 \subsubsection{AP Database servers}
@@ -902,5 +898,5 @@
 The backend database engine for the AP Database stores the tables and
 provides them to the servers on demand.  The AP Database will use a
-\code{mysql} database engine for the function.
+\code{mysql} database engine for this function.
 
 \subsubsection{AP DB Client operations}
@@ -908,6 +904,5 @@
 The AP Database client interactions consist of a collection of basic
 queries of the database, along with more complex operations to perform
-particular tasks.  \tbd{queries are not yet listed; provide list from
-  DVO}.  The complex operations are listed below.
+particular tasks.  The complex operations are listed below.
 
 \paragraph{Insert Image \& Detection Set (addstar)}
@@ -958,5 +953,5 @@
 
 This operation uses the reference and image detections to determine an
-optical distortion model for the camera.  ñ
+optical distortion model for the camera.
 
 \begin{table}
@@ -984,4 +979,6 @@
 \subsubsection{Notes}
 
+discuss AP DB throughput issues
+
 how does the AP Database know about the relationship between a
 collection of chips?  
@@ -1021,33 +1018,36 @@
 Controller receives a variety of inputs from other subsystems,
 described below, and initiates actions such as adding a new process to
-its queue.  The IPP Controller also provides information to other
-subsystems on demand about its processing history and current state.
-Each physical computer may have multiple processors; since the IPP
-Controller is managing processing tasks, it treats each processor
-independently.  It is up to the system configuration if each computer
-needs to reserve one of its CPUs to manage background tasks or if the
-IPP Controller should attempt to send one task per CPU and let the
-kernel handle the I/O load.
-
-\subsubsection{Controller Nodes}
-
-Computers managed by the IPP Controller are allowed to be in one of
-several states, and the IPP Controller must interact with it in an
-appropriate way for each of those states.  A computer may be {\tt
-alive}, {\tt dead} or {\tt off}.  If the computer is {\tt alive}, it
-responds to commands from the IPP Controller and may be used for tasks
-subject to other constraints.  If it is {\tt dead}, the computer is
-not responsive and must not be used for executing tasks.  The IPP
-Controller must identify computers which have died and occasionally
+the queue of pending tasks.  The IPP Controller also provides
+information to other subsystems on demand about its processing history
+and current state.  Each physical computer may have multiple
+processors; since the IPP Controller is managing processing tasks, it
+treats each processor independently.  It is up to the system
+configuration if each computer needs to reserve one of its CPUs to
+manage background tasks or if the IPP Controller should attempt to
+send one task per CPU and let the operating system handle the I/O
+load.
+
+\subsubsection{Nodes}
+
+The Controller maintains a table of processing computers (`Nodes')
+available to it and tracks the status of these Nodes.  Nodes managed
+by the IPP Controller are allowed to be in one of several states, and
+the IPP Controller must interact with it in an appropriate way for
+each of those states.  A computer may be {\tt alive}, {\tt dead} or
+{\tt off}.  If the computer is {\tt alive}, it responds to commands
+from the IPP Controller and may be used for tasks subject to other
+constraints.  If it is {\tt dead}, the computer is not responsive and
+must not be used for executing tasks.  The IPP Controller must
+identify computers which have died (not responding) and occasionally
 test them to see if they are {\tt alive} again.  Computers which are
-{\tt off} are not available for tests and must not be tested.
+{\tt off} are not available for tasks and must not be tested.
 Computers may be set to the {\tt off} or {\tt dead} states by external
 subsystems; it is the responsibility of the IPP Controller to return a
-computer to the {\tt alive} state if possible.  
+computer to the {\tt alive} state if possible.
 
 The IPP Controller must honor requests (normally from the users) to
 change the mode of any computing node on demand between {\tt off} and
 {\tt dead}.  This would normally be done after a computer has been
-rebooted and is release to the IPP Controller for its use.  It must
+rebooted and is released to the IPP Controller for its use.  It must
 also be able to change the list of allowed tasks as requested by
 external commands.
@@ -1061,55 +1061,46 @@
 {\tt dead} for a very long time.  In another scenario, a person needs
 to work on a computer.  They notify the IPP Controller that the
-machine is off, perhaps with a prior notification that the machine
-should be prepared to go off.  When work on the machine is complete,
-it should be placed in the {\tt dead} state.  Only when the person is
-done working and testing the machine, and tells the IPP Controller
-that the machine is now {\tt dead} can the IPP Controller attempt to
-re-start communications and processing on that computer.
-
-CPUs on computers which are in the {\tt alive} state may be in one of
-two modes: {\tt busy} and {\tt free}.  A CPU which is {\tt busy}
-currently has a task assigned to it.  The IPP Controller may only
-assign one task to one CPU at a time.  A CPU which is in the {\tt
-free} state may have tasks assigned to it.  The IPP Controller must
-also respect a list of task restrictions which may require specific
-tasks to run on specific CPUs or exclude specific tasks from specific
-CPUs.
-
-The Controller maintains a table of processing nodes available to it
-and the status of these Nodes.  When the Controller starts, it
-attempts to launch a Node Agent on each of the available processing
-nodes.  Modes which are not responsive are placed into an inactive
-state and retried occasionally.
-
-\subsubsection{Controller Node Agents}
-
-A Node Agent runs on each of the individual nodes to perform the tasks
-as directed by the Controller.  The Node Agents communicate with the
-Controller via a socket connection.
-
-A processing stage is executed in the UNIX user space, and is run as a
-fork by the Node Agent.  The Node Agent must monitor the standard
-error and standard output of the processing stage and save them in
-separate buffers.  If the process dies, the Node Agent must detect the
-crash.  The Node Agent must respond to various commands from the
-Controller, as follows:
+machine is {\tt off}, perhaps with a prior notification that the
+machine should be prepared to go off.  When work on the machine is
+complete, it should be placed in the {\tt dead} state.  Only when the
+person is done working and testing the machine, and tells the IPP
+Controller that the machine is now {\tt dead} can the IPP Controller
+attempt to re-start communications and processing on that computer.
+
+\subsubsection{Node Agents}
+
+When the Controller starts, it attempts to launch a Node Agent on each
+of the available processing Nodes.  Modes which are not responsive are
+placed marked as {\tt dead} so they may be retried.  A Node Agent runs
+on each of the individual nodes to execute the tasks as directed by
+the Controller.  The Node Agents communicate with the Controller via a
+socket connection.
+
+A Node Agent (which is only on Node in the {\tt alive} state) may be
+in one of four modes: {\tt idle}, {\tt busy}, {\tt done}, {\tt crash}.
+A Node Agent which is {\tt busy} currently has a task assigned to it
+which is executing.  The IPP Controller may only assign one task to a
+Node at a time.  A Node Agent which is in the {\tt idle} state may
+have a task assigned to it.  When the Node Agent detects that a tasks
+has finished, it changes to either the {\tt done} or {\tt crash}
+states depending on the outcome of the process execution.  The IPP
+Controller must also respect a list of task restrictions which may
+require specific tasks to run on specific CPUs or exclude specific
+tasks from specific CPUs.
+
+A task being executed by the Node is run in the UNIX user space as a
+forked process.  The Node Agent must monitor the standard error and
+standard output of the executing task and save them in separate
+buffers.  If the process exits or dies, the Node Agent must detect
+this result and change state appropriately.  The Node Agent must
+respond to various commands from the Controller, as follows:
 
 \paragraph{Report status}
 
-The Node Agent returns the state of the Node (idle, busy, done), the
-state of the current processing stage (`none', `busy', `crash',
-`done'), and the exit status of the current processing stage, if
-available.
-
-The four possible states of the Node indicate that the client has no
-current processing stage (`idle'), that it has a processing stage
-which is still running (`busy'), or that it has a processing stage
-which has completed.  The last two states indicate if the current
-processing stage has crashed (`crash'), or if the current processing
-stage has exited gracefully (`done').  The reported exit state, if the
-process has completed without crashing, is the UNIX exit state
-reported by the processing stage: 0--256 with 0 indicating a
-successful completion.
+The Node Agent returns its state ({\tt idle}, {\tt busy}, {\tt done},
+{\tt crash'}) and the exit status of the current processing task, if
+available.  The reported exit state, if the process has completed
+without crashing, is the UNIX exit state reported by the task: 0--256
+with 0 indicating a successful completion.
 
 \paragraph{Report stdout}
@@ -1118,5 +1109,5 @@
 the complete contents of the stdout buffer via a buffered write and
 flush the buffer when it is finished.  The Node Agent will not accept
-more data on the stdout buffer from the current processing stage until
+more data on the stdout buffer from the current processing task until
 the send is complete and the buffer is flushed.  The daemon must
 accept all of the buffer output.
@@ -1126,17 +1117,15 @@
 Identical to `report stdout', but for stderr.
 
-\paragraph{Kill processing stage}
-
-The Node Agent should send a kill signal to the current processing
-stage.  When the processing stage has exited, the Node Agent should
-set the processing stage status to `crash' and the Node status to
-`done'.
-
-\paragraph{Clear processing stage}
-
-The Node Agent should set the current processing stage state to `none'
-and the Node state to `idle'.  If a processing stage is currently
-running, it should be killed (signal 9 or 15) before the processing
-stage is cleared.
+\paragraph{Kill task }
+
+The Node Agent should send a kill signal (signal 9 or 15) to the
+current processing task.  When the processing task has exited, the
+Node Agent should set its state to {\tt crash}.
+
+\paragraph{Clear task}
+
+The Node Agent should set its state {\tt idle}.  If a processing stage
+is currently running, it should be killed (signal 9 or 15) before the
+task is cleared.
 
 \paragraph{Start processing stage}
@@ -1144,6 +1133,6 @@
 The Node Agent forks a specified command.  The command should be a
 standard UNIX command without command line redirection or
-backgrounding.  For this reason, the Node Agent must provide a layer
-of security, for example, by employing SSL authentication.
+backgrounding.  The task is run with the same user ID as the Node
+Agent, which is also the same user ID as the Controller.
 
 \subsubsection{Tasks}
@@ -1152,11 +1141,13 @@
 requests include the specific command to be executed and are in the
 form of a UNIX command which could be performed on any of the
-computing nodes.  Any input or output data structures in the commands
-must be a valid resource regardless of the node on which the task is
-executed.  Input and output data resources must be unique where
-necessary to avoid conflicts.  The IPP Controller gives each task a
-unique identifier, which is returned to the requesting entity.  The
-requestor may then use that ID to obtain status information on that
-task or to send control signals to the specific task.
+computing nodes.  Any input or output data in the commands must be a
+valid resource regardless of the node on which the task is executed.
+Input and output data resources must be unique where necessary to
+avoid conflicts.  \tbd{It is the responsibility of the programs to
+wait for network lags (ie, NFS delays)}.  The IPP Controller gives
+each task a unique identifier, which is returned to the requesting
+entity.  The requestor may then use that ID to obtain status
+information on that task or to send control signals to the specific
+task.
 
 Task requests may specify a desired node for the task execution.  The
@@ -1245,12 +1236,5 @@
 \subsection{Scheduler}
 
-\begin{figure}
-\begin{center}
-\resizebox{6in}{!}{\includegraphics{pics/Scheduler}}
-\caption{ \label{Scheduler} IPP Scheduler}
-\end{center}
-\end{figure}
-
-The IPP is responsible for a variety of analysis tasks: processing of
+The IPP is responsible for a variety of analysis jobs: processing of
 the science images through several stages; routine assessment of the
 detrend (instrumental calibration) images used in processing the
@@ -1270,4 +1254,6 @@
 Scheduler may be viewed as the central brain of the IPP.
 Figure~\ref{Scheduler} illustrates the design of the IPP Scheduler.
+
+\subsubsection{Scheduler Tasks and Tests}
 
 The IPP Scheduler performs two types of actions.  'Tasks' are
@@ -1289,19 +1275,22 @@
 Database or other subsystems.  Based on the successful completion (or
 not!) of the tasks, and the new state of entries in the Metadata
-Database, the Scheduler can define new tasks. 
-
-The IPP Scheduler sends commands to the IPP Controller for execution.
+Database, the Scheduler can define new tasks.
+
+\begin{figure}
+\begin{center}
+\resizebox{6in}{!}{\includegraphics{pics/Scheduler}}
+\caption{ \label{Scheduler} IPP Scheduler}
+\end{center}
+\end{figure}
+
+The IPP Scheduler sends tasks to the IPP Controller for execution.
 While the IPP Scheduler chooses the tasks to be performed, it is the
 IPP Controller's responsibility to manage the specific tasks executing
-on a given processing node.  Examples of these tasks include the
-process of copying or moving data from the Summit data systems to the
-IPP Image Server; image processing analysis stages performed on the
-science images by the appropriate processing nodes; and the analysis
-of the data in the AP Database.  This division of responsibilites
-allows us to isolate and encapsulate the functionality of the IPP
-Scheduler and the IPP Controller.  With this separation, the IPP
-Controller does not need to have any information about the details of
-the tasks which it executes, while the IPP Scheduler does not need to
-monitor the computer hardware.
+on a given processing node.  This division of responsibilites allows
+us to isolate and encapsulate the functionality of the IPP Scheduler
+and the IPP Controller.  With this separation, the IPP Controller does
+not need to have any information about the details of the tasks which
+it executes, while the IPP Scheduler does not need to monitor the
+computer hardware.
 
 Communication between the IPP Scheduler and the IPP Controller is
@@ -1312,13 +1301,13 @@
 but distinct software components.
 
-The IPP Scheduler takes as input the current list of pending images,
-both science and calibration, and a description of the current
-observing plan or strategy on some time-scale.  The IPP Scheduler also
-takes input from humans who manage the IPP.
-
-The IPP Scheduler must choose between several types of analysis tasks
-based on the contents of those lists and on the requirements of the
-users.  The list of tasks which the IPP Scheduler must decide between
-includes:
+\subsubsection{Task Rules}
+
+The IPP Scheduler takes as input a collection of rules which define
+the dependency of tasks on certain tests.  The IPP Scheduler must
+choose between several types of analysis tasks based on those ruls and
+on results of the tests.  The timescale on which different tasks (and
+their related tests) are executed may vary from 10s of seconds to
+hours, days, or even week.  The list of tasks which the IPP Scheduler
+must decide between, and the relevant timescale, follow:
 \begin{itemize}
 \item moving data from the Summit pixel server ($\sim 30$ second timescales)
@@ -1327,24 +1316,18 @@
   nightly)
 \item constructing new detrend images ($\sim$ weekly)
-\item updating and improving the photometric and astrometric reference
-  catalogs ($\sim$ yearly).
 \end{itemize}
-
-The IPP Scheduler chooses between tasks which are relevant on several
-different time-scales.  The time-scales range from 2 times per minute
-to once or twice a year, as noted in the list above.  The IPP
-Scheduler must also make use of user input in managing such choices.
-Users have the option to specify that a particular task or set of
-tasks is of higher or lower priority than the norm.
-
 The scheduler may be viewed as a complex state machine.  Our goal is
-to design the rules independently from the engine which parses the
-rules to detemine which specific jobs to send to the controller.
-
-\subsubsection{Scheduler User Interface}
+to design the scheduler so that rules may be specified independently
+from the engine which parses the rules to detemine which specific jobs
+to send to the controller.
+
+\subsubsection{User Interface}
 
 The IPP Scheduler provides a user interface which allows a human
 operator, or other processes, to monitor the current state of the
-Scheduler.  
+Scheduler.  Users have the option to specify that a particular task or
+set of tasks is of higher or lower priority than the norm, or to
+schedule a particular tasks on a different timescale from the basic
+rule.
 
 The IPP Scheduler defines the operating state of the IPP.  When the
@@ -1360,5 +1343,5 @@
 for tests or maintenance, in which case the IPP Scheduler does not
 perform even the data copy tasks.  Every task is performed on demand
-by the user.  The user command sets the IPP Scheduler in one of these
+by the user.  A user command sets the IPP Scheduler in one of these
 three states, {\em automatic}, {\em interactive}, and {\em paused}.
 
@@ -1411,7 +1394,7 @@
 installation running on the Pan-STARRS cluster.  The {\tt base}
 configuration defines general data sources which may be needed by any
-portion of the IPP.  The list of known telescope or filters might be
+portion of the IPP.  The list of known telescopes or filters might be
 an example.  The {\tt camera} configuration consists of information
-which defines the parameters relevant to the camera known by the IPP.
+which defines the parameters relevant to the cameras known by the IPP.
 For example, the default layout of the detectors or the names of
 specific header keyword values would be defined for each camera in a
@@ -1478,36 +1461,8 @@
 
 \begin{verbatim}
-possible command forms:
-
-P1 filename.fits [FPA is single fits file]
-P1 filename.list [FPA is collection of files]
-P1 FPA IA        [FPA info from metadata db]
-
-sources for the input data:
-
-distortion model:
-  metadata table
-  XML file 
-  FITS table
-  metadata -> image server
-  user provided on command line
-  recipe provided
-
-camera layout:
-  metadata table
-  XML file 
-  FITS table
-  metadata -> image server
-  user provided on command line
-  recipe provided
-
-boresite coordinates guess:
-  image header (keywords from recipe)
-  metadata table
-
-guide stars
-  collection of video streams
-  collection of centroid time histories
-  list of centroids, coordinates
+Phase1 -file filename.fits [FPA is single fits file]
+Phase1 -list filename.list [FPA is collection of files]
+Phase1 -imdb ID            [FPA is single file in image server]
+Phase1 -FPA  ID            [FPA identifier from metadata db]
 \end{verbatim}
 
@@ -1642,12 +1597,12 @@
 with the choice a user-configurable option.
 
-The input science and mask frames are trimmed by the extent of the OT
-convolution kernel in each direction ($+x$, $-x$, $+y$, $-y$).  Within
-the PSLib image handling functions, the trim function is a virtual
-operation which simply marks the boundaries of the trimmed image but
-does not remove the corresponding memory.  This allows the later
-corrections to work with untrimmed correction images and still apply
-the correct pixels.  At the end of Phase 2, the only the trimmed
-portions of the output images are written out to disk.
+The input science and mask frames are additionally trimmed by the
+extent of the OT convolution kernel in each direction ($+x$, $-x$,
+$+y$, $-y$).  Within the PSLib image handling functions, the trim
+function is a virtual operation which simply marks the boundaries of
+the trimmed image but does not remove the corresponding memory.  This
+allows the later corrections to work with untrimmed correction images
+and still apply the correct pixels.  At the end of Phase 2, the only
+the trimmed portions of the output images are written out to disk.
 
 \subsubsection{Non-Linearity Correction}
@@ -1661,5 +1616,5 @@
 \subsubsection{Flat field Correction}
 
-The object image (after bias correction and non-linearity correction)
+The science image (after bias correction and non-linearity correction)
 must be corrected for sensitivity variations as a function of
 position, dividing by a flat-field image.  The mask is also modified
@@ -1760,10 +1715,10 @@
 are sent to the IPP Pixel Server.
 
-\begin{figure}
-\begin{center}
-\resizebox{6in}{!}{\includegraphics{pics/phase2}}
-\caption{ \label{phase2} Phase 2 dataflow}
-\end{center}
-\end{figure}
+%\begin{figure}
+%\begin{center}
+%\resizebox{6in}{!}{\includegraphics{pics/phase2}}
+%\caption{ \label{phase2} Phase 2 dataflow - this diagram is old: update}
+%\end{center}
+%\end{figure}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -1839,11 +1794,4 @@
 Phase 2.
 
-\begin{figure}
-\begin{center}
-\resizebox{4.5in}{!}{\includegraphics{pics/phase3}}
-\caption{ \label{phase3} Phase 3 dataflow}
-\end{center}
-\end{figure}
-
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
@@ -1860,8 +1808,8 @@
 The working concept is that the static sky cells contain roughly the
 same number of pixels as an OTA (4k x 4k) and represent a portion of a
-local tangent plane projection.  As mentioned above
-(Section~\ref{IPP:ImageServer}), the pixel scale of the static sky is
-planned to be 0.2\arcsec, somewhat smaller than the 0.3\arcsec\ raw
-image pixel scale.
+local tangent plane projection.  In order to meet the image
+degredation requirements, the pixel scale of the static sky is planned
+to be 0.2\arcsec, somewhat smaller than the 0.3\arcsec\ raw image
+pixel scale.
 
 For each sky cell, the corresponding pixels are extracted from the
@@ -1896,6 +1844,14 @@
 \subsubsection{Static Sky Subtraction}
 
-\tbd{add some details about the static-sky subtraction issues.
-  Allard-Lupton-Price method}.
+The corresponding static sky image is subtracted from the combined
+image stack.  In this step, it is necessary to match the image kernel
+between the input image and the static sky image.  This will be done
+by solving for a best-fit image kernel which minimizes the difference
+image using a technique equivalent to the Allard-Lupton method.  The
+modification we make is that, rather than represent the components of
+the image difference kernel as a combination of Gaussians, we will
+represent the kernel as a combination of pixels.  This method also
+automatically determines a photometric match between the static sky
+image and the input science image.
 
 \subsubsection{Object Detection and Measurement}
@@ -1944,9 +1900,7 @@
 adding these objects to the database, the transients which are
 correlated with previous detections of an object (and those which are
-not) will automatically be determined.  An independent process will
-query the AP Database for such transient objects of interest which are
-to be sent, along with their associated metadata, to the MOPS and
-other science client pipelines.  This step must be performed at least
-once per night.
+not) will automatically be determined.  A subset of these transient
+objects are sent, along with their associated metadata, to the MOPS
+and other preferred science client pipelines.  
 
 \subsubsection{Static Sky Update}
@@ -1963,10 +1917,10 @@
 a time when the computing infrastructure is not under significant load.
 
-\begin{figure}
-\begin{center}
-\resizebox{6in}{!}{\includegraphics{pics/phase4}}
-\caption{ \label{phase4} Phase 4 dataflow}
-\end{center}
-\end{figure}
+%\begin{figure}
+%\begin{center}
+%\resizebox{6in}{!}{\includegraphics{pics/phase4}}
+%\caption{ \label{phase4} Phase 4 dataflow}
+%\end{center}
+%\end{figure}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -2155,12 +2109,13 @@
 \section{System Design : Miscellaneous Tasks}
 
-In this section, we discuss the design of the science analysis stages
-which perform the fundamental image analysis steps of the IPP.
+In this section, we discuss additional operations which are performed
+by the IPP but which do not fall under the analysis of the science
+images or the creation of the calibration images.  
 
 \subsection{Retrieval}
 
-The retrieval stages simply retrieve pixel data from an external
-source (ordinarily OATS at the Summit, but it could conceivably be
-some other external source) and store it on the nodes.
+The retrieval stage simply retrieves images from an external source
+(ordinarily OTIS at the Summit, but it could conceivably be some other
+external source) and store it in the Image Server.  
 
 \subsection{Static Sky Analysis}
@@ -2225,7 +2180,7 @@
 program in C or through the use of a high-level language such as Perl,
 Python, or Tcl employing the SWIG interfaces.  For the high-level
-functions in the operational system, the IPP will make use of
-\tbd{Python} as the scripting language to provide the required
-flow-control to tie the modules together.
+functions in the operational system, the IPP will make use of Perl as
+the scripting language to provide the required flow-control to tie the
+modules together.
 
 This approach satisfies the requirement that complicated low-level
@@ -2269,5 +2224,5 @@
 therefore represents the maximum amount of effort which can be
 performed in serial without interaction between parallel threads.  The
-stages will be written in \tbd{Python}, linking the modules together.
+stages will be written in Perl, linking the modules together.
 Examples of stages are Phase 2 (detrend images) and Phase 4 (combine
 images from multiple telescopes and search for transients).
@@ -2290,8 +2245,8 @@
 the Controller, and determines the next action based on the contents
 of the Metadata Database.  The various subsystems specify the API for
-client / server interactions with them.  Commands will be sent using
-either text-based commands, SOAP or an equivalent protocol.  The
-format of the exchanged data may be in one of several forms discussed
-below.
+client / server interactions, and are discussed in their individual
+section.  Commands will be sent using either text-based commands, SOAP
+or an equivalent protocol.  The format of the exchanged data may be in
+one of several forms discussed below.
 
 FITS Images will be used to transport images between the components of
@@ -2318,10 +2273,4 @@
 Pan-STARRS systems and the external clients.  The interfaces are
 illustrated in Figure~\ref{overview}.  
-
-Incoming data is received by
-either the IPS (pixels), the IMD (metadata), or the IOD (objects).
-Requests for data by external clients are also made to these three
-databases.  Requests for data made by the IPP are generated by the IPP
-Scheduler or the science processing pipelines.
 
 \subsubsection{OTIS}
@@ -2448,15 +2397,59 @@
 requirements given the above need to 63 processors.  
 
+There are two competing trades we will also want to make.  First, we
+will want to duplicate data to multiple machines in the network to
+protect against catastrophic failures on a single machine.  This
+double the total data space needed.  To compensate, however, we will
+also employ compression to data, especially data which is older.
+These two factors will tend to cancel each other, so we have ignored
+both in out calculations above. 
+
 \tbd{switch information}
 
-\tbd{RAID and compression / duplication plan}
-
 \subsection{PS-1 Cluster Expected Reliability}
-  
+
+With 80 computers and 1920 disks, we must be cautious about component
+failures and their impact on operations and data integrity.  There are
+several factors which mitigate our exposure to hardware failures.
+First, the use of RAID controllers and RAID-5 striping of the data
+will protect the data on a single RAID set against the failure of a
+single disk in the array.  Second, our plan to have duplication across
+the cluster will protect us against catastrophic failures.  Finally,
+the flexibility of the distributed computing plan makes it trivial to
+handle the loss of individual machines as the system can automatically
+redistribute the load across the cluster. 
+
+The components which are most likely to fail in our experience are, in
+order: hard drives, ram, power supplies, and other components.  The
+hard drive failure rate is by far the dominant concern as it
+potentially affects the data integrity.  
+
+Most sources (REFS: UCSD article, Samsung White Paper) currently imply
+hard disk failure rates (MTBF) in the range 400,000 hours and 500,000
+hours.  We take these as an upper limit, and instead adopt a
+conservative value of 100,000 hours.  With 1920 disk, this MTBF
+implies a failure of one disk every 2.2 days.  Since the disks are in
+a RAID which reports the disk failures immediately and drops the array
+into degraded mode, these failures will not have a huge impact on the
+operations, and recovery time is only 10s of minutes.  This failure
+rate implies that we should be checking for hard disk failures daily.
+\tbd{is it necessary to catch failures at night or can the system run
+with a degraded disk?}.  A catastrophic failure for the array would
+require two of the 12 disks to fail before the first failed disk is
+replaced.  If we assume that maintainence is poor and it is possible
+for a disk to take 1 week to be replaced, we calculate a probability
+of a catastrophe of 1.8\% each time a disk fails.  Combined with the
+disk failure rate, we can expect a RAID catastrophe 6 times over the 2
+year operation of PS-1.  We can use these numbers as a guideline for
+our level of support needed to avoid these RAID failures.  Note that
+these 6 failures should not cause loss of data since the data is
+duplicated across the cluster, but they require over 1 day for
+recovery (as the entire array must be replicated across the network).
+
 \subsection{PS-1 Cluster Support}
 
 \begin{figure}
 \begin{center}
-\resizebox{6in}{!}{\includegraphics{pics/ps1_ipp_storage.ps}}
+\resizebox{6in}{!}{\includegraphics[angle=-90]{pics/ps1_ipp_storage.ps}}
 \caption{ \label{StorageProfile} Storage Profile}
 \end{center}
@@ -2470,7 +2463,10 @@
 
 \subsection{Image Server Database Table Contents}
-\ref{ImageServerTableContents}
-
-\begin{table}
+\label{ImageServerTableContents}
+
+Tables~\ref{ImageServerTables:SO} - \ref{ImageServerTables:VOL} list
+the basic contents of the Image Server database tables.  
+
+\begin{table}[bh]
 \begin{center}
 \caption{Storage Object Table Contents\label{ImageServerTables:SO}}
@@ -2489,5 +2485,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Instance Table Contents\label{ImageServerTables:INT}}
@@ -2509,5 +2505,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Volume Table Contents\label{ImageServerTables:VOL}}
@@ -2526,10 +2522,10 @@
 
 \subsection{Metadata Database Table Contents}
-\ref{MetadataTableContents}
-
-Tables \tbd{NN} -- \tbd{NN} list the basic contents of each of the
-Metadata Database tables listed in Section~\ref{Metadata}.
-
-\begin{table}
+\label{MetadataTableContents}
+
+Tables~\ref{WeatherTable} -- \ref{overlaps} list the basic contents of
+each of the Metadata Database tables listed in Section~\ref{Metadata}.
+
+\begin{table}[bh]
 \begin{center}
 \caption{Weather Table: some sample weather points\label{WeatherTable}}
@@ -2550,5 +2546,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{SkyProbe Transparency Table (sample entries)\label{SkyprobeBVTable}}
@@ -2570,5 +2566,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Skyprobe Line Absorption Table (sample entries)\label{SkyprobeATable}}
@@ -2593,5 +2589,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Skyprobe Line Emission Table (sample entries)\label{SkyprobeETable}}
@@ -2614,5 +2610,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{DIMM Measurements Table\label{DimmTable}}
@@ -2635,5 +2631,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Near IR Wide-field Camera Results Table\label{NIR-Table}}
@@ -2654,5 +2650,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Dome Status Table\label{DomeStatusTable}}
@@ -2672,5 +2668,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Telescope Status\label{TelescopeStatusTable}}
@@ -2691,5 +2687,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Raw FPA Images\label{RawFPAs}}
@@ -2721,5 +2717,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Pending Science Chips\label{PendingChips}}
@@ -2737,5 +2733,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Processed Science Chips\label{ProcessedChips}}
@@ -2754,5 +2750,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Observation Group Information\label{OBS}}
@@ -2772,5 +2768,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Observation Frame Information\label{OBS}}
@@ -2790,5 +2786,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Science Processing Stats\label{PSStats}}
@@ -2828,5 +2824,5 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
 \caption{Chip / Sky overlaps\label{overlaps}}
@@ -2844,7 +2840,7 @@
 \end{table}
 
-\begin{table}
+\begin{table}[bh]
 \begin{center}
-\caption{Processed Sky-Cell stats\label{}}
+\caption{Processed Sky-Cell stats\label{ProcessedSky}}
 \begin{tabular}{lll}
 \hline
@@ -2858,6 +2854,6 @@
 Diff image params  & string 	   & Parameters used for the image differencing. \\
 Average weight     & string 	   & The weight of the reference image \\
-P4D object stats   & string 	   & Summary statistics of the object detection (number of objects, depth, other input parameters). \\
-P4S object stats   & string 	   & Summary statistics of the object detection (number of objects, depth, other input parameters). \\
+P4D object stats   & string 	   & Summary statistics of the object detection  \\
+P4S object stats   & string 	   & Summary statistics of the object detection  \\
 Software versions  & string 	   & Software versions of modules used in the sky cell processing. \\
 Processing stats   & string 	   & Summary statistics of the processing (CPU time, etc). \\
@@ -2869,6 +2865,7 @@
 
 \subsection{AP Database Table Contents}
-\ref{APDBTableContents}
-
+\label{APDBTableContents}
+
+\tbd{Table contents to be defined}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Index: /trunk/doc/design/ippSRS.tex
===================================================================
--- /trunk/doc/design/ippSRS.tex	(revision 2191)
+++ /trunk/doc/design/ippSRS.tex	(revision 2192)
@@ -1,3 +1,3 @@
- %%% $Id: ippSRS.tex,v 1.10 2004-10-21 03:55:59 eugene Exp $
+ %%% $Id: ippSRS.tex,v 1.11 2004-10-22 04:43:35 eugene Exp $
 \documentclass[panstarrs,spec]{panstarrs}
 
@@ -180,78 +180,84 @@
 
 \begin{enumerate}
-\item For images obtained in photometric weather, produce reduced
-  science images for each full camera exposure with photometric
-  zero-point scatter less than 1\% across the full
-  field. \VER{ANALYSIS}{SCD:3.2.2.5}
+\item For images obtained in photometric weather with normal detector
+  characteristics and providing appropriate flat-field images and
+  correction data have been obtained, the IPP shall produce reduced
+  science images for each full camera exposure with relative
+  photometric zero-point scatter less than 1\% ($1 \sigma$) across the
+  full field. \VER{ANALYSIS}{SCD:3.2.2.5}
   \label{TLR:1}
 
-\item For images obtained in photometric weather, produce reduced
-  science images for each full camera exposure which are
-  photometrically calibrated with respect to the Pan-STARRS filter
-  system with a 1$\sigma$ accuracy of 1\%.\VER{ANALYSIS}{SCD:3.2.2.5}
+\item For images of reference fields calibrated for the IPP filter set
+  and obtained in photometric weather with normal detector
+  characteristics and providing appropriate flat-field images and
+  correction data have been obtained, the IPP shall determine and
+  track zero-points for these exposures with a 1$\sigma$ accuracy of
+  1\%.\VER{ANALYSIS}{SCD:3.2.2.5}
   \label{TLR:2}
 
 \item For images obtained under normal seeing conditions and optical
-  distortion, produce reduced science images for each full camera
-  exposure with an astrometric calibration providing $< 30$
-  milliarcsecond scatter (1$\sigma$) for sequential images of the same
-  location.\VER{ANALYSIS}{SCD:3.2.2.7}
+  distortion, the IPP shall produce reduced science images for each
+  full camera exposure with an astrometric calibration providing $<
+  30$ milliarcsecond scatter (1$\sigma$) for sequential images of the
+  same location.\VER{ANALYSIS}{SCD:3.2.2.7}
   \label{TLR:4}
 
 \item For images obtained under normal seeing conditions and optical
-  distortion, produce reduced science images for each full camera
-  exposure with an astrometric calibration providing $< 100$
-  milliarcsecond scatter (1$\sigma$) relative to the ICRS reference
-  system.\VER{ANALYSIS}{SCD:3.2.2.6}
+  distortion, the IPP shall produce reduced science images for each
+  full camera exposure with an astrometric calibration providing $<
+  100$ milliarcsecond scatter (1$\sigma$) relative to the ICRS
+  reference system.\VER{ANALYSIS}{SCD:3.2.2.6}
   \label{TLR:3}
 
 \item In photometric weather and under moon conditions listed in
-  Table~\ref{moonconditions}, produce reduced science images for each
-  full camera exposure which have background variations of less than
-  1\% in regions free of large ($> 30$ pixels diameter) astronomical
-  structures.\VER{ANALYSIS}{SCD:3.5.12}
+  Table~\ref{moonconditions}, the IPP shall produce reduced science
+  images for each full camera exposure which have background
+  variations of less than 1\% in regions free of large ($> 30$ pixels
+  diameter) astronomical structures.\VER{ANALYSIS}{SCD:3.5.12}
   \label{TLR:5}
 
-\item In photometric weather, produce reduced science images for each
-  full camera exposure which have background deviations from the
-  static sky in the same filter of less than 1\% for the median in
-  large ($> 30$ pixels diameter) regions.\VER{ANALYSIS}{SCD:3.5.12}
+\item In photometric weather, the IPP shall produce reduced science
+  images for each full camera exposure which have background
+  deviations from the static sky in the same filter of less than 1\%
+  for the median in large ($> 30$ pixels diameter)
+  regions.\VER{ANALYSIS}{SCD:3.5.12}
   \label{TLR:5a}
 
-\item Merge all $g$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
+\item The IPP shall merge all $g$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
   \label{TLR:6}
 
-\item Merge all $r$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
+\item The IPP shall merge all $r$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
   \label{TLR:7}
 
-\item Merge all $i$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
+\item The IPP shall merge all $i$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
   \label{TLR:8}
 
-\item Merge all $z$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
+\item The IPP shall merge all $z$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
   \label{TLR:9}
 
-\item Merge all $y$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
+\item The IPP shall merge all $y$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
   \label{TLR:10}
 
-\item Merge all $w$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
+\item The IPP shall merge all $w$ filter science images into a static sky image.\VER{TASK}{SCD:3.2.2.10}
   \label{TLR:11}
 
-\item Detect and classify objects on the individual processed science
+\item The IPP shall detect and classify objects on the individual processed science
   images.\VER{TASK}{SCD:3.2.2.16}
   \label{TLR:12}
 
-\item Detect and classify objects on the stacked groups of science
-  images.\VER{TASK}{SCD:3.2.2.16}
+\item The IPP shall detect and classify objects on the stacked groups
+  of science images.\VER{TASK}{SCD:3.2.2.16}
   \label{TLR:13}
 
-\item Detect and classify objects on the static sky image.\VER{TASK}{SCD:3.2.2.16}
+\item The IPP shall detect and classify objects on the static sky
+  image.\VER{TASK}{SCD:3.2.2.16}
   \label{TLR:14}
 
-\item Detect transients with significance $>3\sigma$ in the individual
-  science images relative to the static sky
+\item The IPP shall detect transients with significance $>3\sigma$ in
+  the individual science images relative to the static sky
   image.\VER{ANALYSIS}{SCD:3.2.2.16}
   \label{TLR:15}
 
-\item Degrade the stacked image by no more than \tbr{10
+\item The IPP shall degrade the stacked image by no more than \tbr{10
   milliarcseconds (FWHM added in quadrature)} over the theoretical
   limit for the stack under infinite
@@ -259,50 +265,52 @@
   \label{TLR:16}
 
-\item Perform the processing of science images to the level of
-  transient detection and static sky inclusion at a rate such that
-  exposures taken at an \tbr{average cadence of 40 seconds} do not
-  accumulate in the processing buffer (average throughput
+\item The IPP shall perform the processing of science images to the
+  level of transient detection and static sky inclusion at a rate such
+  that exposures taken at an \tbr{average cadence of 40 seconds} do
+  not accumulate in the processing buffer (average throughput
   requirement).\VER{TEST}{SCD:3.2.2.3}
   \label{TLR:17}
 
-\item Limit the \tbr{false alarm rate (FAR) to less than 5\%} for
-  transient detections $> 5\sigma$ sent to the preferred client
+\item The IPP shall limit the false alarm rate (FAR) to less than 5\%
+  for transient detections $> 5\sigma$ sent to the preferred client
   science pipelines.\footnote{note difference with PS-4: 1\%}
   \VER{ANALYSIS}{SCD:3.2.2.13}
  \label{TLR:18}
 
-\item Perform \tbr{transient detection to a completeness of 99\%} at
-  the completeness for transient detections with a significant $>
-  5\sigma$.\VER{ANALYSIS}{SCD:xxx}
-
-\item Publish the static sky images to the Pan-STARRS Published
-  Science Products Subsystem (PSPS) once per \tbr{6
-  months}.\VER{TASK}{SCD:3.2.2.18}
+\item The IPP shall perform transient detection to a completeness of
+  99\% at the completeness for transient detections with a significant
+  $> 5\sigma$.\VER{ANALYSIS}{SCD:xxx}
+
+\item The IPP shall publish the static sky images to the Pan-STARRS
+  Published Science Products Subsystem (PSPS) at a rate so the full
+  sky is transmitted once per year.\VER{TASK}{SCD:3.2.2.18}
   \label{TLR:19}
 
-\item Publish the detected objects to the Pan-STARRS Published Science
-  Products Subsystem (PSPS) once per month.\VER{TASK}{SCD:3.2.2.18}
+\item The IPP shall publish the detected objects to the Pan-STARRS
+  Published Science Products Subsystem (PSPS) at a rate such that the
+  objects from the full sky are transmitted once per
+  year.\VER{TASK}{SCD:3.2.2.18}
   \label{TLR:20}
 
-\item Send the IPP metadata and received OTIS metadata to the
-  Pan-STARRS Published Science Products Subsystem (PSPS)
+\item The IPP shall send the IPP metadata and received OTIS metadata
+  to the Pan-STARRS Published Science Products Subsystem (PSPS)
   weekly.\VER{TASK}{SCD:3.2.2.18}
   \label{TLR:21}
 
-\item Provide access to preferred Pan-STARRS science clients to the
+\item The IPP shall provide access to preferred Pan-STARRS science clients to the
   detected transient objects within \tbr{5 minutes}.\VER{TEST}{SCD:3.5.10}
   \label{TLR:22}
 
-\item Provide sufficent storage volume for raw images from the AP and
+\item The IPP shall provide sufficent storage volume for raw images from the AP and
   IVP Surveys and the \grizy\ Static Sky.\footnote{note difference with
   PS-4: 1 month of raw images} \VER{INSPECT}{allocated}
   \label{TLR:23}
 
-\item Provide sufficient storage volume for all detections from the
+\item The IPP shall provide sufficient storage volume for all detections from the
   AP, IVP, and MVP Surveys.\footnote{note difference with PS-4: 1 year
   of detections}\VER{INSPECT}{allocated}
   \label{TLR:24}
 
-\item Provide sufficient storage volume for 2 years of
+\item The IPP shall provide sufficient storage volume for 2 years of
  metadata.\footnote{note difference with PS-4: 10 years of
  metadata}\VER{INSPECT}{allocated}
@@ -787,8 +795,9 @@
 \item The AP Database shall accept new detections at the rate
   generated by the telescope from the Phase 2 and Phase 4 analysis.
-  \tbr{Except within 10 degrees of the galactic plane, the AP Database
-  shall keep up with the incoming rates.}  The expected rates are
+  Except within 10 degrees of the galactic plane, the AP Database
+  shall keep up with the incoming rates.  The expected rates are
   listed in Table~\ref{APrates}, along with the total data volume
-  required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2, TLR:3, TLR:22}
+  required for storage space over the PS-1 lifetime.\VER{TEST}{TLR:2,
+  TLR:3, TLR:22}
 
 \item The AP Database shall provide access to external Pan-STARRS
@@ -940,13 +949,13 @@
   computers.\VER{TEST}{TLR:17}
 
-\item The IPP Controller shall limit command latency to \tbr{$< 0.1$} seconds.\VER{TEST}{TLR:17}
-
-\item The IPP Controller shall be capable of performing up to \tbr{10 tasks per second}.\VER{TEST}{TLR:17}
-
-\item The IPP Controller shall be capable of buffering up to a total of \tbr{64 MB} of messages.\VER{TEST}{TLR:17}
-
-\item The IPP Controller shall be capable of executing up to \tbr{6 million tasks per month}.\VER{TEST}{TLR:17}
-
-\item The IPP Controller shall be capable of interacting with up to \tbr{256} client processes.\VER{TEST}{TLR:17}
+\item The IPP Controller shall limit command latency to $< 0.1$ seconds.\VER{TEST}{TLR:17}
+
+\item The IPP Controller shall be capable of performing up to 10 tasks per second.\VER{TEST}{TLR:17}
+
+\item The IPP Controller shall be capable of buffering up to a total of 64 MB of messages.\VER{TEST}{TLR:17}
+
+\item The IPP Controller shall be capable of executing up to 6 million tasks per month.\VER{TEST}{TLR:17}
+
+\item The IPP Controller shall be capable of interacting with up to 256 client processes.\VER{TEST}{TLR:17}
 
 \item The IPP Controller shall be capable of accepting up to 2 non-client (external) requests per second.\VER{TEST}{TLR:17}
@@ -987,9 +996,10 @@
 \begin{enumerate}
 \item The IPP Scheduler shall publish the static sky images to the
-  Pan-STARRS PSPS on a time-scale of \tbr{6 month}.\VER{TEST}{TLR:19}
+  Pan-STARRS PSPS at a rate so that the full sky is transmitted once
+  per year.\VER{TEST}{TLR:19}
 
 \item The IPP Scheduler shall query the Databases on a regular basis
   to check for new input information.  These queries shall take place
-  at least once every \tbr{1 seconds}.\VER{TEST}{TLR:17}
+  at least once every second.\VER{TEST}{TLR:17}
 
 \item The IPP Scheduler shall accept new User input in real-time:
@@ -997,8 +1007,9 @@
 
 \item The IPP Scheduler shall publish the detected objects to the
-  Pan-STARRS PSPS on a time-scale of \tbr{1 month}.\VER{TEST}{TLR:20}
+  Pan-STARRS PSPS at a rate so that the objects from the full sky are
+  transmitted once per year.\VER{TEST}{TLR:20}
 
 \item The IPP Scheduler shall publish the IPP and OTIS metadata to the
-  Pan-STARRS PSPS on a time-scale of \tbr{1 week}.\VER{TEST}{TLR:21}
+  Pan-STARRS PSPS on a time-scale of 1 week.\VER{TEST}{TLR:21}
 
 \item The IPP Scheduler shall send the detected single-occurance
@@ -1085,6 +1096,6 @@
 
 \item Calculate the Image cell / Sky cell overlaps for each image.
-  Sky cells which do not have sufficient science image overlap \tbr{$<
-  5\%$} are excluded from the overlap table.
+  Sky cells which do not have sufficient science image overlap $< 5\%$
+  are excluded from the overlap table.
 
 \end{itemize}
@@ -1101,10 +1112,10 @@
 
 \item Bright-star extraction from the image data shall be performed in
-  less than \tbr{1 second}.\VER{TEST}{TLR:17}
+  less than 1 second.\VER{TEST}{TLR:17}
   
 \item In order for blind astrometry of an image to succeed, it is
   necessary that approximate image coordinates be known.  The Phase 1
   analysis shall succeed despite initial coordinate errors as large as
-  \tbr{20\arcsec}.\VER{TEST}{TLR:3}
+  20\arcsec.\VER{TEST}{TLR:3}
   
 \end{enumerate}
@@ -1128,5 +1139,5 @@
 
 \item Mask ghosts of bright stars which introduce residual feature
-  more significant than \tbr{1\%} of the background.
+  more significant than 1\% of the background.
 
 \item Bias subtract the image.
@@ -1188,6 +1199,6 @@
   time. \VER{TEST}{TLR:17}
 
-\item The bias subtraction shall leave no residuals greater than
-  \tbr{1 DN} peak-to-peak for images within the normal range of bias
+\item The bias subtraction shall leave no residuals greater than 1 DN
+  peak-to-peak for images within the normal range of bias
   variations.\VER{TEST}{TLR:1}
 
@@ -1206,9 +1217,10 @@
 
 \item The background residuals shall have peak-to-peak variations of
-  less than \tbr{1\%} of the input background amplitude.\VER{ANALYSIS}{TLR:5}
-
-\item The background residuals shall have a scatter of less than
-  \tbr{1\%} of the input background amplitude for apertures of less
-  than \tbr{10 arcsec}.\VER{ANALYSIS}{TLR:1}
+  less than 1\% of the input background
+  amplitude.\VER{ANALYSIS}{TLR:5}
+
+\item The background residuals shall have a scatter of less than 1\%
+  of the input background amplitude for apertures of less than 10
+  arcsec.\VER{ANALYSIS}{TLR:1}
 
 \item The Phase 2 analysis shall detect cosmic rays with flux $> 5\sigma$ by
@@ -1235,6 +1247,6 @@
   photometric zero point and zero-point variations across the field.
 
-\item If zero-point variations are significant (\tbr{$> 0.01$ mag
-  peak-to-peak}), the zero-point variations are modeled with a
+\item If zero-point variations are significant ($> 0.01$ mag
+  peak-to-peak), the zero-point variations are modeled with a
   polynomial correction of order 3 or less.
 
@@ -1437,9 +1449,10 @@
 
 \begin{enumerate}
-\item The IPP Calibration Analysis shall produce master calibration images
-from the raw calibration images in less \tbr{2 hours}.\VER{TEST}{TLR:17, TLR:22}
+\item The IPP Calibration Analysis shall produce master calibration
+  images from the raw calibration images in less 2
+  hours.\VER{TEST}{TLR:17, TLR:22}
 
 \item Master calibration images shall not introduce systematic
- uncertainties in the photometry greater than \tbr{0.2\%}.\VER{TEST}{TLR:1}
+ uncertainties in the photometry greater than 0.2\%.\VER{TEST}{TLR:1}
 
 \end{enumerate}
@@ -1477,6 +1490,5 @@
 
 \item The Dark calibration stage raises an error if the input images
-  have exposure times which deviate by more than 
-  \tbr{2\%}.
+  have exposure times which deviate by more than 2\%.
 
 \item The Dark calibration stage corrects the input dark images for
@@ -1538,10 +1550,10 @@
 
 \item The Mask calibration stage masks the pixels which are
-  inconsistent in the input flats by more than \tbr{1\%}, given
-  sufficient signal-to-noise in the input flats.
+  inconsistent in the input flats by more than 1\%, given sufficient
+  signal-to-noise in the input flats.
 
 \item The Mask calibration stage mask the pixels which are
   consistently low or high in the input flats by more than a factor of
-  \tbr{3} beyond the typical pixel.
+  3 beyond the typical pixel.
 
 \item The Mask calibration stage masks the pixels identified in a
@@ -1660,5 +1672,5 @@
 \item The IPP Calibration system monitors changes in the telescope
   astrometry parameters and issue a warning if the parameters change
-  by more than \tbr{2\%}.
+  by more than 2\%.
 \end{itemize}