Changeset 840

Timestamp:

Jun 2, 2004, 5:13:23 PM (22 years ago)

Author:

eugene

Message:

added the old SRS text after document end

File:

• trunk/doc/design/hardware.tex (modified) (1 diff)

trunk/doc/design/hardware.tex

@@ -806 +806 @@
 
 \end{document}
+
+
+\subsection{Computer Hardware}
+
+\subsubsection{Overview}
+
+This section discusses the Pan-STARRS Image Processing Pipeline (IPP)
+PS-1 hardware requirements.  The hardware requirements addressed in
+this section consist of:
+
+\begin{itemize}
+\item Total Disk Volume
+\item Total Processing Power
+\item Sustained Switch Bandwidth
+\item Sustained Node Network I/O
+\item Sustained Disk I/O
+\end{itemize}
+
+Even without the complete IPP design, it is possible to identify the
+major drivers on the hardware requirements.  The total disk volume
+requirements are dominated by the need to store raw images for a
+certain period, the need to store calibration images for a longer
+period, and the need to store the static sky images.  Of the various
+analysis stages, Phase 2 and Phase 4 present the most significant
+demands in terms of data I/O throughput on the network.  Phase 2 and
+Phase 4 also present the most significant CPU demands.  In this
+discusion, Phase 2 refers to the per-OTA image pre-processing in which
+the instrumental signature is removed and a first pass object
+detection is performed.  Phase 4 refers to the multiple OTA
+combination in which the pre-processed images are merged and combined,
+in both addition and subtraction, with the static sky image, and up to
+three object detection passes are performed.
+
+This document does not address the hardware requirements implied by
+Phase 1 or 3, nor the load required by the calibration or reference
+catalog creation stages.  In the first instance, the operations are
+only performed on the metadata and are extremely minimal both in terms
+of data I/O and computation requirements.  In the second case, the
+processing is less time critical than the per-image processing and is
+performed only infrequently (once per night to once per week, month or
+year).  \tbd{The software implementation for metadata storage (RDBMS,
+FITS tables, etc) will have a very large impact and will be evaluated
+along with the needed hardware at a later date.}
+
+We will address the various hardware requirements by referring to an
+assumed data processing and data organization scenario.  The
+organization of the data and certain aspects of the data processing
+scheme have very large implications for the hardware requirements.  In
+this analysis, we assume that data types are chosen to minimize the
+data volume and that the data is organized to minimize the I/O
+bandwidth needs, as defined below.  We address the data requirements
+of the single-telescope Pan-STARRS-1 scenario based on the Design
+Reference Mission \tbd{REF}.
+
+\subsubsection{Data Organization}
+
+The IPP hardware system must provide both data storage and
+computational resources.  The IPP requires relatively large amounts of
+data storage space, primarily for the image data.  Image data is
+organized in two categories.  First, there is the per-OTA data -- data
+associated with specific OTAs, including the raw images, the
+calibration images, and temporary processed images at various stages.
+Second, there is the data associated with the static sky imagery,
+which is in turn organized into smaller sky-cell units.  The first
+assumption we make is that the hardware is organized into nodes which
+provide both data storage and computational resources.  The second
+assumption we make is that the data storage nodes are divided into two
+classes: those which deal with the per-OTA data and those that provide
+the static sky storage.  In addition, we assume that the computational
+tasks related to Phase 2 take place on the per-OTA storage nodes and
+the Phase 4 computation takes place on the static sky storage nodes.
+
+Figure~\ref{hardware} shows our basic concept for the hardware
+organization for the IPP.  This diagram shows the two types of compute
+nodes: OTA-level processing and storage nodes (dominated by Phase 2)
+and static sky processing and storage nodes (mostly Phase 4).  Also
+shown are two switches used in this configuration; although it is
+currently possible to buy a single switch with sufficient number of
+ports, this organization represents a minimal configuration for the
+PS-1 IPP hardware.  In such a case, the interswitch communication must
+also meet the required throughput needs.  We discuss the hardware
+requirements in the assumption that such an organization will be
+necessary.
+
+The way in which the images are distributed among the storage and
+compute nodes will largely determine the I/O bandwidth requirements.
+For data bandwidth requirements calculations, it is necessary to make
+some assumptions about the data organization.  We make the assumption
+that the OTA data is optimally distributed to the OTA nodes such that
+the OTA processing is always on a machine with local OTA data.  This
+implies that all OTA data from a specific OTA are targetted to a
+specific machine.  (see below for discussion of data duplication).
+
+A second factor which will have a significant impact on the I/O
+requirements is the image storage format for the processed and
+calibration images.  We have two basic choices: 32 bit floating point
+format or 16 bit integer format with appropriate scaling.  In the
+former case, additional dynamic range is retained, while in the latter
+case, we reduce the data volume by a factor of 2.  Since the science
+requirements for PS-1 do not specify a need for dynamic range greater
+than 16 bits, we assume all images are stored as 16 bit data.
+
+A third determining factor is the number of calibration images needed
+by the processing system.  Since the complete analysis is not yet
+defined, this number is difficult to ascertain.  However, we can make
+a reasonable guess at the total number for scaling purposes.  We
+assume that each frame requires a total of 4 calibration frames on
+average 
+
+\begin{table}[b]
+\begin{center}
+\caption{Data Storage Requirements \label{storage}}
+\begin{tabular}{lrrrr}
+\hline
+\hline
+Raw data           & 200 TB \\ 
+static sky         & 256 TB \\
+calibration frames &   5 TB \\
+metadata db        & 0.3 TB \\
+object db          &   4 TB \\
+\hline
+total              & 116 TB \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\subsubsection{Data Storage Requirements}
+
+The Pan-STARRS IPP data storage requirements may be divided into five
+principal areas: raw image data, static sky image data, master
+calibration images, the metadata database, and the object database.
+We discuss each of these data items and their impact on the data
+storage requirements for the IPP for PS-1.  Table~\ref{storage}
+summarizes the data storage requirements in the different scenarios.
+
+\paragraph{Raw Data Storage}
+
+There are two basic image types which will be acquired: night-time
+science images and calibration images.  The night-time science images
+consist of 1Gpix per image, or 2GB in raw format.  At nominal cadence,
+the PS-1 telescope can obtain images at a sustained rate of 1 image
+per 30 seconds for the entire night of 10 hours (36000 seconds).  A
+total of 100 calibration images per night would be a substantial
+overestimate of the typical expectation.  Combining these numbers, we
+can expect to receive a total of 1300 images, or 2.6 TB of data per
+night.  The total data storage requirements for the raw data are
+governed by the number of nights' worth of data we are required to
+keep online.  \tbd{for the first year, we are required to keep all
+images from the AP and IPV surveys.  This amounts to a total of 200
+TB of data}.
+
+\paragraph{Static Sky Data Storage}
+
+The static sky is represented by images with 0.2 arcsec per pixel.
+There will be one summed image and one weight image for each of the
+\tbd{6} filters, each stored with 16 bits of resolution, for a total
+of 24 bytes per sky pixel.  At this resolution, there are 324 Mpix per
+square degree, and we will observe a potential total area of 30,000
+square degrees.  Allowing for 10\% overage for overlapping tiling, we
+require a total of 10.7 Tpix to cover the sky once, or a total of
+$\sim 256$ TB to maintain a single image of the static sky in all 6
+filters.
+
+\paragraph{Calibration Frame Storage}
+
+The possible required calibration frames consist of the bias, dark,
+and mask images, along with one flat, one flat-correction, and
+multiple sky/fringe library frames per filter.  In fact, not all types
+are needed at all stages.  It is very likely that we will not require
+bias or dark images, and mask images may be represented by a single
+byte per pixel.  Nonetheless, it is necessary for us to generate and
+store all master calibration frames at least until we prove that they
+are not needed.  We assume a total of 21 calibration images are
+necessary (one flat, fringe, and sky per filter, along with a bias,
+dark, and mask).  If we intend to keep all master calibration frames
+for the project lifetime, and generate a new master on a weekly basis
+(a reasonable time-scale), then we can expect to require a total of 5
+TB of calibration image by the end of the 2 years of PS-1.  We note
+that this is likely to be a drastic overestimate as we are unlikely to
+need to regenerate all master calibration frames on a weekly
+time-scale.
+
+\paragraph{Metadata Database Storage}
+
+The metadata data storage requirements are driven by the need to store
+the data for the project lifetime.  There are two types of metadata
+generated at the summit: data associated with images and environmental
+data.  The environmental data consists of measurements on a regular
+cadence, roughly 1 per minute, of a variety of parameters.  We suggest
+an expected of 1kB per entry, for a total of 1 GB over the two-year
+term of PS-1.  The additional systems, such as the DIMM, SkyProbe, NIR
+Sky Camera, and the LRProbe will have higher data requirements, but
+should be considered as separate, self-contained systems.  Their data
+products are distilled to a limited number of parameters per minute
+which are included in the 1kB given above.  Furthermore, items such as
+guide-star history, if saved, will be saved with the image data and
+represents only a small fraction of the total image data volume.  Some
+subset of the telescope diagnosic information may be a high volume
+data product as well, but only retained by the telescope control
+system for the purpose of diagnostic studies.  Such data will be
+excluded from this analysis.
+
+The image metadata consists of values associated with the FPA (1), the
+OTAs (64), and the Cells (4096).  Aside from the guide star history,
+the total data requirements for each of these entries will be scaled
+by the number of bytes required for the metadata from each data level.
+Clearly, if the Cell entry is allowed to be large, it will dominate
+the total Metadata data volume.  We suggest an expected number of 64
+bytes per Cell, 256 B per OTA, and 1k per FPA, yielding a total
+metadata volume per exposure of roughly 0.3 MB, completely dominated
+by the Cell metadata.  With the exposure rates above, we find a total
+of metadata volume of 0.3 TB over the two-year term of PS-1. 
+
+\paragraph{Object Database Storage}
+
+The hardware requirements for the IPP object database are rather
+flexible: the total volume depends critically on the depth to which
+the object detection analyses are performed (and thus the total number
+of object detections) and the number of object parameters which are
+measured.  We can make very rough estimates that the total number of
+detections over the 2 year lifetime of the project may be in the
+vicinity of $10^{11}$.  We can conservatively estimate the number of
+bytes needed to represent each detection as 128 B, resulting in a
+total data storage for the object detections of 12 TB.  However, this
+number depends strongly on the timescale for which the IPP is required
+to maintain all object detections, and may potentially be
+significantly reduced.
+
+\subsubsection{CPU Requirements}
+
+Phase 2 and Phase 4 dominate the processing requirement, primarily
+because they must keep up with the image delivery rate of 1 per 30
+seconds.  We have performed benchmarks of a demonstration version for
+both the Phase 2 and Phase 4 analyses.
+
+For the Phase 2, a substantial fraction of the processing time is
+consumed by the need to perform FFTs on the images in order to
+convolve them with the guide-star kernel, and in the smoothing used
+for the object detection process.  Additional processing time is
+needed by the object detection, deblending, and analysis.  Experiments
+with the FFTW package show that FFTs may be performed on Intel
+processors at rates of approximately 0.25 GHz-sec / Mpix for data sets
+of order 1 Megapixel.  The FFTs required for the Phase 2 analysis are
+performed on the 512$^2$ pixel cells, so these numbers may roughly be
+scaled linearly to determine the total time required for OTA
+processing.  A single FFT on a full OTA, with 64 Cells, therefore
+requires roughly 4 GHz-sec.  For the full Phase 2 analysis, there are
+roughly 4 single direction FFTs required excluding those associated
+with object detection; thus the total processing time for these FFTs
+is approximately 16 GHz-sec.  The addtional analysis steps, excluding
+object detection and characterization, account for a small fraction of
+this compute time, which we estimate at 10\%.  The object detection
+stage depends somewhat on the depth to which the analysis is
+performed, and the number of measurements made per object.  Typical
+analysis performed by the Sextractor routine, which performs a
+substantial number of per-object analyses, requires 27 GHz-sec for a
+full OTA, including the FFTs used for smoothing.  We can therefore
+assume a total of 50 GHz-sec per OTA for the Phase 2 processing.  This
+converts to a total of 12800 GHz-sec for a complete major frame.
+
+For Phase 4, the main computational tasks are combining the multiple
+images, with cosmic-ray rejection, and performing the object detection
+tasks.  Nick Kaiser has done tests of the Phase 4 image combine and
+rejection stages, and finds a total processing time of roughly 96
+GHz-sec for a full stack of 4 OTA images.  If we add in an additional
+34 GHz-sec for detailed object detection and image differencing, we
+find a conservative estimage of 130 GHz-sec for a 4-image OTA stack,
+equivalent to 7800 GHz-sec for a major frame.
+
+For PS-1, the typical time for a major frame is $4 \times 30$ seconds.
+Some reduction in the load may be gained by reducing the complexity
+and depth of analysis for PS-1.  Depending on the details and depth of
+the analysis, we may reduce the computational load by a factor of 2.
+
+\begin{table}
+\begin{center}
+\caption{Data I/O (MB per OTA or Sky-cell) \label{scenarios}}
+\begin{tabular}{lrrrr}
+\hline
+\hline
+{\em Phase 2 input}                                \\
+from summit    &                 $2 \times 32$ MB  \\
+input image    &                       {\bf 32 MB} \\
+calibration    &            {\bf 4 $\times$ 32 MB} \\
+mask image     &                       {\bf  8 MB} \\
+\hline
+network I/O:   &                            64 MB  \\
+disk I/O:      &                           176 MB  \\
+               &                                   \\
+{\em Phase 2 output}                               \\
+output image   &                      {\bf  32 MB} \\
+output mask    &                      {\bf   8 MB} \\
+image to P4    &               $1.5 \times 32$ MB  \\
+mask to P4     &               $1.5 \times  8$ MB  \\
+\hline
+network I/O:   &                            60 MB  \\
+disk I/O:      &                            40 MB  \\
+               &                                   \\
+{\em Phase 4}  &                                   \\
+input images   &      $1.5 \times 4 \times 32$ MB  \\
+input masks    &      $1.5 \times 4 \times  8$ MB  \\
+static sky     &                            32 MB  \\
+static weight  &                            32 MB  \\
+\hline
+input:         &                           304 MB  \\
+output:        &                            96 MB  \\
+\hline
+\multicolumn{5}{l}{\em Bold-faced entries are access to local-disk} \\ 
+\multicolumn{5}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\ 
+\end{tabular}
+\end{center}
+\end{table}
+
+\subsubsection{Per-Node I/O Requirements}
+
+Data I/O per node is defined as the number of bytes per second passed
+through the node's network adapter.  The data throughput for each node
+depends strongly on the how the data is organized and processed.  In
+this section, we identify the data which is passed between nodes for
+the two stages of the science analysis process.  Table~\ref{scenarios}
+lists the per-node data I/O for the analysis stages.
+
+For PS-1, there are 120 seconds of compute time allowed for each of
+the Phase 2 and Phase 4 analyses for the collection of four images
+which makes up a cannonical major frame.  We use the data I/O volumes
+and some assumptions about expected network and disk bandwidth to
+estimate the I/O and processing timeline for the four scenarios. From
+this analysis, we can judge the total CPU requirements in terms of
+GHz, not just GHz-sec.  We have assumed that GigE network adapters are
+capable of delivering data at 50MB/sec sustained and that a disk RAID
+can deliver sustained 100 MB/sec reads and writes.  These numbers are
+conservative estimates based on recent tests discussed below.  Using
+these assumptions, Table~\ref{throughput} lists the time allocations
+for the processing stages.
+
+\paragraph{Phase 2 Node I/O Requirements}
+
+In the assumed data distribution scenario, there is a single CPU
+allocated to each OTA in the OTA farm and a single CPU for each Sky
+cell process.  In addition, all data for the specified OTA are stored
+on local disks attached to the same computer as the CPU, with the
+result that all Phase 2 I/O is made to a local disk.  For each science
+OTA image which is observed, each OTA node will read from the network
+a total of 2 raw images (one for the original image, one for a backup
+copy) and write an average of roughly 1.5 processed images and masks
+to the Phase 4 machines for a total of 124 MB of network I/O.  During
+the processing stage, the OTA node will read from disk a total of 176
+MB (4 calibration frames at 32 MB each, one 16 MB mask, and one raw
+science image at 32 MB) and write a total of 40 MB (one processed
+image at 32 MB and one mask at 8 MB).  Given the assumptions for the
+network and disk bandwidths (50 MB/s and 100 MB/s respectively), the
+data volumes imply a total I/O period of 4.6 seconds.  In this
+instance, the network I/O is presumed to be sequential with the disk
+I/O.
+
+\paragraph{Phase 4 Node I/O Requirements}
+
+Although it is easy to arrange the OTA data in such a way that the
+majority of I/O is performed locally, it is not as easy to arrange
+this for the Static Sky data used by the Phase 4 analysis.  We
+therefore make the assumption that the Phase 4 analysis will require
+all input OTA data to be loaded across the network, as well as all
+Static Sky data.  This is somewhat of an overestimate as some of the
+Static Sky data will be processed by machines with the data stored
+locally, and clever Static-Sky data organization schemes can enhance
+this chance.  
+
+In the Phase 4 analysis, the images from the 4 separate telescopes are
+combined into a single image, confronted with the appropriate segment
+of the static sky, with output difference image and updated static sky
+image.  If we restrict input access to the individual OTA cells, the
+maximum read overhead is 50\% (need to read a 10x10 set of cells for
+an 8x8 input image).  If the processing is performed on Static Sky
+segments equivalent in size to the OTAs, the total volume of input
+data per node is 304 MB (192 MB of processed science image, 48 MB of
+input mask, 32 MB of static sky image and 32 MB of static sky weight
+map) while the output data is 96 MB (32 MB static sky, 32 MB weight
+map, and 32 MB difference image).  Thus, we require a total of 400 MB
+network I/O, which implies an I/O period of 8 seconds.
+
+\begin{table}
+\begin{center}
+\caption{Data Throughput \label{throughput}}
+\begin{tabular}{lrrrr}
+\hline
+\hline
+Phase 2 per-node network I/O       & 2.2 s           \\
+Phase 2 per-node disk I/O (read)   & 1.3 s           \\
+Phase 2 per-node disk I/O (write)  & 1.2 s           \\        
+Phase 2 CPU total                  &  25 s : 128 GHz \\
+Phase 4 per-node I/O               &   8 s           \\
+Phase 4 CPU total                  & 112 s : 70 GHz  \\
+Phase 2 switch load                & 264 MB/s \\
+Phase 4 switch load                & 215 MB/s \\
+Phase 2 to Phase 4 switch load     & 160 MB/s \\
+Summit to Phase 2 switch load      &  70 MB/s \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\subsubsection{Switch I/O Requirements}
+
+The switch I/O requirements are defined by the total number of bytes
+per second serviced by the two switches in the system.  
+
+The Phase 2 network I/O is 124 MB per OTA.  With 64 OTAs per image,
+and 30 seconds average between images, this implies a total of 264
+MB/s switch bandwidth.  The Phase 4 network I/O is 400 MB per sky
+cell.  With 64 cells and 120 seconds between major frames, this is an
+average switch bandwidth of 215 MB/s switch bandwidth.  The total
+switch-to-switch load is 304 MB per OTA, with an average timescale of
+120 seconds.  With 64 OTAs, this corresponds to 160 MB/s.  The
+summit-to-Phase 2 switch load is 70 MB/s.
+
+\begin{table}
+\begin{center}
+\caption{Hardware Throughput Tests \label{existing-hardware}}
+\begin{tabular}{lrrrr}
+\hline
+\hline
+Test        & where \& when     & model                & result                             \\
+\hline
+node I/O    & CFHT 11/2002      & Intel 1000 Gigabit   & 35 - 40 MB/s sustained             \\
+node I/O    & CFHT 2/2004       & Intel 1000 Gigabit   & 65 - 70 MB/s sustained             \\
+RAID write  & CFHT 2/2004       & 3ware RAID cntl + IDE & 110 MB/s sustained                 \\
+Switch Load & VeriTest          & Cisco                & 3 GB/s (for 32 ports)              \\
+\hline
+\end{tabular}
+\end{center}
+\end{table}
+
+\subsubsection{Existing Hardware Throughput}
+
+We have collected a few representative tests of various pieces of
+modern hardware to give a reference for the throughput capabilities.
+A number of hardware configurations have been tested at CFHT for the
+Elixir project, and we include here their recent reported hardware
+RAID-5 I/O speeds and GigE card speeds.  We also have included data
+from VeriTest studies of Cisco switch throughput, commissioned by
+Cisco for a 32 port GigE switch.  These tests are summarized in
+Table~\ref{existing-hardware}.
+