Index: trunk/doc/design/hardware.tex =================================================================== --- trunk/doc/design/hardware.tex (revision 669) +++ trunk/doc/design/hardware.tex (revision 840) @@ -806,2 +806,446 @@ \end{document} + + +\subsection{Computer Hardware} + +\subsubsection{Overview} + +This section discusses the Pan-STARRS Image Processing Pipeline (IPP) +PS-1 hardware requirements. The hardware requirements addressed in +this section consist of: + +\begin{itemize} +\item Total Disk Volume +\item Total Processing Power +\item Sustained Switch Bandwidth +\item Sustained Node Network I/O +\item Sustained Disk I/O +\end{itemize} + +Even without the complete IPP design, it is possible to identify the +major drivers on the hardware requirements. The total disk volume +requirements are dominated by the need to store raw images for a +certain period, the need to store calibration images for a longer +period, and the need to store the static sky images. Of the various +analysis stages, Phase 2 and Phase 4 present the most significant +demands in terms of data I/O throughput on the network. Phase 2 and +Phase 4 also present the most significant CPU demands. In this +discusion, Phase 2 refers to the per-OTA image pre-processing in which +the instrumental signature is removed and a first pass object +detection is performed. Phase 4 refers to the multiple OTA +combination in which the pre-processed images are merged and combined, +in both addition and subtraction, with the static sky image, and up to +three object detection passes are performed. + +This document does not address the hardware requirements implied by +Phase 1 or 3, nor the load required by the calibration or reference +catalog creation stages. In the first instance, the operations are +only performed on the metadata and are extremely minimal both in terms +of data I/O and computation requirements. In the second case, the +processing is less time critical than the per-image processing and is +performed only infrequently (once per night to once per week, month or +year). \tbd{The software implementation for metadata storage (RDBMS, +FITS tables, etc) will have a very large impact and will be evaluated +along with the needed hardware at a later date.} + +We will address the various hardware requirements by referring to an +assumed data processing and data organization scenario. The +organization of the data and certain aspects of the data processing +scheme have very large implications for the hardware requirements. In +this analysis, we assume that data types are chosen to minimize the +data volume and that the data is organized to minimize the I/O +bandwidth needs, as defined below. We address the data requirements +of the single-telescope Pan-STARRS-1 scenario based on the Design +Reference Mission \tbd{REF}. + +\subsubsection{Data Organization} + +The IPP hardware system must provide both data storage and +computational resources. The IPP requires relatively large amounts of +data storage space, primarily for the image data. Image data is +organized in two categories. First, there is the per-OTA data -- data +associated with specific OTAs, including the raw images, the +calibration images, and temporary processed images at various stages. +Second, there is the data associated with the static sky imagery, +which is in turn organized into smaller sky-cell units. The first +assumption we make is that the hardware is organized into nodes which +provide both data storage and computational resources. The second +assumption we make is that the data storage nodes are divided into two +classes: those which deal with the per-OTA data and those that provide +the static sky storage. In addition, we assume that the computational +tasks related to Phase 2 take place on the per-OTA storage nodes and +the Phase 4 computation takes place on the static sky storage nodes. + +Figure~\ref{hardware} shows our basic concept for the hardware +organization for the IPP. This diagram shows the two types of compute +nodes: OTA-level processing and storage nodes (dominated by Phase 2) +and static sky processing and storage nodes (mostly Phase 4). Also +shown are two switches used in this configuration; although it is +currently possible to buy a single switch with sufficient number of +ports, this organization represents a minimal configuration for the +PS-1 IPP hardware. In such a case, the interswitch communication must +also meet the required throughput needs. We discuss the hardware +requirements in the assumption that such an organization will be +necessary. + +The way in which the images are distributed among the storage and +compute nodes will largely determine the I/O bandwidth requirements. +For data bandwidth requirements calculations, it is necessary to make +some assumptions about the data organization. We make the assumption +that the OTA data is optimally distributed to the OTA nodes such that +the OTA processing is always on a machine with local OTA data. This +implies that all OTA data from a specific OTA are targetted to a +specific machine. (see below for discussion of data duplication). + +A second factor which will have a significant impact on the I/O +requirements is the image storage format for the processed and +calibration images. We have two basic choices: 32 bit floating point +format or 16 bit integer format with appropriate scaling. In the +former case, additional dynamic range is retained, while in the latter +case, we reduce the data volume by a factor of 2. Since the science +requirements for PS-1 do not specify a need for dynamic range greater +than 16 bits, we assume all images are stored as 16 bit data. + +A third determining factor is the number of calibration images needed +by the processing system. Since the complete analysis is not yet +defined, this number is difficult to ascertain. However, we can make +a reasonable guess at the total number for scaling purposes. We +assume that each frame requires a total of 4 calibration frames on +average + +\begin{table}[b] +\begin{center} +\caption{Data Storage Requirements \label{storage}} +\begin{tabular}{lrrrr} +\hline +\hline +Raw data & 200 TB \\ +static sky & 256 TB \\ +calibration frames & 5 TB \\ +metadata db & 0.3 TB \\ +object db & 4 TB \\ +\hline +total & 116 TB \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\subsubsection{Data Storage Requirements} + +The Pan-STARRS IPP data storage requirements may be divided into five +principal areas: raw image data, static sky image data, master +calibration images, the metadata database, and the object database. +We discuss each of these data items and their impact on the data +storage requirements for the IPP for PS-1. Table~\ref{storage} +summarizes the data storage requirements in the different scenarios. + +\paragraph{Raw Data Storage} + +There are two basic image types which will be acquired: night-time +science images and calibration images. The night-time science images +consist of 1Gpix per image, or 2GB in raw format. At nominal cadence, +the PS-1 telescope can obtain images at a sustained rate of 1 image +per 30 seconds for the entire night of 10 hours (36000 seconds). A +total of 100 calibration images per night would be a substantial +overestimate of the typical expectation. Combining these numbers, we +can expect to receive a total of 1300 images, or 2.6 TB of data per +night. The total data storage requirements for the raw data are +governed by the number of nights' worth of data we are required to +keep online. \tbd{for the first year, we are required to keep all +images from the AP and IPV surveys. This amounts to a total of 200 +TB of data}. + +\paragraph{Static Sky Data Storage} + +The static sky is represented by images with 0.2 arcsec per pixel. +There will be one summed image and one weight image for each of the +\tbd{6} filters, each stored with 16 bits of resolution, for a total +of 24 bytes per sky pixel. At this resolution, there are 324 Mpix per +square degree, and we will observe a potential total area of 30,000 +square degrees. Allowing for 10\% overage for overlapping tiling, we +require a total of 10.7 Tpix to cover the sky once, or a total of +$\sim 256$ TB to maintain a single image of the static sky in all 6 +filters. + +\paragraph{Calibration Frame Storage} + +The possible required calibration frames consist of the bias, dark, +and mask images, along with one flat, one flat-correction, and +multiple sky/fringe library frames per filter. In fact, not all types +are needed at all stages. It is very likely that we will not require +bias or dark images, and mask images may be represented by a single +byte per pixel. Nonetheless, it is necessary for us to generate and +store all master calibration frames at least until we prove that they +are not needed. We assume a total of 21 calibration images are +necessary (one flat, fringe, and sky per filter, along with a bias, +dark, and mask). If we intend to keep all master calibration frames +for the project lifetime, and generate a new master on a weekly basis +(a reasonable time-scale), then we can expect to require a total of 5 +TB of calibration image by the end of the 2 years of PS-1. We note +that this is likely to be a drastic overestimate as we are unlikely to +need to regenerate all master calibration frames on a weekly +time-scale. + +\paragraph{Metadata Database Storage} + +The metadata data storage requirements are driven by the need to store +the data for the project lifetime. There are two types of metadata +generated at the summit: data associated with images and environmental +data. The environmental data consists of measurements on a regular +cadence, roughly 1 per minute, of a variety of parameters. We suggest +an expected of 1kB per entry, for a total of 1 GB over the two-year +term of PS-1. The additional systems, such as the DIMM, SkyProbe, NIR +Sky Camera, and the LRProbe will have higher data requirements, but +should be considered as separate, self-contained systems. Their data +products are distilled to a limited number of parameters per minute +which are included in the 1kB given above. Furthermore, items such as +guide-star history, if saved, will be saved with the image data and +represents only a small fraction of the total image data volume. Some +subset of the telescope diagnosic information may be a high volume +data product as well, but only retained by the telescope control +system for the purpose of diagnostic studies. Such data will be +excluded from this analysis. + +The image metadata consists of values associated with the FPA (1), the +OTAs (64), and the Cells (4096). Aside from the guide star history, +the total data requirements for each of these entries will be scaled +by the number of bytes required for the metadata from each data level. +Clearly, if the Cell entry is allowed to be large, it will dominate +the total Metadata data volume. We suggest an expected number of 64 +bytes per Cell, 256 B per OTA, and 1k per FPA, yielding a total +metadata volume per exposure of roughly 0.3 MB, completely dominated +by the Cell metadata. With the exposure rates above, we find a total +of metadata volume of 0.3 TB over the two-year term of PS-1. + +\paragraph{Object Database Storage} + +The hardware requirements for the IPP object database are rather +flexible: the total volume depends critically on the depth to which +the object detection analyses are performed (and thus the total number +of object detections) and the number of object parameters which are +measured. We can make very rough estimates that the total number of +detections over the 2 year lifetime of the project may be in the +vicinity of $10^{11}$. We can conservatively estimate the number of +bytes needed to represent each detection as 128 B, resulting in a +total data storage for the object detections of 12 TB. However, this +number depends strongly on the timescale for which the IPP is required +to maintain all object detections, and may potentially be +significantly reduced. + +\subsubsection{CPU Requirements} + +Phase 2 and Phase 4 dominate the processing requirement, primarily +because they must keep up with the image delivery rate of 1 per 30 +seconds. We have performed benchmarks of a demonstration version for +both the Phase 2 and Phase 4 analyses. + +For the Phase 2, a substantial fraction of the processing time is +consumed by the need to perform FFTs on the images in order to +convolve them with the guide-star kernel, and in the smoothing used +for the object detection process. Additional processing time is +needed by the object detection, deblending, and analysis. Experiments +with the FFTW package show that FFTs may be performed on Intel +processors at rates of approximately 0.25 GHz-sec / Mpix for data sets +of order 1 Megapixel. The FFTs required for the Phase 2 analysis are +performed on the 512$^2$ pixel cells, so these numbers may roughly be +scaled linearly to determine the total time required for OTA +processing. A single FFT on a full OTA, with 64 Cells, therefore +requires roughly 4 GHz-sec. For the full Phase 2 analysis, there are +roughly 4 single direction FFTs required excluding those associated +with object detection; thus the total processing time for these FFTs +is approximately 16 GHz-sec. The addtional analysis steps, excluding +object detection and characterization, account for a small fraction of +this compute time, which we estimate at 10\%. The object detection +stage depends somewhat on the depth to which the analysis is +performed, and the number of measurements made per object. Typical +analysis performed by the Sextractor routine, which performs a +substantial number of per-object analyses, requires 27 GHz-sec for a +full OTA, including the FFTs used for smoothing. We can therefore +assume a total of 50 GHz-sec per OTA for the Phase 2 processing. This +converts to a total of 12800 GHz-sec for a complete major frame. + +For Phase 4, the main computational tasks are combining the multiple +images, with cosmic-ray rejection, and performing the object detection +tasks. Nick Kaiser has done tests of the Phase 4 image combine and +rejection stages, and finds a total processing time of roughly 96 +GHz-sec for a full stack of 4 OTA images. If we add in an additional +34 GHz-sec for detailed object detection and image differencing, we +find a conservative estimage of 130 GHz-sec for a 4-image OTA stack, +equivalent to 7800 GHz-sec for a major frame. + +For PS-1, the typical time for a major frame is $4 \times 30$ seconds. +Some reduction in the load may be gained by reducing the complexity +and depth of analysis for PS-1. Depending on the details and depth of +the analysis, we may reduce the computational load by a factor of 2. + +\begin{table} +\begin{center} +\caption{Data I/O (MB per OTA or Sky-cell) \label{scenarios}} +\begin{tabular}{lrrrr} +\hline +\hline +{\em Phase 2 input} \\ +from summit & $2 \times 32$ MB \\ +input image & {\bf 32 MB} \\ +calibration & {\bf 4 $\times$ 32 MB} \\ +mask image & {\bf 8 MB} \\ +\hline +network I/O: & 64 MB \\ +disk I/O: & 176 MB \\ + & \\ +{\em Phase 2 output} \\ +output image & {\bf 32 MB} \\ +output mask & {\bf 8 MB} \\ +image to P4 & $1.5 \times 32$ MB \\ +mask to P4 & $1.5 \times 8$ MB \\ +\hline +network I/O: & 60 MB \\ +disk I/O: & 40 MB \\ + & \\ +{\em Phase 4} & \\ +input images & $1.5 \times 4 \times 32$ MB \\ +input masks & $1.5 \times 4 \times 8$ MB \\ +static sky & 32 MB \\ +static weight & 32 MB \\ +\hline +input: & 304 MB \\ +output: & 96 MB \\ +\hline +\multicolumn{5}{l}{\em Bold-faced entries are access to local-disk} \\ +\multicolumn{5}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\ +\end{tabular} +\end{center} +\end{table} + +\subsubsection{Per-Node I/O Requirements} + +Data I/O per node is defined as the number of bytes per second passed +through the node's network adapter. The data throughput for each node +depends strongly on the how the data is organized and processed. In +this section, we identify the data which is passed between nodes for +the two stages of the science analysis process. Table~\ref{scenarios} +lists the per-node data I/O for the analysis stages. + +For PS-1, there are 120 seconds of compute time allowed for each of +the Phase 2 and Phase 4 analyses for the collection of four images +which makes up a cannonical major frame. We use the data I/O volumes +and some assumptions about expected network and disk bandwidth to +estimate the I/O and processing timeline for the four scenarios. From +this analysis, we can judge the total CPU requirements in terms of +GHz, not just GHz-sec. We have assumed that GigE network adapters are +capable of delivering data at 50MB/sec sustained and that a disk RAID +can deliver sustained 100 MB/sec reads and writes. These numbers are +conservative estimates based on recent tests discussed below. Using +these assumptions, Table~\ref{throughput} lists the time allocations +for the processing stages. + +\paragraph{Phase 2 Node I/O Requirements} + +In the assumed data distribution scenario, there is a single CPU +allocated to each OTA in the OTA farm and a single CPU for each Sky +cell process. In addition, all data for the specified OTA are stored +on local disks attached to the same computer as the CPU, with the +result that all Phase 2 I/O is made to a local disk. For each science +OTA image which is observed, each OTA node will read from the network +a total of 2 raw images (one for the original image, one for a backup +copy) and write an average of roughly 1.5 processed images and masks +to the Phase 4 machines for a total of 124 MB of network I/O. During +the processing stage, the OTA node will read from disk a total of 176 +MB (4 calibration frames at 32 MB each, one 16 MB mask, and one raw +science image at 32 MB) and write a total of 40 MB (one processed +image at 32 MB and one mask at 8 MB). Given the assumptions for the +network and disk bandwidths (50 MB/s and 100 MB/s respectively), the +data volumes imply a total I/O period of 4.6 seconds. In this +instance, the network I/O is presumed to be sequential with the disk +I/O. + +\paragraph{Phase 4 Node I/O Requirements} + +Although it is easy to arrange the OTA data in such a way that the +majority of I/O is performed locally, it is not as easy to arrange +this for the Static Sky data used by the Phase 4 analysis. We +therefore make the assumption that the Phase 4 analysis will require +all input OTA data to be loaded across the network, as well as all +Static Sky data. This is somewhat of an overestimate as some of the +Static Sky data will be processed by machines with the data stored +locally, and clever Static-Sky data organization schemes can enhance +this chance. + +In the Phase 4 analysis, the images from the 4 separate telescopes are +combined into a single image, confronted with the appropriate segment +of the static sky, with output difference image and updated static sky +image. If we restrict input access to the individual OTA cells, the +maximum read overhead is 50\% (need to read a 10x10 set of cells for +an 8x8 input image). If the processing is performed on Static Sky +segments equivalent in size to the OTAs, the total volume of input +data per node is 304 MB (192 MB of processed science image, 48 MB of +input mask, 32 MB of static sky image and 32 MB of static sky weight +map) while the output data is 96 MB (32 MB static sky, 32 MB weight +map, and 32 MB difference image). Thus, we require a total of 400 MB +network I/O, which implies an I/O period of 8 seconds. + +\begin{table} +\begin{center} +\caption{Data Throughput \label{throughput}} +\begin{tabular}{lrrrr} +\hline +\hline +Phase 2 per-node network I/O & 2.2 s \\ +Phase 2 per-node disk I/O (read) & 1.3 s \\ +Phase 2 per-node disk I/O (write) & 1.2 s \\ +Phase 2 CPU total & 25 s : 128 GHz \\ +Phase 4 per-node I/O & 8 s \\ +Phase 4 CPU total & 112 s : 70 GHz \\ +Phase 2 switch load & 264 MB/s \\ +Phase 4 switch load & 215 MB/s \\ +Phase 2 to Phase 4 switch load & 160 MB/s \\ +Summit to Phase 2 switch load & 70 MB/s \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\subsubsection{Switch I/O Requirements} + +The switch I/O requirements are defined by the total number of bytes +per second serviced by the two switches in the system. + +The Phase 2 network I/O is 124 MB per OTA. With 64 OTAs per image, +and 30 seconds average between images, this implies a total of 264 +MB/s switch bandwidth. The Phase 4 network I/O is 400 MB per sky +cell. With 64 cells and 120 seconds between major frames, this is an +average switch bandwidth of 215 MB/s switch bandwidth. The total +switch-to-switch load is 304 MB per OTA, with an average timescale of +120 seconds. With 64 OTAs, this corresponds to 160 MB/s. The +summit-to-Phase 2 switch load is 70 MB/s. + +\begin{table} +\begin{center} +\caption{Hardware Throughput Tests \label{existing-hardware}} +\begin{tabular}{lrrrr} +\hline +\hline +Test & where \& when & model & result \\ +\hline +node I/O & CFHT 11/2002 & Intel 1000 Gigabit & 35 - 40 MB/s sustained \\ +node I/O & CFHT 2/2004 & Intel 1000 Gigabit & 65 - 70 MB/s sustained \\ +RAID write & CFHT 2/2004 & 3ware RAID cntl + IDE & 110 MB/s sustained \\ +Switch Load & VeriTest & Cisco & 3 GB/s (for 32 ports) \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\subsubsection{Existing Hardware Throughput} + +We have collected a few representative tests of various pieces of +modern hardware to give a reference for the throughput capabilities. +A number of hardware configurations have been tested at CFHT for the +Elixir project, and we include here their recent reported hardware +RAID-5 I/O speeds and GigE card speeds. We also have included data +from VeriTest studies of Cisco switch throughput, commissioned by +Cisco for a 32 port GigE switch. These tests are summarized in +Table~\ref{existing-hardware}. +