Index: trunk/doc/design/hardware-org.fig =================================================================== --- trunk/doc/design/hardware-org.fig (revision 669) +++ trunk/doc/design/hardware-org.fig (revision 669) @@ -0,0 +1,61 @@ +#FIG 3.2 +Landscape +Center +Inches +Letter +100.00 +Single +-2 +1200 2 +6 6075 2175 9225 5100 +6 6375 2925 8925 4575 +2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 7050 4500 6450 4500 6450 3000 7050 3000 7050 4500 +2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 7950 4500 7350 4500 7350 3000 7950 3000 7950 4500 +2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 8850 4500 8250 4500 8250 3000 8850 3000 8850 4500 +-6 +2 2 0 2 0 7 50 0 45 0.000 0 0 -1 0 0 5 + 6150 2250 9150 2250 9150 2700 6150 2700 6150 2250 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 6750 2700 6750 3000 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 7650 2700 7650 3000 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 8550 2700 8550 3000 +4 0 0 50 0 16 24 0.0000 4 360 2730 6375 5025 Static Sky nodes\001 +-6 +2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 1500 4500 900 4500 900 3000 1500 3000 1500 4500 +2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 2400 4500 1800 4500 1800 3000 2400 3000 2400 4500 +2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 3300 4500 2700 4500 2700 3000 3300 3000 3300 4500 +2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 + 4200 4500 3600 4500 3600 3000 4200 3000 4200 4500 +2 2 0 2 0 7 50 0 45 0.000 0 0 -1 0 0 5 + 750 2250 4350 2250 4350 2700 750 2700 750 2250 +2 1 0 3 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 4350 2475 6150 2475 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 1200 2700 1200 3000 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 2100 2700 2100 3000 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 3000 2700 3000 3000 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 3900 2700 3900 3000 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 + 1200 2250 1200 975 +2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 1 2 + 0 0 1.00 60.00 120.00 + 0 0 1.00 60.00 120.00 + 1200 975 1200 600 +4 0 0 50 0 16 24 0.0000 4 285 1890 1050 2625 OTA switch\001 +4 0 0 50 0 16 24 0.0000 4 285 2190 1425 1050 connection to\001 +4 0 0 50 0 16 24 0.0000 4 360 3210 1425 1485 observatory system\001 +4 0 0 50 0 16 24 0.0000 4 285 1845 1650 5025 OTA nodes\001 +4 0 0 50 0 16 24 0.0000 4 360 2655 6300 2625 static sky switch\001 +4 0 0 50 0 16 18 0.0000 4 210 1335 4575 2400 interswitch\001 +4 0 0 50 0 16 18 0.0000 4 210 405 5025 2775 link\001 Index: trunk/doc/design/hardware.tex =================================================================== --- trunk/doc/design/hardware.tex (revision 669) +++ trunk/doc/design/hardware.tex (revision 669) @@ -0,0 +1,807 @@ +\documentclass[panstarrs,psreport]{panstarrs} +%\documentclass[panstarrs]{panstarrs} + +% basic document variables +\title{Hardware Requirements for the Pan-STARRS Image Processing Pipeline} +\shorttitle{IPP Hardware Requirements} +\author{Eugene Magnier, Josh Hoblitt} +\group{Pan-STARRS Algorithm Group} +\project{Pan-STARRS Image Processing Pipeline} +\organization{Institute for Astronomy} +\version{00} +\docnumber{PSDC-400-007} + +\begin{document} +\maketitle + +% -- Revision History -- +\RevisionsStart +% version Date Description +DR-00 & 2004/01/30 & Proposal \\ +\hline +DR-01 & 2004/02/19 & Fleshed-out, Cleaned-up, Reorganized \\ +\hline +00 & 2003.03.15 & Merged with hardware cost doc (JH) \\ +\hline +\RevisionsEnd + +\pagebreak +\tableofcontents + +\pagebreak +\listoffigures + +\pagebreak +\pagenumbering{arabic} +\section{Overview} + +This document discusses the likely range of the Pan-STARRS Image +Processing Pipeline (IPP) hardware requirements. The hardware +requirements addressed in this document consist of: + +\begin{itemize} +\item Total Disk Volume +\item Total Processing Power +\item Sustained Switch Bandwidth +\item Sustained Node Network I/O +\item Sustained Disk I/O +\end{itemize} + +Even without the complete IPP design, it is possible to identify the +major drivers on the hardware requirements. The total disk volume +requirements are dominated by the need to store raw images for a +certain period, the need to store calibration images for a longer +period, and the need to store the static sky images. Of the various +analysis pipelines, and depending on the data organization as +discussed below, Phase 2 and Phase 4 present the most significant +demands in terms of data I/O throughput on the network. Phase 2 and +Phase 4 also present the most significant CPU demands. In this +discusion, Phase 2 refers to the per-OTA image pre-processing in which +the instrumental signature is removed and a first pass object +detection is performed. Phase 4 refers to the multiple OTA +combination in which the pre-processed images are merged and combined, +in both addition and subtraction, with the static sky image, and up to +three object detection passes are performed. + +This document does not address the hardware requirements implied by +the Phase 0, 1, or 3 stages, nor the load required by the calibration +image creation stages. In the first instance, the operations are only +performed on the metadata and are extremely minimal both in terms of +data I/O and computation requirements. In the second case, the +processing is less time critical than the per-image processing and is +performed only infrequently (once per night to once per week or +month). This document also does not address any hardware requirements +introduced by the metadata manipulation. The software implementation +for metadata storage (RDBMS, FITS tables, etc) will have a very large +impact and will be evaluated along with the needed hardware at a later +date. + +\section{Scenarios} + +We will address the various hardware requirements by referring to a +set of data processing and data organization scenarios. The actual +hardware requirements will depend on design decisions which are not +yet available. It is possible to define the data organization in ways +which will minimize the hardware requirements, but which will increase +the software development effort. We will discuss both the worst-case +data organization scenario, which does not require significant +intelligence in the software systems, and the optimal data +organization scenario, which will require the software to track the +location of data products more carefully. In addition, this document +will address the data requirements of the complete Pan-STARRS pipeline +with 4 telescopes as well as the single-telescope Pan-STARRS-1 scenario +based on the Design Reference Mission [REF]. + +The IPP hardware system must provide both data storage and +computational resources. The IPP requires relativley large amounts of +data storage space, primarily for the image data. Image data is +organized in two categories. First, there is the per-OTA data -- data +associated with specific OTAs, including the raw images, the +calibration images, and temporary processed images at various stages. +Second, there is the data associated with the static sky imagery, +which is in turn organized into smaller sky-cell units. The first +assumption we make is that the hardware is organized into nodes which +provide both data storage and computational resources. The second +assumption we make is that the data storage nodes are divided into two +classes: those which deal with the per-OTA data and those that provide +the static sky storage. In addition, we assume that the computational +tasks related to Phase 2 take place on the per-OTA storage nodes and +the Phase 4 computation takes place on the static sky storage nodes. + +Figure~\ref{layout} shows our basic concept for the hardware +organization for the IPP. This diagram shows the two types of compute +nodes: OTA-level processing and storage nodes (dominated by Phase 2) +and static sky processing and storage nodes (mostly Phase 4). Also +shown are two switches used in this configuration; although it is +currently possible to buy a single switch which would have a +sufficient number of GigE ports for both sections of the PS-1 system, +such a two-switch organization may be needed for the full Pan-STARRS +system. In such a case, the interswitch communication must also meet +the required throughput needs. We discuss the hardware requirements +in the assumption that such an organization will be necessary. + +\begin{figure} +\begin{center} +\begin{tabular}{c} +\includegraphics[height=10cm]{hardware-org.eps} +\end{tabular} +\end{center} +\caption{ +\label{layout} +Schematic IPP hardware organization. +} +\end{figure} + +The way in which the images are distributed among the storage and +compute nodes will largely determine the I/O bandwidth requirements. +For data bandwidth requirements calculations, it is necessary to make +some assumptions about the data organization. For the purposes of +this document, we explore two extreme-case options: +\begin{itemize} +\item Random Data Distribution - OTA \& Sky data is randomly + distributed within the compute node of a given type (ie, OTA data is + randomly distributed among the OTA compute nodes). +\item Optimal Data Distribution - OTA \& Sky data is optimally + distributed to compute OTA/Sky nodes (OTA processing is always on a + machine with local OTA data). +\end{itemize} +A second factor which will have a significant impact on the I/O +requirements is the image storage format for the processed and +calibration images. We have two basic choices: 32 bit floating point +format or 16 bit integer format with appropriate scaling. In the +former case, additional dynamic range is retained, while in the latter +case, we reduce the data volume by a factor of 2. While some may +argue that the higher dynamic range is necessary, arguments can be +made that the 16 bit range is sufficient. (In particular, the 16 bit +data provides a dynamic range far above the expected 1/1000 fractional +accuracy of the flat-field images). A related question is the number +of calibration images needed by the processing system. Since the +complete analysis is not yet defined, this number is difficult to +ascertain. However, we can make a range of assumptions which are +reasonable. We therefore adopt two data volume scenarios to explore +these possibilites: +\begin{itemize} +\item Standard Data Volume - 32 bit data for processed and calibration + images, average of 7 calibration frames per image. +\item Minimal Data Volume - 16 bit data for processed and calibration + images, average of 4 calibration frames per image. +\end{itemize} +In the discussion that follows, we explore the hardware requirements +implied by the collection of four combinations of these two sets of +scenario options. + +\begin{table} +\begin{center} +\caption{Hardware Throughput Tests \label{hardware}} +\begin{tabular}{lrrrr} +\hline +\hline +Test & where \& when & model & result \\ +\hline +node I/O & CFHT 11/2002 & Intel 1000 Gigabit & 35 - 40 MB/s sustained \\ +node I/O & CFHT 2/2004 & Intel 1000 Gigabit & 65 - 70 MB/s sustained \\ +RAID write & CFHT 2/2004 & 3ware RAID cntl + IDE & 110 MB/s sustained \\ +Switch Load & VeriTest & Cisco & 3 GB/s (for 32 ports) \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\section{Existing Hardware Throughput} + +We have collected a few representative tests of various pieces of +modern hardware to give a reference for the throughput capabilities. +A number of hardware configurations have been tested at CFHT for the +Elixir project, and we include here their recent reported hardware +RAID-5 I/O speeds and GigE card speeds. We also have included data +from VeriTest studies of Cisco switch throughput, commissioned by +Cisco for a 32 port GigE switch. These tests are summarized in +Table~\ref{hardware}. + +\begin{table}[b] +\begin{center} +\caption{Data Storage Requirements \label{storage}} +\begin{tabular}{lrrrr} +\hline +\hline + & Standard / PS-4 + & Standard / PS-1 + & Minimal / PS-4 + & Minimal / PS-1 \\ +\hline +Raw data & 300 TB & 75 TB & 300 TB & 75 TB \\ +static sky & 512 TB & 64 TB & 256 TB & 32 TB \\ +calibration frames & 175 TB & 18 TB & 17 TB & 5 TB \\ +metadata db & 2 TB & 2 TB & 0.2 TB & 0.2 TB \\ +object db & 60 TB & 4 TB & 60 TB & 4 TB \\ +\hline +totals & 1050 TB & 163 TB & 633 TB & 116 TB \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\section{Data Storage Requirements} + +The Pan-STARRS IPP data storage requirements may be divided into five +principal areas: raw image data, static sky image data, master +calibration images, the metadata database, and the object database. +We discuss each of these data items and their impact on the data +storage requirements for the IPP, and identify the impact of the +minimal vs standard data storage requirements as well as the +requirements specifically for PS-1. Table~\ref{storage} summarizes +the data storage requirements in the different scenarios. + +\subsection{Raw Data Storage} + +There are two basic image types which will be acquired: night-time +science images and calibration images. The night-time science images +consist of 1Gpix per image, or 2GB in raw format. At nominal cadence, +the 4 telescopes can obtain images at a sustained rate of 1 image per +30 seconds per telescope for the entire night of 10 hours (36000 +minutes). A total of 100 calibration images per night would be a +substantial overestimate of the typical expectation. Combining these +numbers, we can expect to receive a total of 1300 image per telescope +per night, 5200 image total, or 10.4 TB of data per night. The total +data storage requirements for the raw data are governed by the number +of nights' worth of data we are required to keep online. A reasonable +number is one month to allow a full moon's cycle. Thus, for raw image +storage, we require a total of 300 TB data storage. For PS-1, this +number is simply scaled down by a factor of 4. The choice of the +minimal data volume does not affect these numbers because the raw data +is already stored with 16 bit pixels. ({\bf note: the PS-1 design +reference may now require storage of the entire first year of data, +calculated to be 200 TB}). + +\subsection{Static Sky Data Storage} + +The static sky is represented by images with 0.2 arcsec per pixel. +There will be one summed image and one weight image for each of the 6 +filters, each stored in floating point format. At this resolution, +there are 324 Mpix per square degree, and we will observe a potential +total area of 30,000 square degrees. Allowing for 10\% overage for +overlapping tiling, we require a total of 10.7 Gpix to cover the sky +once, or a total of $\sim 512$ TB for the static sky images. In the +minimal data volume scenario, this value is reduced by a factor of 2, +while in PS-1, the reduction is a factor of roughly 8 because we only +intend to store the static sky for the ecliptic plane survey and the +small IPP verification program ({\bf note: this last point is no +longer valid - the PS-1 static sky may require the entire 3pi}). + +\subsection{Calibration Frame Storage} + +The possible required calibration frames consist of the bias, dark, +and mask images, along with one flat, one flat-correction, and +multiple sky/fringe library frames per filter. In fact, not all types +are needed at all stages. For the standard data volume, we assume an +average of 7 calibration frames per image and filter. This results in +a total of 42 master calibration image per telescope. If we intend to +keep all master calibration frames for the project lifetime, and +generate a new master on a weekly basis (a reasonable time-scale), +then we can expect to require a total of 175 TB of calibration image +by the end of the 5 year lifetime of the project. For the case of +PS-1, the time period is only 2 years, and there is only 1 telescope, +resulting in a factor of 10 reduction in the volume. For the minimal +data case, we reduce the volume by another factor of 3.5. We also note +that this is likely to be a drastic overestimate as we are unlikely to +need to regenerate all master calibration frames on a weekly +time-scale. + +\subsection{Metadata Database Storage} + +The metadata data storage requirements are driven by the need to store +the data for the project lifetime. There are two types of metadata +generated at the summit: data associated with images and environmental +data. The environmental data consists of measurements on a regular +cadence, roughly 1 per minute, of a variety of parameters. We suggest +an expected of 1kB per entry, for a total of 2.6 GB over the lifetime +of the project. PS-1 will represent a smaller amount of data per +minute, and also a factor of 2.5 fewer minutes. We suggest PS-1 may +have a total environmental metadata set smaller by a factor of 5. The +additional systems, such as the DIMM, SkyProbe, NIR Sky Camera, and +the LRProbe will have higher data requirements, but should be +considered as separate, self-contained systems. Their data products +are distilled to a limited number of parameters per minute which are +included in the 1kB given above. Furthermore, items such as +guide-star history, if saved, will be saved with the image data and +represents only a small fraction of the total image data volume. Some +subset of the telescope diagnosic information may be a high volume +data product as well, but only retained by the telescope control +system for the purpose of diagnostic studies. Such data will be +excluded from this analysis. + +The image metadata consists of values associated with the FPA (4), the +OTAs (240), and the Cells (15360). Aside from the guide star history, +the total data requirements for each of these entries will be scaled +by the number of bytes required for the metadata from each data level. +Clearly, if the Cell entry is allowed to be large, it will dominate +the total Metadata data volume. If we suggest an expected number of +64 bytes per Cell, 256 B per OTA, and 1k per FPA, we find a total +metadata volume per exposure of roughly 1 MB, completely dominated by +the Cell metadata. With the exposure rates above, we find a total of +metadata volume of 1.8 TB over the lifetime of the project. For PS-1, +the total volume is reduced by a factor of 2.5 (for the shorter +lifetime) and another factor of 4 (for the lone telescope). Neither +data quantity is affected by the minimal vs standard data volume +choice. + +\subsection{Object Database Storage} + +The hardware requirements for the IPP object database are rather +flexible: the total volume depends critically on the depth to which +the object detection analyses are performed (and thus the total number +of object detections) and the number of object parameters which are +measured. We can make very rough estimates that the total number of +detections over the 5 year lifetime of the project may be in the +vicinity of $5\times10^{11}$. We can conservatively estimate the +number of bytes needed to represent each detection as 128 B, resulting +in a total data storage for the object detections of 60 TB. However, +this number depends strongly on the timescale for which the IPP is +required to maintain all object detections, and may potentially be +significantly reduced. For the case of PS-1, the total number of +detections is likely to be reduced by a factor of 4 for the number of +telescopes, and potentially another significant factor ($\sim 4?$) by +limiting the depth of object detections. Again, the minimal data +volume scenario is irrelevant to the object database volume. + +\section{CPU Requirements} + +Phase 2 and Phase 4 dominate the processing requirement, primarily +because they must keep up with the image delivery rate of 1 per 30 +seconds. We have performed benchmarks of a demonstration version for +both the Phase 2 and Phase 4 analyses. + +For the Phase 2, a substantial fraction of the processing time is +consumed by the need to perform FFTs on the images in order to +convolve them with the guide-star kernel, and in the smoothing used +for the object detection process. Additional processing time is +needed by the object detection, deblending, and analysis. Experiments +with the FFTW package show that FFTs may be performed on Intel +processors at rates of approximately 0.25 GHz-sec / Mpix for data sets +of order 1 Megapixel. The FFTs required for the Phase 2 analysis are +performed on the 512$^2$ pixel cells, so these numbers may roughly be +scaled linearly to determine the total time required for OTA +processing. A single FFT on a full OTA, with 64 Cells, therefore +requires roughly 4 GHz-sec. For the full Phase 2 analysis, there are +roughly 4 single direction FFTs required excluding those associated +with object detection; thus the total processing time for these FFTs +is approximately 16 GHz-sec. The addtional analysis steps, excluding +object detection and characterization, account for a small fraction of +this compute time, which we estimate at 10\%. The object detection +stage depends somewhat on the depth to which the analysis is +performed, and the number of measurements made per object. Typical +analysis performed by the Sextractor routine, which performs a +substantial number of per-object analyses, requires 27 GHz-sec for a +full OTA, including the FFTs used for smoothing. We can therefore +assume a total of 50 GHz-sec per OTA for the Phase 2 processing. This +converts to a total of 12000 GHz-sec for a complete major frame. + +For Phase 4, the main computainal tasks are combining the multiple +images, with cosmic-ray rejection, and performing the object detection +tasks. Nick Kaiser has done tests of the Phase 4 image combine and +rejection stages, and finds a total processing time of roughly 96 +GHz-sec for a full stack of 4 OTA images. If we add in an additional +34 GHz-sec for detailed object detection and image differencing, we +find a conservative estimage of 130 GHz-sec for a 4-image OTA stack, +equivalent to 7800 GHz-sec for a major frame. + +For PS-1, the data processing will clearly require a smaller amount of +computational resources because of the lower image rate. However, the +total number of GHz-sec required for the complete analysis of 4 input +images and the combination with the static sky will remain +more-or-less the same. Some reduction in the load may be gained by +reducing the complexity and depth of analysis for PS-1. Depending on +the details and depth of the analysis, we may reduce the computational +load by a factor of 2. + +\begin{table} +\begin{center} +\caption{Data Scenarios (MB per OTA or Sky-cell) \label{scenarios}} +\begin{tabular}{lrrrr} +\hline +\hline + & Random / Standard & Random / Minimal & Optimal / Standard & Optimal / Minimal \\ +\hline +{\em Phase 2 input} & & & & \\ +from summit & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB \\ +input image & 32 MB & 32 MB & {\bf 32 MB} & {\bf 32 MB} \\ +calibration & $7 \times 64$ MB & $4 \times 32$ MB & {\bf 7 $\times$ 64 MB} & {\bf 4 $\times$ 32 MB} \\ +mask image & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ +\hline +network I/O: & 560 MB & 232 MB & 64 MB & 64 MB \\ +disk I/O: & (560 MB) & (232 MB) & 496 MB & 168 MB \\ + & & & & \\ +{\em Phase 2 output} & & & & \\ +output image & 64 MB & 32 MB & {\bf 64 MB} & {\bf 32 MB} \\ +output mask & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ +image to P4 & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB \\ +mask to P4 & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB \\ +\hline +network I/O: & 200 MB & 100 MB & 120 MB & 60 MB \\ +disk I/O: & (80 MB) & (40 MB) & 80 MB & 40 MB \\ + & & & & \\ +{\em Phase 4} & & & & \\ +input images & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & & \\ +input masks & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & & \\ +static sky & 64 MB & 64 MB & & \\ +static weight & 64 MB & 32 MB & & \\ +\hline +input: & 608 MB & 336 MB & & \\ +output: & 192 MB & 128 MB & & \\ +\hline +\multicolumn{5}{l}{\em Bold-faced entries are access to local-disk} \\ +\multicolumn{5}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\ +\end{tabular} +\end{center} +\end{table} + +\section{Per-Node I/O Requirements} + +Data I/O per node is defined as the number of bytes per second passed +through the node's network adapter. The data throughput for each node +depends strongly on the scenarios identified above. In this section, +we identify the data which is passed between nodes for each of the +different scenarios. Table~\ref{scenarios} lists the per-node data +I/O for the four scenarios. + +For PS-4, there are only 30 seconds of compute time allowed for each +of the Phase 2 and Phase 4 analyses. We use the data I/O volumes and +some assumptions about expected network and disk bandwidth to estimate +the I/O and processing timeline for the four scenarios. From this +analysis, we can judge the total CPU requirements in terms of GHz, not +just GHz-sec. We have assumed that GigE network adapters are capable +of delivering data at 50MB/sec sustained and that a disk RAID can +deliver sustained 100 MB/sec reads and writes. These numbers are +conservative estimates based on recent tests discussed above. Using +these assumptions, Table~\ref{throughput} lists the time allocations +for the complete set of scenarios for the case of PS-4. + +\subsection{Random / Standard Data Scenario} + +In the Random Data Distribution scenario, there is a single CPU +allocated to each OTA in the OTA farm and a single CPU for each Sky +cell process. The OTA data are stored across random machines in the +OTA farm, with the result that every Phase 2 processing requires +network access to the data. For each science OTA image which is +observed, each OTA node will read from the network a total of 560 MB +(the 2 raw images for data storage and the 7 calibration frames, along +with one mask and one raw input image) and write a total of 200 MB +(one processed image and the mask along with the 1.5 processed images +and masks for the Phase 4 analysis). Given the assumption of 50 MB/s +from the network adapter, the total data volume implies an I/O period +of 15.2 seconds. Note that the disk I/O is parallel with the network +I/O and substantially underfills the disk bandwidth. + +\subsection{Random / Minimal Data Scenario} + +In the Random-Minimal, there is a single CPU allocated to each OTA in +the OTA farm and a single CPU for each Sky cell process, and the OTA +data are stored across random machines in the OTA farm. However, the +calibration and the processed science images are stored at 2 bytes per +pixel, the mask is set at 4 bits per pixel, and only 4 calibration +images are assumed. For each science OTA image which is observed, +each OTA node will read from the network a total of 232 MB (the 2 raw +images for data storage and the 4 calibration frames, along with one +mask and one raw input image) and write a total of 100 MB (one +processed image and the mask along with the 1.5 processed images for +the Phase 4 analysis). Given the assumption of 50 MB/s from the +network adapter, the total data volume implies an I/O period of 6.6 +seconds. Again, note that the disk I/O is parallel with the network +I/O and substantially underfills the disk bandwidth. + +\subsection{Optimal / Standard Data Scenario} + +In the Optimal Data Distribution scenario, there is a single CPU +allocated to each OTA in the OTA farm and a single CPU for each Sky +cell process. In addition, all data for the specified OTA are stored +on local disks attached to the same computer as the CPU, with the +result that all Phase 2 I/O is made to a local disk. For each science +OTA image which is observed, each OTA node will read from the network +a total of 2 raw images (one for the original image, one for the +backup copy) and write an average of roughly 1.5 processed images and +masks to the Phase 4 machines for a total of 184 MB of network I/O. +During the processing stage, the OTA node will read from disk a total +of 496 MB (7 calibration frames at 64 MB each, one 16 MB mask, and one +raw science image at 32 MB) and write a total of 80 MB (one processed +image at 64 MB and one mask at 8 MB). Given the assumptions for the +network and disk bandwidths (50 MB/s and 100 MB/s respectively), the +data volumes imply a total I/O period of 9.5 seconds. In this +instance, the network I/O is presumed to be sequential with the disk +I/O. + +\subsection{Optimal / Minimal Data Scenario} + +In the Optimal / Minimal Scenario, the minimal data sizes are used +with the optimal data distribution scheme. In this case, we reduce +the disk I/O volume to 168 read and 40 MB write, and the network +traffic to 124 MB. Given the assumptions for the network and disk +bandwidths, the data volumes imply a total I/O period of 4.6 seconds. +Again, the network I/O is presumed to be sequential with the disk I/O. + +\subsection{Phase 4 Node I/O Requirements / Standard Data Volume} + +Although it is easy to arrange the OTA data in such a way that the +majority of I/O is performed locally, it is not as easy to arrange +this for the Static Sky data used by the Phase 4 analysis. We +therefore make the assumption that the Phase 4 analysis will require +all input OTA data to be loaded across the network, as well as all +Static Sky data. This is somewhat of an overestimate as some of the +Static Sky data will be processed by machines with the data stored +locally, and clever Static-Sky data organization schemes can enhance +this chance. + +In the Phase 4 analysis, the images from the 4 separate telescopes are +combined into a single image, confronted with the appropriate segment +of the static sky, with output difference image and updated static sky +image. If we restrict input access to the individual OTA cells, the +maximum read overhead is 50\% (need to read a 10x10 set of cells for +an 8x8 input image). If the processing is performed on Static Sky +segments equivalent in size to the OTAs, the input data is 608 MB (384 +MB of processed science image, 96 MB of mask images, 64 MB of static +sky image and 64 MB of static sky weight map) while the output data is +192 MB (static sky, weight map, and difference image, each 64 MB). +Thus, we require a total of 800 MB network I/O. Given the network +bandwidth, this implies an I/O period of 16 seconds for Phase 4. + +\subsection{Phase 4 Node I/O Requirements / Minimal Data Volume} + +In the minimal data volume scenario, the Phase 4 analysis volume is +significantly reduced. The total volume of input data is 336 MB (192 +MB of processed science image, 48 MB of input mask, 64 MB of static +sky image and 32 MB of static sky weight map) while the output data is +128 MB (64 MB static sky, 32 MB weight map, and 32 MB difference +image). Thus, we require a total of 464 MB network I/O, which implies +an I/O period of 9.3 seconds. + +\begin{table} +\begin{center} +\caption{Data Throughput for 4 Scenarios \label{throughput}} +\begin{tabular}{lrrrr} +\hline +\hline +& +\multicolumn{1}{c}{Random / Standard} & +\multicolumn{1}{c}{Random / Minimal} & +\multicolumn{1}{c}{Optimal / Standard} & +\multicolumn{1}{c}{Optimal / Minimal} \\ +\hline +Phase 2 per-node network I/O & 15.2 s & 6.6 s & 3.7 s & 2.5 s \\ +Phase 2 per-node disk I/O (read) & (5.6 s) & (2.3 s) & 5.0 s & 1.7 s \\ +Phase 2 per-node disk I/O (write) & (0.8 s) & (0.4 s) & 0.8 s & 0.4 s \\ +Phase 2 CPU total & 14 s : 860 GHz & 23 s : 520 GHz & 20 s : 600 GHz & 25 s : 480 GHz \\ +Phase 4 per-node I/O & 16 s & 9.3 s & & \\ +Phase 4 CPU total & 14 s : 490 GHz & 20 s : 390 GHz & & \\ +Phase 2 switch load & 6.1 GB/s & 2.7 GB/s & 1.5 GB/s & 1.0 GB/s \\ +Phase 4 switch load & 0.8 GB/s & 0.5 GB/s & 0.8 GB/s & 0.5 GB/s \\ +Phase 2 to Phase 4 switch load & 1.1 GB/s & 0.6 GB/s & 1.1 GB/s & 0.6 GB/s \\ +Summit to Phase 2 switch load & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\section{Switch I/O Requirements} + +The switch I/O requirements are defined by the total number of bytes +per second serviced by the two switches in the system. For the +analysis of the Switch I/O requirements, the choice of data +distribution again has a major impact. We again test the four +scenarios discussed above: Random Data Distribution, Random / Minimal, +Optimal Data Distribution, and Optimal / Minimal. + +\subsection{Random / Standard Data Scenario} + +In the Random Data Distribution scenario, each OTA node needs to read +a total of 560 MB from the network and write a total of 200 MB every +30 seconds. With 240 OTA nodes, this corresponds to a total bandwidth +of 6080 MB/sec, or 49 Gb/sec. Note that this includes the bandwidth +needed to copy data from the summit and make two copies on the OTA +machines, as well as the bandwidth to send the processed image +portions to the Phase 4 machines. The Phase 4 processing adds an +additional 320 MB of network I/O per Sky-Cell group, and there are +roughly 60-70 Sky-cells per exposure set. Thus the Phase 4 processing +adds an additional 750 MB/sec network bandwidth. In the architecture +defined in Figure NN, the Sky nodes and the OTA nodes are each +attached to separate switches. An additional bandwidth requirement is +derived by the need to exchange data between these switches in for +Phase 4. The total amount of data exchanged between these switches is +480 MB per Sky-cell, for a total bandwidth of 1120 MB/sec. In +addition, the connection to the summit is a single, separate line +which needs to support the bandwidth requirement of copying all intial +raw images. In our simple model, each raw image is copied twice, +accounting for a total of 15360 MB every 30 seconds, or a bandwidth +load of 512 MB/sec. (Note that this last is double the actual +bandwidth requirement to the summit: a dedicated local circular buffer +would reduce the need for the second copy to come directly from the +summit.) + +\subsection{Random / Minimal Data Scenario} + +In the Random / Minimal Scenario, the data volumes are significantly +reduced. The total Phase 2 bandwidth contribution is 332 MB over 30 +seconds for 240 nodes (2656 MB/sec) and the residual Phase 4 bandwidth +load is 224 MB per Sky cell over 30 seconds (522 MB/sec). The +inter-switch communication is now 240 MB per sky cell over 30 seconds, +or 560 MB/sec. + +\subsection{Optimal / Standard Data Scenario} + +In the Optimal Data Distribution, the Phase 2 network bandwidth is +reduced significantly to 184 MB per OTA node, for a total of +1.5GB/sec, while the Phase 4 network bandwidth remains unchanged at +750 MB/sec. The inter-switch communication also remains the same at +1.12 GB/sec. + +\subsection{Optimal / Minimal Data Scenario} + +In the Optimal / Minimal Scenario, the total Phase 2 network bandwidth +drops to 124 MB per OTA node, for a total of 1.0GB/sec, while the +Phase 4 network bandwidth is 552 MB/sec. The inter-switch +communication remains the same as the Random/Minimal Scenario at 560 +MB/sec. + +\begin{table}[t] +\begin{center} +\caption{\label{NP2} Phase 2 load per major frame (12000 GHz-sec)} +\begin{tabular}{lrrrr} +\hline +\hline +& Random/Standard +& Random/Minimal +& Optimal/Standard +& Optimal/Minimal \\ +\hline +network I/O (GB) & 182 & 80 & 44 & 30 \\ +PS-1 & & & & \\ + I/O (cpu-sec) & 3640 & 1600 & 880 & 600 \\ + CPU (cpu-sec) & 4000 & 4000 & 4000 & 4000 \\ + \# cpus & 64 & 47 & 41 & 38 \\ +PS-4 & & & & \\ + I/O (cpu-sec) & 1820 & 800 & 440 & 300 \\ + CPU (cpu-sec) & 2000 & 2000 & 2000 & 2000 \\ + \# cpus & 127 & 93 & 81 & 77 \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[b] +\begin{center} +\caption{\label{NP4} Phase 4 load per major frame (7800 GHz-sec)} +\begin{tabular}{lrr} +\hline +\hline +& Standard +& Minimal \\ +\hline +network I/O (GB) & 48 & 28 \\ +PS-1 & & \\ + I/O (cpu-sec) & 960 & 557 \\ + CPU (cpu-sec) & 2600 & 2600 \\ + \# cpus & 30 & 26 \\ +PS-4 & & \\ + I/O (cpu-sec) & 480 & 278 \\ + CPU (cpu-sec) & 1300 & 1300 \\ + \# cpus & 59 & 53 \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\section{Conclusions} + +Table~\ref{throughput} presents one way of analysing the hardware +requirements, making a specific set of assumptions about the number of +nodes for the two phases and the expected network and disk +bandwidths. The important conclusion in this analysis is the implied +number of GHz per processor, given the assumptions laid out. +Phase 2 is specified to have 240 OTA nodes, while Phase 4 is specified +to have roughly 60 static sky nodes. The range of Phase 2 CPU +requirements implies that each CPU needs to have speeds in the range +of 2.0 - 3.6 GHz, which sound very plausible for the year 2007, since +these apply to PS-4. + +Another way to represent this information is to use the total number +of MB I/O and the total number of GHz-sec required for the two stages, +confront these with an assumption for the bandwidth per network +adapter and an assumption for the CPU speed and use those numbers to +calculate the minimum number of nodes (CPUs) needed to sustain the +timing requirements. There are quite a few parameters and options to +choose from. We have assumed that for PS-1, the time between major +frames (4 images combined in Phase 4) is 120 seconds, and 30 seconds +for PS-4. We have also assumed that each CPU has one network adapter +associated with it, and use the numbers of 50 MB/sec for PS-1 era +network adapters and 100 MB/sec for the PS-4 network adapters (since +there has been some steady improvement in GigE hardware over the past +year). We have also assumed each PS-1 CPU is rated at 3 GHz and those +for PS-4 are rated at 6 GHz (somewhat conservative since 3 GHz +machines are already available). Tables~\ref{NP2} and \ref{NP4} show +the load and resulting number of nodes for both Phase 2 and Phase 4 +for both the PS-1 and PS-4 assumptions, using the I/O numbers for all +of the scenarios above. Note that in these discussions, we make the +idealized assumption that the computational and I/O portions of each +process are completely serial. As a result, the CPU is completely +used to perform the I/O during the I/O phase, avoiding any concern +about I/O load on the processor during analysis. + +\end{document} + +\section{Data Throughput Calculations} +\label{datarate} + +The following data throughput discussion is cut from an email by Josh +Hoblitt. + +We can do some quick 'back of the envelope' estimates to figure out +the worst case bandwidth needs. Let's assume that control messages +and db traffic is negligible compared to the image data. Let's also +assume that there isn't even trivial data locality optimization {\em +and} we have to reload all calibration data for every exposure. That +gives us: + +\begin{verbatim} + 4 fully populated focal planes + + 2bytes * ( 4096^2pixels * 1.125overclocks ) * 240otas = 72477573120b + + slew/settle/integrate/read cadence of 30s + + 72477573120b / 30s = 2415919104b/s + + exposure data stored as float + + 4bytes * 4096^2pixels * 240otas = 128849018880b + + 128849018880b / 30s = 4294967296b/s + + best/working stacked stored as float + + 4bytes * 4096^2pixels * 60otas = 32212254720 + + 32212254720b / 30s = 1073741824b/s + + debias, dark, flat, 2x fringe, and 2x sky calibration frames + + 7 * 4294967296b/s = 30064771072b/s + + Which gives us a grand total of: + + [phase 2] + 2415919104b/s summit -> disk (exposure) + + 2415919104b/s non-local disk -> memory (exposure) + + 30064771072b/s non-local disk -> memory (calibration) + + 4294967296b/s memory -> non-local disk (reduced) + ------------------ + 39,191,576,576b/s phase 2 total + + [phase 4] + 4294967296b/s non-local disk -> memory (reduced) + + 1073741824b/s non-local disk -> memory (best) + + 1073741824b/s non-local disk -> memory (working) + + 1073741824b/s memory -> non-local disk (diff) + + 1073741824b/s memory -> non-local disk (working) + ------------------ + 8,589,934,592b/s phase 4 total + + [total] + 39191576576b/s phase 2 total + + 8589934592b/s phase 4 total + ------------------ + 47,781,511,168b/s total is ~48Gb/s +\end{verbatim} + + 48Gb/s is 1/15th of the bandwidth available in a 65xx/SUP720. Given + these numbers I don't think that engineering effort to optimize for a + hierarchical network topology is justifiable. + + To address your concerns about the backplane numbers being accurate, + I can tell you that I have a significant amount of experience with + enterprise class switches from Cisco, Foundry, and HP. I've always + found the switching fabric specs to be accurate from all three of + these manufactures. In addition, as soon as I have equipment + available for use by Pan-Starrs I will generate my own bandwidth + tests. However it should be noted that as demonstrated above that + even our worst case bandwidth needs are rather modest compared to + COTS equipment that is available today. + +\end{document}