Index: /trunk/doc/design/specs.tex =================================================================== --- /trunk/doc/design/specs.tex (revision 417) +++ /trunk/doc/design/specs.tex (revision 418) @@ -1,3 +1,3 @@ -%%% $Id: specs.tex,v 1.3 2004-04-12 19:21:27 eugene Exp $ +%%% $Id: specs.tex,v 1.4 2004-04-13 02:18:48 eugene Exp $ \documentclass[panstarrs]{panstarrs} @@ -13,4 +13,6 @@ \docnumber{PSDC-430-005} +\setcounter{tocdepth}{4} % lowest level to be included in toc + \begin{document} \maketitle @@ -249,5 +251,5 @@ \item Object Database \item Metadata Database -\item Analysis Pipelines +\item Analysis Stages \item Controller \item Scheduler @@ -380,10 +382,10 @@ Third, stars in the general vicinity of the solar system fall in between these first two classes of objects. Their proper motion and -parallax response is significant enough ($>1\asec$ in 10 years) that +parallax response is significant enough ($>1$ arcsec in 10 years) that they are not well-described by an average location and a collection of offsets. These objects must be described by a distance and a proper motion vector. The PnA Database must be able to find and associate detections of objects for which either of the parallax or the proper -motion are substantial. +motion are substantial. Fourth, many detections, especially in their initial states, will not @@ -664,4 +666,6 @@ \subsubsection{Analysis Stages} +\paragraph{Overview} + We now consider the collection of analysis tasks which are performed by the IPP. Depending on the task, they may be performed on @@ -670,11 +674,11 @@ can be performed in parallel because, for example, the analysis of an OTA in one image does not depend on the results from another OTA. We -define the analysis pipelines to be the largest complete analysis task -which may be performed on a single data item. {\bf drop the word -'pipeline' and use something else?}. The data analysis pipelines are -divided into three categories, and further subdivided as follows: +define the term 'analysis stage' to refer to the largest complete +analysis task which may be performed on a single data item. The +analysis stages are divided into three categories, and further +subdivided as follows: \begin{enumerate} - \item Science Image Pipelines + \item Science Image Analysis Stages \begin{enumerate} \item Phase 1 : image processing preparation @@ -683,5 +687,5 @@ \item Phase 4 : image combination \end{enumerate} - \item Calibration Image Pipelines + \item Calibration Image Analysis Stages \begin{enumerate} \item Calibration 1 : basic master-detrend creation @@ -689,5 +693,5 @@ \item Calibration 3 : Flat-field correction image Creation \end{enumerate} - \item Reference Catalog Pipelines + \item Reference Catalog Analysis Stages \begin{enumerate} \item Astrometry reference catalog generation @@ -696,49 +700,134 @@ \end{enumerate} -Figure~\ref{pipelines} shows the flow of data between the various IPP -software systems and the different analysis tasks, each managed by the -controller. The thick lines represent the flow of pixel data, the +Figure~\ref{stages} shows the flow of data between the various IPP +software systems and the different analysis stages, each managed by +the Controller. The thick lines represent the flow of pixel data, the thin lines represent the flow of metadata and object data, and the -grey lines represent the flow of commands. {\bf All subsystem +grey lines represent the flow of commands. \tbd{All subsystem interactions, except that between the scheduler and controller, are in the form of updates to and queries from the databases}. The hatched systems represent external PanSTARRS systems (OATS, the Sky Server, the SAIC Object Database, the Moving/Transient Object Pipeline, and -other Client Science Pipelines. +other Client Science Pipelines. + +The individual analysis stages can be accessed as a UNIX command-line +program. Each command represents the action of the stage on a single +quantum of data. These analysis stages are built of lower-level +C-functions wrapped in a higher-level programming language, +\tbd{Python}. + +\subparagraph{Science Image Pipelines} + +The IPP science image pipelines perform analyses on the night-sky +science images to extract the science data from these images. These +consist of: Phase 0, the night preparation stage; Phase 1, the image +processing preparation stage; Phase 2, the image reduction stage; +Phase 3, the exposure analysis stage; and Phase 4, the image +combination stage. These pipelines must process the images in a +timely manner so that the incoming data stream will not overload the +IPS. The decision to execute a specific pipeline for a specific +dataset is made by the Scheduler, which sends the infomation to the +Controller. The Controller executes the pipeline for the data on an +appropriate machine and monitors the success or failure of the job. + +\subparagraph{Calibration Image Pipelines} + +The IPP Calibration Image Pipelines perform the tasks needed to +generate high-quality calibration images from the input image +dataset. These operations may be performed on whatever timescales are +appropriate and necessary to maintain the quality and relevance of the +calibration images. There are four distinct types of calibration +image pipelines: the basic detrend creation pipeline, the photometric +correction image creation pipeline, the fringe pattern generation +pipeline, and the sky foreground pattern generation pipeline. + +\subparagraph{Reference Catalog Pipelines} + +The IPP reference catalog pipelines use the data in the IPP Internal +Database and the IPP Object Database to determined improved +astrometric and photometric calibration references. \begin{figure} \begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/pipelines.ps}} -\caption{ \label{pipelines} IPP System Overview} +\resizebox{8cm}{!}{\includegraphics{pics/stages.ps}} +\caption{ \label{stages} IPP System Overview} \end{center} \end{figure} -\paragraph{Phase 2 Concept} +\paragraph{Phase 1 : image processing preparation} + +The Phase 1 analysis stage is performed on each science FPA to +calculate basic astrometric \tbd{and photometric} data needed by the +later stages. Phase 1 must use the static (pre-determined) telescope +distortion model, combined with the guide star pixel and celestial +coordinates, to determine the correct telescope bore-site, field +rotation and magnification. The astrometric accurate required from +this analysis stage is 2 arcsec across the field, sufficient to match +the vast majority of reference stars with their detections. + +In some circumstances, science images may have no guide stars. This +may occur if the detectors are not run in OTA mode, especially for +short snapshot images. In such a circumstance, the Phase 1 stage must +perform extremely basic object detection, determining the detector +coordinates for stars which are not excessively saturated and which +are significantly above the background level. The threshold levels +for this object detection stage must be configurable. The object +extraction must be performed in less than 3 seconds. + +In order for astrometry of an image to succeed, it is necessary that +approximate image coordinates be known. The Phase 1 analysis must be +able to succeed despite initial coordinate errors as large as 5 times +the field width. However, the search process must attempt the near +matches first in the assumption that the given coordinates are +accurate. + +A table of the overlaps between the science image to be processed and +the static sky images must be constructed. This table will be used to +guide the processing of the static sky in Phase 4. The overlaps must +be generously calculated so that small errors in astrometry at Phase 1 +will not cause any valid static sky / science image pairs to be +missed. It is acceptable for a small number of invalid overlaps to be +identified as these will be excluded in Phase 4. + +It is not unusual that an image be obtained with invalid coordinates +or without any valid stars. For example, the telescope control system +may make an error an report the wrong time or coordinates. Or, the +image may be obtained in exceptionally poor conditions with no +detected stars. Phase 1 must fail gracefully in these conditions, +reporting an appropriate error. Such images must be identified for +possible human intervention, or future follow-up after metadata +repairs are made. + +\paragraph{Phase 2 : image reduction} Phase~2 processing within the Pan-STARRS image processing pipeline is -the de-trend stage, where the images from the detector are processed -to remove instrumental signatures. The following operations need to -occur within Phase~2 processing: +the detrend stage, where the images from the detector are processed to +remove instrumental signatures. In addition, basic object detection +is performed along with improved astrometric and photometric +calibration. The following operations need to occur within Phase~2 +processing: + \begin{enumerate} -\item Convolve de-trend images with the OT kernel; -\item Flag bad and saturated pixels; -\item Bias correction via overscan subtraction; -\item Trim object image to remove overscan and edges corrupted by OT; -\item Correct for non-linearity; -\item Flat-field correction; -\item Sky subtraction; -\item Identify CRs; -\item Find objects in the image; and +\item Convolve detrend images with the OT kernel, if available +\item Flag bad and saturated pixels +\item Bias correction via overscan subtraction +\item Trim object image to remove overscan and edges corrupted by OT +\item Correct for non-linearity +\item Flat-field correction +\item Sky subtraction +\item Identify CRs +\item Find objects in the image \item Make postage stamps of bright objects. \end{enumerate} -These operations are each explained below. - -\paragraph{Convolve de-trend images with the OT kernel} - -De-trend images must be convolved by the OT kernel, so that -they accurately represent the de-trend images appropriate for -the object images, which have been shifted using OT. - -\paragraph{Flag bad and saturated pixels} + +\subparagraph{Convolve detrend images with the OT kernel} + +Detrend images must be convolved by the OT kernel, so that +they accurately represent the detrend images appropriate for +the object images, which have been shifted using OT. The detrend +images which must be convolved include: the flat-field and the +high-spatial-frequency fringe images. + +\subparagraph{Flag bad and saturated pixels} A static bad pixel mask needs to be used to identify pixels which are @@ -750,14 +839,31 @@ Pixels saturated in the A/D converter should also be masked, and this area should be grown by an additional pixel to mask excess charge -spillover. - -\paragraph{Bias correction via overscan subtraction} - -The overscan must be averaged (either in bulk, or individually by -rows) or fit with a polynomial, and the result subtracted from the -image. Overscan rows with a standard deviation which exceeds a -given threshold should be masked. - -\paragraph{Trim object image} +spillover. + +The bad pixel mask must be carried with the science images. Different +bits must be set to identify different reasons for masking the pixel. + +\subparagraph{Bias correction via overscan subtraction} + +The image bias must be subtracted. Since different detectors behave in +different ways, several options for modelling the bias must be +available. The bias must be measured from the image overscan region. +The bias subtraction method must be capable of applying a single +constant to the complete image, or to represent the bias as a function +which varies along the overscan. The function to be used must include +a spline or a chebychev polynomial derived from the data values along +the overscan. The values used to determine both the single constant +or the inputs to the spline and polynomial fits must be derived from +groups of pixels on the basis of one of several statistics, including +the sample and robust mean, median, and modes. In the case of a +single constant, all of the overscan pixel values are used in the +calculation of this statistic. In the case of the 1D functional +representation, the input values to the fit should represent the +coordinate along the overscan, with the statistic derived from the +pixel in the perpedicular direction at each location. Sigma-clipping +on the input data values must be an option. \tbd{accuracy of the bias +subtraction?} + +\subparagraph{Trim object image} The overscan must be trimmed from the object image, along with @@ -765,23 +871,29 @@ operation. -\paragraph{Correct for non-linearity} - -The object image (after bias correction) must be corrected for the -effects of non-linearity through a polynomial fit. - -\paragraph{Flat-field correction} +\subparagraph{Correct for non-linearity} + +The object image (after bias correction) must be optionally corrected +for the effects of non-linearity through a provided polynomial fit to +the pixel data values. \tbd{what IPP component produces the +non-linear correction function?} + +\subparagraph{Flat-field correction} The object image (after bias correction and non-linearity correction) -must be corrected for sensitivity differences as a function of position, -through dividing by a flat field image. - - -\paragraph{Sky subtraction} - -The flux contribution of the sky (both continuum emission and the line -emission that causes fringing) must be subtracted from the -flat-fielded object image. - -\paragraph{Identify CRs} +must be corrected for sensitivity variations as a function of +position, dividing by a flat-field image. The flat-field images must +be appropriately normalized (see section \ref{mkcal}. \tbd{what +component selects the appropriate flat-field image? scheduler or +flat-field module?} The flat-fielded image must have a consistent +photometric zero-point across the chip, and across the full FPA, to +within 0.2\%. + +\subparagraph{Sky subtraction} + +The flux contribution of the sky (from both continuum emission and the +line emission that causes fringing) must be subtracted from the +flat-fielded object image. + +\subparagraph{Identify CRs} CRs should be identified, if possible on the basis of their morphology @@ -789,10 +901,14 @@ masked. The mask must be grown by an additional pixel. -\paragraph{Find objects in the image} +\subparagraph{Find objects in the image} Objects on the flat-fielded object image must be found, and general parameters, including the centre, magnitude and shape measured. -\paragraph{Postage Stamps} +\subparagraph{astrometry} + +\tbd{per-OTA astrometry to improve per-OTA parameters} + +\subparagraph{Postage Stamps} Objects on the flat-fielded object image falling within a specified @@ -800,4 +916,25 @@ accurate photometry and astrometry. +\paragraph{Phase 3} + +The Phase 3 analysis stage works with the results from a complete FPA +obtained during Phase 2 to improve the photometric and astrometric +calibrations. + +Phase 3 must use the objects detected in Phase 2, matched with an +appropriate reference catalog, to determine the image zero point and +zero-point variations across the field. If zero-point variations are +significant \tbd{level TBD}, the zero-point variations must be modeled +with an up-to 3rd order chebychev polynomial correction. The complete +FPA image must be categoriezed as photometric on the basis of the +zero-point consistency, the transparency compared with recent +long-term measurements in the filter, and with the external indicators +of photometricity. + +Phase 3 must use the objects detected in Phase 2, matched with an +appropriate reference catalog, to determine improvements to the +astrometric solutions. The distortion model appropriate to this image +must be determined. The resulting astrometric accuracy must be +\tbd{50 mas? 10 mas?} \paragraph{Phase 4 Concept} @@ -806,5 +943,5 @@ the final stage of processing. It operates on each sky cell that has overlapping imaging data from the exposure(s) being processed, and -produces the main output image data products of the pipeline --- the +produces the main output image data products of the stage --- the difference images and a deep static sky image --- along with the associated catalogues of static and variable sources. @@ -814,5 +951,5 @@ -\paragraph{Functionality} +\subparagraph{Functionality} Phase 4 must consist of the following elements: @@ -841,6 +978,4 @@ -\paragraph{Performance} - \subparagraph{Timing} @@ -877,6 +1012,53 @@ to an error upstream in the processing). +\subsubsection{Calibration Stage 1} + +The IPP must generate basic calibration images using the raw +flat-field, bias and dark images obtained by the telescope as the +input. The analysis of these images requires relatively simple +stacking of the input set of images. Outlier rejection, both of +complete input images as well as pixels within the input stack, must +be performed. In addition, each type of image requires an appropriate +normalization which may depend on the data levels in other detectors +in the input set. Each of these calibration stages must be able to +determine from the input stack if the relevant calibration image needs +to be updated and perform an initial test to see which input images +are consistent and valid. + +\paragraph{bias images} + +\paragraph{dark images} + +\paragraph{flat-field images} + +\subsubsection{Calibration Stage 2} + +\paragraph{mask images} + +\paragraph{fringe frames} + +\paragraph{low-k sky models} + +\subsubsection{Calibration Stage 3} + +Flat-field correction frame + +\subsubsection{Astrometry Reference Creation} + +\subsubsection{Photometry Reference Creation} \subsubsection{Modules} + +In order to encapsulation functionality, the analysis stages are +constructed of a sequence of steps. The analysis stages consist of a +\tbd{python} script which executes a sequence of C-level functions. +The C-level functions called by the \tbd{python} script are called +{\em modules} and represent basic data analysis operations. + +The required set of Pan-STARRS modules and their functionality is +specfied in the document `Pan-STARRS Image Processing Pipeline Modules +Supplementary Design Requirements' (PSDC-430-xxx), and details of +specific apgorithms are specfied in the document `Pan-STARRS Image +Processing Pipeline Algorithm Design Document' (PSDC-430-006). \subsubsection{PanSTARRS IPP Library} @@ -950,7 +1132,7 @@ \subsubsection{Overview} -This document discusses the likely range of the Pan-STARRS Image -Processing Pipeline (IPP) hardware requirements. The hardware -requirements addressed in this document consist of: +This section discusses the Pan-STARRS Image Processing Pipeline (IPP) +PS-1 hardware requirements. The hardware requirements addressed in +this section consist of: \begin{itemize} @@ -967,6 +1149,5 @@ certain period, the need to store calibration images for a longer period, and the need to store the static sky images. Of the various -analysis pipelines, and depending on the data organization as -discussed below, Phase 2 and Phase 4 present the most significant +analysis stages, Phase 2 and Phase 4 present the most significant demands in terms of data I/O throughput on the network. Phase 2 and Phase 4 also present the most significant CPU demands. In this @@ -979,31 +1160,25 @@ This document does not address the hardware requirements implied by -the Phase 0, 1, or 3 stages, nor the load required by the calibration -image creation stages. In the first instance, the operations are only -performed on the metadata and are extremely minimal both in terms of -data I/O and computation requirements. In the second case, the +Phase 1 or 3, nor the load required by the calibration or reference +catalog creation stages. In the first instance, the operations are +only performed on the metadata and are extremely minimal both in terms +of data I/O and computation requirements. In the second case, the processing is less time critical than the per-image processing and is -performed only infrequently (once per night to once per week or -month). This document also does not address any hardware requirements -introduced by the metadata manipulation. The software implementation -for metadata storage (RDBMS, FITS tables, etc) will have a very large -impact and will be evaluated along with the needed hardware at a later -date. - -\subsubsection{Scenarios} - -We will address the various hardware requirements by referring to a -set of data processing and data organization scenarios. The actual -hardware requirements will depend on design decisions which are not -yet available. It is possible to define the data organization in ways -which will minimize the hardware requirements, but which will increase -the software development effort. We will discuss both the worst-case -data organization scenario, which does not require significant -intelligence in the software systems, and the optimal data -organization scenario, which will require the software to track the -location of data products more carefully. In addition, this document -will address the data requirements of the complete Pan-STARRS pipeline -with 4 telescopes as well as the single-telescope Pan-STARRS-1 scenario -based on the Design Reference Mission [REF]. +performed only infrequently (once per night to once per week, month or +year). \tbd{The software implementation for metadata storage (RDBMS, +FITS tables, etc) will have a very large impact and will be evaluated +along with the needed hardware at a later date.} + +We will address the various hardware requirements by referring to an +assumed data processing and data organization scenario. The +organization of the data and certain aspects of the data processing +scheme have very large implications for the hardware requirements. In +this analysis, we assume that data types are chosen to minimize the +data volume and that the data is organized to minimize the I/O +bandwidth needs, as defined below. We address the data requirements +of the single-telescope Pan-STARRS-1 scenario based on the Design +Reference Mission \tbd{REF}. + +\subsubsection{Data Organization} The IPP hardware system must provide both data storage and @@ -1028,24 +1203,20 @@ and static sky processing and storage nodes (mostly Phase 4). Also shown are two switches used in this configuration; although it is -currently possible to buy a single switch which would have a -sufficient number of GigE ports for both sections of the PS-1 system, -such a two-switch organization may be needed for the full Pan-STARRS -system. In such a case, the interswitch communication must also meet -the required throughput needs. We discuss the hardware requirements -in the assumption that such an organization will be necessary. +currently possible to buy a single switch with sufficient number of +ports, this organization represents a minimal configuration for the +PS-1 IPP hardware. In such a case, the interswitch communication must +also meet the required throughput needs. We discuss the hardware +requirements in the assumption that such an organization will be +necessary. The way in which the images are distributed among the storage and compute nodes will largely determine the I/O bandwidth requirements. For data bandwidth requirements calculations, it is necessary to make -some assumptions about the data organization. For the purposes of -this document, we explore two extreme-case options: -\begin{itemize} -\item Random Data Distribution - OTA \& Sky data is randomly - distributed within the compute node of a given type (ie, OTA data is - randomly distributed among the OTA compute nodes). -\item Optimal Data Distribution - OTA \& Sky data is optimally - distributed to compute OTA/Sky nodes (OTA processing is always on a - machine with local OTA data). -\end{itemize} +some assumptions about the data organization. We make the assumption +that the OTA data is optimally distributed to the OTA nodes such that +the OTA processing is always on a machine with local OTA data. This +implies that all OTA data from a specific OTA are targetted to a +specific machine. (see below for discussion of data duplication). + A second factor which will have a significant impact on the I/O requirements is the image storage format for the processed and @@ -1053,51 +1224,14 @@ format or 16 bit integer format with appropriate scaling. In the former case, additional dynamic range is retained, while in the latter -case, we reduce the data volume by a factor of 2. While some may -argue that the higher dynamic range is necessary, arguments can be -made that the 16 bit range is sufficient. (In particular, the 16 bit -data provides a dynamic range far above the expected 1/1000 fractional -accuracy of the flat-field images). A related question is the number -of calibration images needed by the processing system. Since the -complete analysis is not yet defined, this number is difficult to -ascertain. However, we can make a range of assumptions which are -reasonable. We therefore adopt two data volume scenarios to explore -these possibilites: -\begin{itemize} -\item Standard Data Volume - 32 bit data for processed and calibration - images, average of 7 calibration frames per image. -\item Minimal Data Volume - 16 bit data for processed and calibration - images, average of 4 calibration frames per image. -\end{itemize} -In the discussion that follows, we explore the hardware requirements -implied by the collection of four combinations of these two sets of -scenario options. - -\begin{table} -\begin{center} -\caption{Hardware Throughput Tests \label{existing-hardware}} -\begin{tabular}{lrrrr} -\hline -\hline -Test & where \& when & model & result \\ -\hline -node I/O & CFHT 11/2002 & Intel 1000 Gigabit & 35 - 40 MB/s sustained \\ -node I/O & CFHT 2/2004 & Intel 1000 Gigabit & 65 - 70 MB/s sustained \\ -RAID write & CFHT 2/2004 & 3ware RAID cntl + IDE & 110 MB/s sustained \\ -Switch Load & VeriTest & Cisco & 3 GB/s (for 32 ports) \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Existing Hardware Throughput} - -We have collected a few representative tests of various pieces of -modern hardware to give a reference for the throughput capabilities. -A number of hardware configurations have been tested at CFHT for the -Elixir project, and we include here their recent reported hardware -RAID-5 I/O speeds and GigE card speeds. We also have included data -from VeriTest studies of Cisco switch throughput, commissioned by -Cisco for a 32 port GigE switch. These tests are summarized in -Table~\ref{existing-hardware}. +case, we reduce the data volume by a factor of 2. Since the science +requirements for PS-1 do not specify a need for dynamic range greater +than 16 bits, we assume all images are stored as 16 bit data. + +A third determining factor is the number of calibration images needed +by the processing system. Since the complete analysis is not yet +defined, this number is difficult to ascertain. However, we can make +a reasonable guess at the total number for scaling purposes. We +assume that each frame requires a total of 4 calibration frames on +average \begin{table}[b] @@ -1107,16 +1241,11 @@ \hline \hline - & Standard / PS-4 - & Standard / PS-1 - & Minimal / PS-4 - & Minimal / PS-1 \\ -\hline -Raw data & 300 TB & 75 TB & 300 TB & 75 TB \\ -static sky & 512 TB & 64 TB & 256 TB & 32 TB \\ -calibration frames & 175 TB & 18 TB & 17 TB & 5 TB \\ -metadata db & 2 TB & 2 TB & 0.2 TB & 0.2 TB \\ -object db & 60 TB & 4 TB & 60 TB & 4 TB \\ -\hline -totals & 1050 TB & 163 TB & 633 TB & 116 TB \\ +Raw data & 200 TB \\ +static sky & 256 TB \\ +calibration frames & 5 TB \\ +metadata db & 0.3 TB \\ +object db & 4 TB \\ +\hline +total & 116 TB \\ \hline \end{tabular} @@ -1130,8 +1259,6 @@ calibration images, the metadata database, and the object database. We discuss each of these data items and their impact on the data -storage requirements for the IPP, and identify the impact of the -minimal vs standard data storage requirements as well as the -requirements specifically for PS-1. Table~\ref{storage} summarizes -the data storage requirements in the different scenarios. +storage requirements for the IPP for PS-1. Table~\ref{storage} +summarizes the data storage requirements in the different scenarios. \paragraph{Raw Data Storage} @@ -1140,34 +1267,26 @@ science images and calibration images. The night-time science images consist of 1Gpix per image, or 2GB in raw format. At nominal cadence, -the 4 telescopes can obtain images at a sustained rate of 1 image per -30 seconds per telescope for the entire night of 10 hours (36000 -minutes). A total of 100 calibration images per night would be a -substantial overestimate of the typical expectation. Combining these -numbers, we can expect to receive a total of 1300 image per telescope -per night, 5200 image total, or 10.4 TB of data per night. The total -data storage requirements for the raw data are governed by the number -of nights' worth of data we are required to keep online. A reasonable -number is one month to allow a full moon's cycle. Thus, for raw image -storage, we require a total of 300 TB data storage. For PS-1, this -number is simply scaled down by a factor of 4. The choice of the -minimal data volume does not affect these numbers because the raw data -is already stored with 16 bit pixels. ({\bf note: the PS-1 design -reference may now require storage of the entire first year of data, -calculated to be 200 TB}). +the PS-1 telescope can obtain images at a sustained rate of 1 image +per 30 seconds for the entire night of 10 hours (36000 seconds). A +total of 100 calibration images per night would be a substantial +overestimate of the typical expectation. Combining these numbers, we +can expect to receive a total of 1300 images, or 2.6 TB of data per +night. The total data storage requirements for the raw data are +governed by the number of nights' worth of data we are required to +keep online. \tbd{for the first year, we are required to keep all +images from the PnA and IPV surveys. This amounts to a total of 200 +TB of data}. \paragraph{Static Sky Data Storage} The static sky is represented by images with 0.2 arcsec per pixel. -There will be one summed image and one weight image for each of the 6 -filters, each stored in floating point format. At this resolution, -there are 324 Mpix per square degree, and we will observe a potential -total area of 30,000 square degrees. Allowing for 10\% overage for -overlapping tiling, we require a total of 10.7 Gpix to cover the sky -once, or a total of $\sim 512$ TB for the static sky images. In the -minimal data volume scenario, this value is reduced by a factor of 2, -while in PS-1, the reduction is a factor of roughly 8 because we only -intend to store the static sky for the ecliptic plane survey and the -small IPP verification program ({\bf note: this last point is no -longer valid - the PS-1 static sky may require the entire 3pi}). +There will be one summed image and one weight image for each of the +\tbd{6} filters, each stored with 16 bits of resolution, for a total +of 24 bytes per sky pixel. At this resolution, there are 324 Mpix per +square degree, and we will observe a potential total area of 30,000 +square degrees. Allowing for 10\% overage for overlapping tiling, we +require a total of 10.7 Tpix to cover the sky once, or a total of +$\sim 256$ TB to maintain a single image of the static sky in all 6 +filters. \paragraph{Calibration Frame Storage} @@ -1176,14 +1295,14 @@ and mask images, along with one flat, one flat-correction, and multiple sky/fringe library frames per filter. In fact, not all types -are needed at all stages. For the standard data volume, we assume an -average of 7 calibration frames per image and filter. This results in -a total of 42 master calibration image per telescope. If we intend to -keep all master calibration frames for the project lifetime, and -generate a new master on a weekly basis (a reasonable time-scale), -then we can expect to require a total of 175 TB of calibration image -by the end of the 5 year lifetime of the project. For the case of -PS-1, the time period is only 2 years, and there is only 1 telescope, -resulting in a factor of 10 reduction in the volume. For the minimal -data case, we reduce the volume by another factor of 3.5. We also note +are needed at all stages. It is very likely that we will not require +bias or dark images, and mask images may be represented by a single +byte per pixel. Nonetheless, it is necessary for us to generate and +store all master calibration frames at least until we prove that they +are not needed. We assume a total of 21 calibration images are +necessary (one flat, fringe, and sky per filter, along with a bias, +dark, and mask). If we intend to keep all master calibration frames +for the project lifetime, and generate a new master on a weekly basis +(a reasonable time-scale), then we can expect to require a total of 5 +TB of calibration image by the end of the 2 years of PS-1. We note that this is likely to be a drastic overestimate as we are unlikely to need to regenerate all master calibration frames on a weekly @@ -1197,13 +1316,10 @@ data. The environmental data consists of measurements on a regular cadence, roughly 1 per minute, of a variety of parameters. We suggest -an expected of 1kB per entry, for a total of 2.6 GB over the lifetime -of the project. PS-1 will represent a smaller amount of data per -minute, and also a factor of 2.5 fewer minutes. We suggest PS-1 may -have a total environmental metadata set smaller by a factor of 5. The -additional systems, such as the DIMM, SkyProbe, NIR Sky Camera, and -the LRProbe will have higher data requirements, but should be -considered as separate, self-contained systems. Their data products -are distilled to a limited number of parameters per minute which are -included in the 1kB given above. Furthermore, items such as +an expected of 1kB per entry, for a total of 1 GB over the two-year +term of PS-1. The additional systems, such as the DIMM, SkyProbe, NIR +Sky Camera, and the LRProbe will have higher data requirements, but +should be considered as separate, self-contained systems. Their data +products are distilled to a limited number of parameters per minute +which are included in the 1kB given above. Furthermore, items such as guide-star history, if saved, will be saved with the image data and represents only a small fraction of the total image data volume. Some @@ -1213,18 +1329,14 @@ excluded from this analysis. -The image metadata consists of values associated with the FPA (4), the -OTAs (240), and the Cells (15360). Aside from the guide star history, +The image metadata consists of values associated with the FPA (1), the +OTAs (64), and the Cells (4096). Aside from the guide star history, the total data requirements for each of these entries will be scaled by the number of bytes required for the metadata from each data level. Clearly, if the Cell entry is allowed to be large, it will dominate -the total Metadata data volume. If we suggest an expected number of -64 bytes per Cell, 256 B per OTA, and 1k per FPA, we find a total -metadata volume per exposure of roughly 1 MB, completely dominated by -the Cell metadata. With the exposure rates above, we find a total of -metadata volume of 1.8 TB over the lifetime of the project. For PS-1, -the total volume is reduced by a factor of 2.5 (for the shorter -lifetime) and another factor of 4 (for the lone telescope). Neither -data quantity is affected by the minimal vs standard data volume -choice. +the total Metadata data volume. We suggest an expected number of 64 +bytes per Cell, 256 B per OTA, and 1k per FPA, yielding a total +metadata volume per exposure of roughly 0.3 MB, completely dominated +by the Cell metadata. With the exposure rates above, we find a total +of metadata volume of 0.3 TB over the two-year term of PS-1. \paragraph{Object Database Storage} @@ -1235,15 +1347,11 @@ of object detections) and the number of object parameters which are measured. We can make very rough estimates that the total number of -detections over the 5 year lifetime of the project may be in the -vicinity of $5\times10^{11}$. We can conservatively estimate the -number of bytes needed to represent each detection as 128 B, resulting -in a total data storage for the object detections of 60 TB. However, -this number depends strongly on the timescale for which the IPP is -required to maintain all object detections, and may potentially be -significantly reduced. For the case of PS-1, the total number of -detections is likely to be reduced by a factor of 4 for the number of -telescopes, and potentially another significant factor ($\sim 4?$) by -limiting the depth of object detections. Again, the minimal data -volume scenario is irrelevant to the object database volume. +detections over the 2 year lifetime of the project may be in the +vicinity of $10^{11}$. We can conservatively estimate the number of +bytes needed to represent each detection as 128 B, resulting in a +total data storage for the object detections of 12 TB. However, this +number depends strongly on the timescale for which the IPP is required +to maintain all object detections, and may potentially be +significantly reduced. \subsubsection{CPU Requirements} @@ -1252,5 +1360,5 @@ because they must keep up with the image delivery rate of 1 per 30 seconds. We have performed benchmarks of a demonstration version for -both the Phase 2 and Phase 4 analyses. +both the Phase 2 and Phase 4 analyses. For the Phase 2, a substantial fraction of the processing time is @@ -1277,5 +1385,5 @@ full OTA, including the FFTs used for smoothing. We can therefore assume a total of 50 GHz-sec per OTA for the Phase 2 processing. This -converts to a total of 12000 GHz-sec for a complete major frame. +converts to a total of 12800 GHz-sec for a complete major frame. For Phase 4, the main computational tasks are combining the multiple @@ -1288,47 +1396,41 @@ equivalent to 7800 GHz-sec for a major frame. -For PS-1, the data processing will clearly require a smaller amount of -computational resources because of the lower image rate. However, the -total number of GHz-sec required for the complete analysis of 4 input -images and the combination with the static sky will remain -more-or-less the same. Some reduction in the load may be gained by -reducing the complexity and depth of analysis for PS-1. Depending on -the details and depth of the analysis, we may reduce the computational -load by a factor of 2. +For PS-1, the typical time for a major frame is $4 \times 30$ seconds. +Some reduction in the load may be gained by reducing the complexity +and depth of analysis for PS-1. Depending on the details and depth of +the analysis, we may reduce the computational load by a factor of 2. \begin{table} \begin{center} -\caption{Data Scenarios (MB per OTA or Sky-cell) \label{scenarios}} +\caption{Data I/O (MB per OTA or Sky-cell) \label{scenarios}} \begin{tabular}{lrrrr} \hline \hline - & Random / Standard & Random / Minimal & Optimal / Standard & Optimal / Minimal \\ -\hline -{\em Phase 2 input} & & & & \\ -from summit & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB \\ -input image & 32 MB & 32 MB & {\bf 32 MB} & {\bf 32 MB} \\ -calibration & $7 \times 64$ MB & $4 \times 32$ MB & {\bf 7 $\times$ 64 MB} & {\bf 4 $\times$ 32 MB} \\ -mask image & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ -\hline -network I/O: & 560 MB & 232 MB & 64 MB & 64 MB \\ -disk I/O: & (560 MB) & (232 MB) & 496 MB & 168 MB \\ - & & & & \\ -{\em Phase 2 output} & & & & \\ -output image & 64 MB & 32 MB & {\bf 64 MB} & {\bf 32 MB} \\ -output mask & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ -image to P4 & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB \\ -mask to P4 & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB \\ -\hline -network I/O: & 200 MB & 100 MB & 120 MB & 60 MB \\ -disk I/O: & (80 MB) & (40 MB) & 80 MB & 40 MB \\ - & & & & \\ -{\em Phase 4} & & & & \\ -input images & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & & \\ -input masks & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & & \\ -static sky & 64 MB & 64 MB & & \\ -static weight & 64 MB & 32 MB & & \\ -\hline -input: & 608 MB & 336 MB & & \\ -output: & 192 MB & 128 MB & & \\ +{\em Phase 2 input} \\ +from summit & $2 \times 32$ MB \\ +input image & {\bf 32 MB} \\ +calibration & {\bf 4 $\times$ 32 MB} \\ +mask image & {\bf 8 MB} \\ +\hline +network I/O: & 64 MB \\ +disk I/O: & 176 MB \\ + & \\ +{\em Phase 2 output} \\ +output image & {\bf 32 MB} \\ +output mask & {\bf 8 MB} \\ +image to P4 & $1.5 \times 32$ MB \\ +mask to P4 & $1.5 \times 8$ MB \\ +\hline +network I/O: & 60 MB \\ +disk I/O: & 40 MB \\ + & \\ +{\em Phase 4} & \\ +input images & $1.5 \times 4 \times 32$ MB \\ +input masks & $1.5 \times 4 \times 8$ MB \\ +static sky & 32 MB \\ +static weight & 32 MB \\ +\hline +input: & 304 MB \\ +output: & 96 MB \\ \hline \multicolumn{5}{l}{\em Bold-faced entries are access to local-disk} \\ @@ -1342,57 +1444,25 @@ Data I/O per node is defined as the number of bytes per second passed through the node's network adapter. The data throughput for each node -depends strongly on the scenarios identified above. In this section, -we identify the data which is passed between nodes for each of the -different scenarios. Table~\ref{scenarios} lists the per-node data -I/O for the four scenarios. - -For PS-4, there are only 30 seconds of compute time allowed for each -of the Phase 2 and Phase 4 analyses. We use the data I/O volumes and -some assumptions about expected network and disk bandwidth to estimate -the I/O and processing timeline for the four scenarios. From this -analysis, we can judge the total CPU requirements in terms of GHz, not -just GHz-sec. We have assumed that GigE network adapters are capable -of delivering data at 50MB/sec sustained and that a disk RAID can -deliver sustained 100 MB/sec reads and writes. These numbers are -conservative estimates based on recent tests discussed above. Using +depends strongly on the how the data is organized and processed. In +this section, we identify the data which is passed between nodes for +the two stages of the science analysis process. Table~\ref{scenarios} +lists the per-node data I/O for the analysis stages. + +For PS-1, there are 120 seconds of compute time allowed for each of +the Phase 2 and Phase 4 analyses for the collection of four images +which makes up a cannonical major frame. We use the data I/O volumes +and some assumptions about expected network and disk bandwidth to +estimate the I/O and processing timeline for the four scenarios. From +this analysis, we can judge the total CPU requirements in terms of +GHz, not just GHz-sec. We have assumed that GigE network adapters are +capable of delivering data at 50MB/sec sustained and that a disk RAID +can deliver sustained 100 MB/sec reads and writes. These numbers are +conservative estimates based on recent tests discussed below. Using these assumptions, Table~\ref{throughput} lists the time allocations -for the complete set of scenarios for the case of PS-4. - -\paragraph{Random / Standard Data Scenario} - -In the Random Data Distribution scenario, there is a single CPU -allocated to each OTA in the OTA farm and a single CPU for each Sky -cell process. The OTA data are stored across random machines in the -OTA farm, with the result that every Phase 2 processing requires -network access to the data. For each science OTA image which is -observed, each OTA node will read from the network a total of 560 MB -(the 2 raw images for data storage and the 7 calibration frames, along -with one mask and one raw input image) and write a total of 200 MB -(one processed image and the mask along with the 1.5 processed images -and masks for the Phase 4 analysis). Given the assumption of 50 MB/s -from the network adapter, the total data volume implies an I/O period -of 15.2 seconds. Note that the disk I/O is parallel with the network -I/O and substantially underfills the disk bandwidth. - -\paragraph{Random / Minimal Data Scenario} - -In the Random-Minimal, there is a single CPU allocated to each OTA in -the OTA farm and a single CPU for each Sky cell process, and the OTA -data are stored across random machines in the OTA farm. However, the -calibration and the processed science images are stored at 2 bytes per -pixel, the mask is set at 4 bits per pixel, and only 4 calibration -images are assumed. For each science OTA image which is observed, -each OTA node will read from the network a total of 232 MB (the 2 raw -images for data storage and the 4 calibration frames, along with one -mask and one raw input image) and write a total of 100 MB (one -processed image and the mask along with the 1.5 processed images for -the Phase 4 analysis). Given the assumption of 50 MB/s from the -network adapter, the total data volume implies an I/O period of 6.6 -seconds. Again, note that the disk I/O is parallel with the network -I/O and substantially underfills the disk bandwidth. - -\paragraph{Optimal / Standard Data Scenario} - -In the Optimal Data Distribution scenario, there is a single CPU +for the processing stages. + +\paragraph{Phase 2 Node I/O Requirements} + +In the assumed data distribution scenario, there is a single CPU allocated to each OTA in the OTA farm and a single CPU for each Sky cell process. In addition, all data for the specified OTA are stored @@ -1400,26 +1470,17 @@ result that all Phase 2 I/O is made to a local disk. For each science OTA image which is observed, each OTA node will read from the network -a total of 2 raw images (one for the original image, one for the -backup copy) and write an average of roughly 1.5 processed images and -masks to the Phase 4 machines for a total of 184 MB of network I/O. -During the processing stage, the OTA node will read from disk a total -of 496 MB (7 calibration frames at 64 MB each, one 16 MB mask, and one -raw science image at 32 MB) and write a total of 80 MB (one processed -image at 64 MB and one mask at 8 MB). Given the assumptions for the +a total of 2 raw images (one for the original image, one for a backup +copy) and write an average of roughly 1.5 processed images and masks +to the Phase 4 machines for a total of 124 MB of network I/O. During +the processing stage, the OTA node will read from disk a total of 176 +MB (4 calibration frames at 32 MB each, one 16 MB mask, and one raw +science image at 32 MB) and write a total of 40 MB (one processed +image at 32 MB and one mask at 8 MB). Given the assumptions for the network and disk bandwidths (50 MB/s and 100 MB/s respectively), the -data volumes imply a total I/O period of 9.5 seconds. In this +data volumes imply a total I/O period of 4.6 seconds. In this instance, the network I/O is presumed to be sequential with the disk I/O. -\paragraph{Optimal / Minimal Data Scenario} - -In the Optimal / Minimal Scenario, the minimal data sizes are used -with the optimal data distribution scheme. In this case, we reduce -the disk I/O volume to 168 read and 40 MB write, and the network -traffic to 124 MB. Given the assumptions for the network and disk -bandwidths, the data volumes imply a total I/O period of 4.6 seconds. -Again, the network I/O is presumed to be sequential with the disk I/O. - -\paragraph{Phase 4 Node I/O Requirements / Standard Data Volume} +\paragraph{Phase 4 Node I/O Requirements} Although it is easy to arrange the OTA data in such a way that the @@ -1439,43 +1500,27 @@ maximum read overhead is 50\% (need to read a 10x10 set of cells for an 8x8 input image). If the processing is performed on Static Sky -segments equivalent in size to the OTAs, the input data is 608 MB (384 -MB of processed science image, 96 MB of mask images, 64 MB of static -sky image and 64 MB of static sky weight map) while the output data is -192 MB (static sky, weight map, and difference image, each 64 MB). -Thus, we require a total of 800 MB network I/O. Given the network -bandwidth, this implies an I/O period of 16 seconds for Phase 4. - -\paragraph{Phase 4 Node I/O Requirements / Minimal Data Volume} - -In the minimal data volume scenario, the Phase 4 analysis volume is -significantly reduced. The total volume of input data is 336 MB (192 -MB of processed science image, 48 MB of input mask, 64 MB of static -sky image and 32 MB of static sky weight map) while the output data is -128 MB (64 MB static sky, 32 MB weight map, and 32 MB difference -image). Thus, we require a total of 464 MB network I/O, which implies -an I/O period of 9.3 seconds. +segments equivalent in size to the OTAs, the total volume of input +data per node is 304 MB (192 MB of processed science image, 48 MB of +input mask, 32 MB of static sky image and 32 MB of static sky weight +map) while the output data is 96 MB (32 MB static sky, 32 MB weight +map, and 32 MB difference image). Thus, we require a total of 400 MB +network I/O, which implies an I/O period of 8 seconds. \begin{table} \begin{center} -\caption{Data Throughput for 4 Scenarios \label{throughput}} +\caption{Data Throughput \label{throughput}} \begin{tabular}{lrrrr} \hline \hline -& -\multicolumn{1}{c}{Random / Standard} & -\multicolumn{1}{c}{Random / Minimal} & -\multicolumn{1}{c}{Optimal / Standard} & -\multicolumn{1}{c}{Optimal / Minimal} \\ -\hline -Phase 2 per-node network I/O & 15.2 s & 6.6 s & 3.7 s & 2.5 s \\ -Phase 2 per-node disk I/O (read) & (5.6 s) & (2.3 s) & 5.0 s & 1.7 s \\ -Phase 2 per-node disk I/O (write) & (0.8 s) & (0.4 s) & 0.8 s & 0.4 s \\ -Phase 2 CPU total & 14 s : 860 GHz & 23 s : 520 GHz & 20 s : 600 GHz & 25 s : 480 GHz \\ -Phase 4 per-node I/O & 16 s & 9.3 s & & \\ -Phase 4 CPU total & 14 s : 490 GHz & 20 s : 390 GHz & & \\ -Phase 2 switch load & 6.1 GB/s & 2.7 GB/s & 1.5 GB/s & 1.0 GB/s \\ -Phase 4 switch load & 0.8 GB/s & 0.5 GB/s & 0.8 GB/s & 0.5 GB/s \\ -Phase 2 to Phase 4 switch load & 1.1 GB/s & 0.6 GB/s & 1.1 GB/s & 0.6 GB/s \\ -Summit to Phase 2 switch load & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s \\ +Phase 2 per-node network I/O & 2.2 s \\ +Phase 2 per-node disk I/O (read) & 1.3 s \\ +Phase 2 per-node disk I/O (write) & 1.2 s \\ +Phase 2 CPU total & 25 s : 128 GHz \\ +Phase 4 per-node I/O & 8 s \\ +Phase 4 CPU total & 112 s : 70 GHz \\ +Phase 2 switch load & 264 MB/s \\ +Phase 4 switch load & 215 MB/s \\ +Phase 2 to Phase 4 switch load & 160 MB/s \\ +Summit to Phase 2 switch load & 70 MB/s \\ \hline \end{tabular} @@ -1486,81 +1531,27 @@ The switch I/O requirements are defined by the total number of bytes -per second serviced by the two switches in the system. For the -analysis of the Switch I/O requirements, the choice of data -distribution again has a major impact. We again test the four -scenarios discussed above: Random Data Distribution, Random / Minimal, -Optimal Data Distribution, and Optimal / Minimal. - -\paragraph{Random / Standard Data Scenario} - -In the Random Data Distribution scenario, each OTA node needs to read -a total of 560 MB from the network and write a total of 200 MB every -30 seconds. With 240 OTA nodes, this corresponds to a total bandwidth -of 6080 MB/sec, or 49 Gb/sec. Note that this includes the bandwidth -needed to copy data from the summit and make two copies on the OTA -machines, as well as the bandwidth to send the processed image -portions to the Phase 4 machines. The Phase 4 processing adds an -additional 320 MB of network I/O per Sky-Cell group, and there are -roughly 60-70 Sky-cells per exposure set. Thus the Phase 4 processing -adds an additional 750 MB/sec network bandwidth. In the architecture -defined in Figure NN, the Sky nodes and the OTA nodes are each -attached to separate switches. An additional bandwidth requirement is -derived by the need to exchange data between these switches in for -Phase 4. The total amount of data exchanged between these switches is -480 MB per Sky-cell, for a total bandwidth of 1120 MB/sec. In -addition, the connection to the summit is a single, separate line -which needs to support the bandwidth requirement of copying all intial -raw images. In our simple model, each raw image is copied twice, -accounting for a total of 15360 MB every 30 seconds, or a bandwidth -load of 512 MB/sec. (Note that this last is double the actual -bandwidth requirement to the summit: a dedicated local circular buffer -would reduce the need for the second copy to come directly from the -summit.) - -\paragraph{Random / Minimal Data Scenario} - -In the Random / Minimal Scenario, the data volumes are significantly -reduced. The total Phase 2 bandwidth contribution is 332 MB over 30 -seconds for 240 nodes (2656 MB/sec) and the residual Phase 4 bandwidth -load is 224 MB per Sky cell over 30 seconds (522 MB/sec). The -inter-switch communication is now 240 MB per sky cell over 30 seconds, -or 560 MB/sec. - -\paragraph{Optimal / Standard Data Scenario} - -In the Optimal Data Distribution, the Phase 2 network bandwidth is -reduced significantly to 184 MB per OTA node, for a total of -1.5GB/sec, while the Phase 4 network bandwidth remains unchanged at -750 MB/sec. The inter-switch communication also remains the same at -1.12 GB/sec. - -\paragraph{Optimal / Minimal Data Scenario} - -In the Optimal / Minimal Scenario, the total Phase 2 network bandwidth -drops to 124 MB per OTA node, for a total of 1.0GB/sec, while the -Phase 4 network bandwidth is 552 MB/sec. The inter-switch -communication remains the same as the Random/Minimal Scenario at 560 -MB/sec. - -\begin{table}[t] +per second serviced by the two switches in the system. + +The Phase 2 network I/O is 124 MB per OTA. With 64 OTAs per image, +and 30 seconds average between images, this implies a total of 264 +MB/s switch bandwidth. The Phase 4 network I/O is 400 MB per sky +cell. With 64 cells and 120 seconds between major frames, this is an +average switch bandwidth of 215 MB/s switch bandwidth. The total +switch-to-switch load is 304 MB per OTA, with an average timescale of +120 seconds. With 64 OTAs, this corresponds to 160 MB/s. The +summit-to-Phase 2 switch load is 70 MB/s. + +\begin{table} \begin{center} -\caption{\label{NP2} Phase 2 load per major frame (12000 GHz-sec)} +\caption{Hardware Throughput Tests \label{existing-hardware}} \begin{tabular}{lrrrr} \hline \hline -& Random/Standard -& Random/Minimal -& Optimal/Standard -& Optimal/Minimal \\ -\hline -network I/O (GB) & 182 & 80 & 44 & 30 \\ -PS-1 & & & & \\ - I/O (cpu-sec) & 3640 & 1600 & 880 & 600 \\ - CPU (cpu-sec) & 4000 & 4000 & 4000 & 4000 \\ - \# cpus & 64 & 47 & 41 & 38 \\ -PS-4 & & & & \\ - I/O (cpu-sec) & 1820 & 800 & 440 & 300 \\ - CPU (cpu-sec) & 2000 & 2000 & 2000 & 2000 \\ - \# cpus & 127 & 93 & 81 & 77 \\ +Test & where \& when & model & result \\ +\hline +node I/O & CFHT 11/2002 & Intel 1000 Gigabit & 35 - 40 MB/s sustained \\ +node I/O & CFHT 2/2004 & Intel 1000 Gigabit & 65 - 70 MB/s sustained \\ +RAID write & CFHT 2/2004 & 3ware RAID cntl + IDE & 110 MB/s sustained \\ +Switch Load & VeriTest & Cisco & 3 GB/s (for 32 ports) \\ \hline \end{tabular} @@ -1568,62 +1559,14 @@ \end{table} -\begin{table}[b] -\begin{center} -\caption{\label{NP4} Phase 4 load per major frame (7800 GHz-sec)} -\begin{tabular}{lrr} -\hline -\hline -& Standard -& Minimal \\ -\hline -network I/O (GB) & 48 & 28 \\ -PS-1 & & \\ - I/O (cpu-sec) & 960 & 557 \\ - CPU (cpu-sec) & 2600 & 2600 \\ - \# cpus & 30 & 26 \\ -PS-4 & & \\ - I/O (cpu-sec) & 480 & 278 \\ - CPU (cpu-sec) & 1300 & 1300 \\ - \# cpus & 59 & 53 \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Conclusions} - -Table~\ref{throughput} presents one way of analysing the hardware -requirements, making a specific set of assumptions about the number of -nodes for the two phases and the expected network and disk -bandwidths. The important conclusion in this analysis is the implied -number of GHz per processor, given the assumptions laid out. -Phase 2 is specified to have 240 OTA nodes, while Phase 4 is specified -to have roughly 60 static sky nodes. The range of Phase 2 CPU -requirements implies that each CPU needs to have speeds in the range -of 2.0 - 3.6 GHz, which sound very plausible for the year 2007, since -these apply to PS-4. - -Another way to represent this information is to use the total number -of MB I/O and the total number of GHz-sec required for the two stages, -confront these with an assumption for the bandwidth per network -adapter and an assumption for the CPU speed and use those numbers to -calculate the minimum number of nodes (CPUs) needed to sustain the -timing requirements. There are quite a few parameters and options to -choose from. We have assumed that for PS-1, the time between major -frames (4 images combined in Phase 4) is 120 seconds, and 30 seconds -for PS-4. We have also assumed that each CPU has one network adapter -associated with it, and use the numbers of 50 MB/sec for PS-1 era -network adapters and 100 MB/sec for the PS-4 network adapters (since -there has been some steady improvement in GigE hardware over the past -year). We have also assumed each PS-1 CPU is rated at 3 GHz and those -for PS-4 are rated at 6 GHz (somewhat conservative since 3 GHz -machines are already available). Tables~\ref{NP2} and \ref{NP4} show -the load and resulting number of nodes for both Phase 2 and Phase 4 -for both the PS-1 and PS-4 assumptions, using the I/O numbers for all -of the scenarios above. Note that in these discussions, we make the -idealized assumption that the computational and I/O portions of each -process are completely serial. As a result, the CPU is completely -used to perform the I/O during the I/O phase, avoiding any concern -about I/O load on the processor during analysis. +\subsubsection{Existing Hardware Throughput} + +We have collected a few representative tests of various pieces of +modern hardware to give a reference for the throughput capabilities. +A number of hardware configurations have been tested at CFHT for the +Elixir project, and we include here their recent reported hardware +RAID-5 I/O speeds and GigE card speeds. We also have included data +from VeriTest studies of Cisco switch throughput, commissioned by +Cisco for a 32 port GigE switch. These tests are summarized in +Table~\ref{existing-hardware}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%