Index: /trunk/doc/design/design.tex =================================================================== --- /trunk/doc/design/design.tex (revision 507) +++ /trunk/doc/design/design.tex (revision 508) @@ -1,3 +1,3 @@ -%%% $Id: design.tex,v 1.6 2004-04-23 01:15:45 price Exp $ +%%% $Id: design.tex,v 1.7 2004-04-23 02:44:14 price Exp $ \documentclass[panstarrs]{panstarrs} @@ -49,5 +49,14 @@ \subsection{System Overview} -\tbd{description of the Pan-STARRS System and PS-1.} +\PS{} is a survey telescope system being developed by the University +of Hawaii Institute for Astronomy (IfA), the Maui High Performance +Computing Center (MHPCC), Science Applications International +Corporation (SAIC), and Massachusetts Institute of Technology (MIT) +Lincoln Laboratory. The baseline system will consist of four 1.8m +telescopes, each with a 1 gigapixel camera capable of sustained image +rates of 2 per minute. A single initial test telescope (PS-1) will +be constructed on Haleakala and will see first light at the beginning +of 2006. The full four-telescope system (PS-4) will follow PS-1 by +roughly 2 years. \subsection{Document Overview} @@ -59,5 +68,5 @@ Open Issues and TBDs in this document are marked \tbd{in bold, red -with surrounding square brackets}. +type with surrounding square brackets}. \section{Referenced Documents} @@ -68,5 +77,5 @@ \DocumentsInternalSection -PSDC-430-xxx & PS-1 Design Reference Mission \\ \hline +PSDC-130-001 & PS-1 Design Reference Mission \\ \hline PSDC-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline PSDC-430-006 & Pan-STARRS IPP ADD \\ \hline @@ -79,15 +88,4 @@ \section{System Design Decisions} - -\PS{} is a survey telescope system being developed by the -University of Hawaii Institute for Astronomy (IfA), the Maui High -Performance Computing Center (MHPCC), Science Applications -International Corporation (SAIC), and Massachusetts Institute of -Technology (MIT) Lincoln Laboratory. The baseline system will consist -of four 1.8m telescopes, each with a 1 gigapixel camera capable of -sustained image rates of 2 per minute. An single initial test -telescope (PS-1) will be constructed on Haleakala and will see first -light at the beginning of 2006. The full four-telescope system (PS-4) -will follow PS-1 by roughly 2 years. Since \PS{} is a survey project, all data from the telescopes @@ -112,4 +110,6 @@ Processing System (MOPS), and potentially other client science pipelines. + +\subsection{System Overview} The \PS{} Image Processing Pipeline (IPP) consists of a @@ -152,5 +152,4 @@ requirements. -\subsection{System Overview} \subsection{System Architecture} \subsubsection{Architectural Components} @@ -191,40 +190,40 @@ \begin{center} \resizebox{8cm}{!}{\includegraphics{pics/overview}} -\caption{ \label{overview} IPP System Overview} +\caption{ \label{overview} IPP System Overview. \tbd{``Processing +Jobs'' should be renamed ``Analysis Pipelines''.} } \end{center} \end{figure} -\subsubsection{Analysis Stages} - -We now consider the collection of analysis tasks which are performed -by the IPP. Depending on the task, they may be performed on -individual images, collections of images, or on derived data products. -Because of the nature of the image data, many of the analysis tasks -can be performed in parallel because, for example, the analysis of an -OTA in one image does not depend on the results from another OTA. We -define the analysis pipelines to be the largest complete analysis task -which may be performed on a single data item. The data analysis -pipelines are divided into three categories, and further subdivided as -follows: - -\begin{enumerate} - \item Science Image Pipelines - \begin{enumerate} - \item Phase 1 : image processing preparation - \item Phase 2 : image reduction - \item Phase 3 : exposure analysis - \item Phase 4 : image combination - \end{enumerate} - \item Calibration Image Pipelines - \begin{enumerate} - \item Calibration 1 : basic master-detrend creation - \item Calibration 2 : Sky-model/fringe-mode generation - \item Calibration 3 : Flat-field correction image Creation - \end{enumerate} - \item Reference Catalog Pipelines - \begin{enumerate} +\subsubsection{Analysis Pipelines} + +We now consider the collection of IPP analysis pipelines. Depending +on the particular pipeline, they may be run on individual images, +collections of images, or on derived data products. Because of the +nature of the image data, many of the analysis pipelines can be run in +parallel because, for example, the analysis of a chip in one image +does not depend on the results from another chip. We define the +analysis pipelines to be the largest complete analysis task which may +be performed on a single data item. The data analysis tasks are +divided into three categories, and further subdivided as follows: + +\begin{enumerate} +\item Science Image Pipelines + \begin{enumerate} + \item Phase 1: image processing preparation + \item Phase 2: image reduction + \item Phase 3: exposure analysis + \item Phase 4: image combination + \end{enumerate} +\item Calibration Image Pipelines + \begin{enumerate} + \item Calibration 1: basic master-detrend creation + \item Calibration 2: Sky-model/fringe-mode generation + \item Calibration 3: Flat-field correction image Creation + \end{enumerate} +\item Reference Catalog Pipelines + \begin{enumerate} \item Astrometry reference catalog generation \item Photometry reference catalog generation - \end{enumerate} + \end{enumerate} \end{enumerate} @@ -241,5 +240,6 @@ \begin{center} \resizebox{8cm}{!}{\includegraphics{pics/pipelines}} -\caption{ \label{pipelines} IPP System Overview} +\caption{ \label{pipelines} IPP System Overview. \tbd{Small part at +top is missing.} } \end{center} \end{figure} @@ -248,6 +248,6 @@ The basic IPP hardware organization is shown in Figure~\ref{hardware}. -The overall hardware organization, with an OTA subcluster and a -Static-Sky subcluster, is largely chosen to reduce the I/O load during +The overall hardware organization, with a Detector subcluster and a +Static Sky subcluster, is largely chosen to reduce the I/O load during the pre-reduction analysis of the raw science images. In addition, we have specified distinct machines to maintain the object and metadata @@ -255,5 +255,5 @@ defined the details of these databases; it may be more appropriate depending on the eventual solutions to distribute these database -elements across the OTA and Static Sky subclusters. +elements across the Detector and Static Sky subclusters. \begin{figure} @@ -358,8 +358,8 @@ requirements, the IPS may maintain the pixel data distributed across the processor nodes in an organized fashion, i.e.\ associating -specific machines with specific OTAs. The IPS interacts with the IPP -Metadata Database to allow other systems or subsystems to identify the -available images meeting specified criteria. IPS specifications are -described in the IPS subsystem specification. +specific machines with specific detectors. The IPS interacts with the +IPP Metadata Database to allow other systems or subsystems to identify +the available images meeting specified criteria. IPS specifications +are described in the IPS subsystem specification. In addition to storing the pixel data, the IPS is responsible for @@ -572,28 +572,28 @@ \begin{tabular}{l} \hline -\multicolumn{1}{l}{\bf Metadata Tables} \\ -Weather \\ -SkyProbe \\ -LRProbe \\ -DIMM \\ -NIR \\ -Dome Status \\ -Telescope Status \\ -Raw FPAs \\ -Raw OTAs \\ -Raw Cells \\ -Observation Group \\ -OTA Guide Stars \\ -Science OTA stats \\ -Science Cell stats \\ -Science FPA stats -Sky-OTA overlaps \\ -Processed Sky-Cell stats \\ -Calibration 1 input OTA stats \\ -Calibration 1 output OTA stats \\ -Calibration 2 input OTA stats \\ -Calibration 2 output OTA stats \\ -Calibration 3 input stats \\ -Calibration 3 output stats \\ +\multicolumn{2}{l}{\bf Metadata Tables} \\ +Weather & Details on the weather, including internal temperatures. \\ +SkyProbe & Analysis of SkyProbe data. \\ +LRProbe & Analysis of LRProbe data. \\ +DIMM & Analysis of DIMM data. \\ +NIR & Analysis of NIR data. \\ +Dome Status & The status of the dome. \\ +Telescope Status & The status of the telescope. \\ +Raw FPAs & Details on raw FPA exposures. \\ +Raw Chips & Details on raw chips. \\ +Raw Cells & Details on raw cells. \\ +Observation Group & Details on a group of observations to be processed. \\ +Chip Guide Stars & Details on guide stars \\ +Science Chip stats & Details on processed chips. \\ +Science Cell stats & Details on processed cells. \\ +Science FPA stats & Details on processed FPAs. +Sky-Detector overlaps & List of overlaps between sky cells and detectors. \\ +Processed Sky-Cell stats & Details on sky cells. \\ +Calibration 1 input stats & Details on input images for Cal 1. \\ +Calibration 1 output stats & Details on output detrend images from Cal 1. \\ +Calibration 2 input stats & Details on input images for Cal 2. \\ +Calibration 2 output stats & Details on output detrend images from Cal 2. \\ +Calibration 3 input stats & Details on input images for Cal 3. \\ +Calibration 3 output stats & Details on output detrend images from Cal 3. \\ \hline \end{tabular} @@ -781,5 +781,5 @@ Reference catalog & The reference catalog that was used for the photometry. \\ PSF stats & Summary statistics of the PSF. \\ -OTA state & \tbd{The state of the OTA?} \\ +Chip state & \tbd{The state of the chip?} \\ Software versions & Versions of each of the modules used in the processing. \\ \hline @@ -822,5 +822,5 @@ \begin{tabular}{ll} \hline -\multicolumn{2}{l}{\bf Sky-Chip overlaps} \\ +\multicolumn{2}{l}{\bf Sky-Detector overlaps} \\ Chip ID & The identification number of the chip. \\ Sky Cell ID & The identification number of the sky cell. \\ @@ -849,5 +849,5 @@ \begin{tabular}{ll} \hline -\multicolumn{2}{l}{\bf Calibration 1 input Chip stats} \\ +\multicolumn{2}{l}{\bf Calibration 1 input stats} \\ Input ID & The input chip identification number. \\ Output ID & The identification number of the output detrend image. \\ @@ -861,5 +861,5 @@ \begin{tabular}{ll} \hline -\multicolumn{2}{l}{\bf Calibration 1 output Chip stats} \\ +\multicolumn{2}{l}{\bf Calibration 1 output stats} \\ Output ID & The identification number of the output detrend image. \\ Data type & The type of the detrend image (bias | dark | flat) \\ @@ -875,5 +875,5 @@ \begin{tabular}{ll} \hline -\multicolumn{2}{l}{\bf Calibration 2 input Chip stats} \\ +\multicolumn{2}{l}{\bf Calibration 2 input stats} \\ Input ID & The input chip identification number. \\ Output ID & The identification number of the output detrend image. \\ @@ -889,5 +889,5 @@ \begin{tabular}{ll} \hline -\multicolumn{2}{l}{\bf Calibration 2 output OTA stats } \\ +\multicolumn{2}{l}{\bf Calibration 2 output stats } \\ Output ID & The identification number of the output detrend image. \\ Data type & The type of the detrend image (bias | dark | flat) \\ @@ -903,4 +903,18 @@ \begin{tabular}{ll} \hline +\multicolumn{2}{l}{\bf Calibration 3 input stats} \\ +Input ID & The input chip identification number. \\ +Output ID & The identification number of the output detrend image. \\ +State & \tbd{State of the processing?} \\ +Accepted? & Is the detrend image of acceptable quality? \\ +Image stats & Assorted image statistics (mean flux, exposure time, airmass). \\ +Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ +Applied reduction & \tbd{Reduction method used?} \\ +Applied params & Parameters of reduction. \\ +\hline +\end{tabular} + +\begin{tabular}{ll} +\hline \multicolumn{1}{l}{\bf Calibration 3 output metadata } \\ Input images & Identification numbers of the input chips. \\ @@ -946,6 +960,6 @@ access to objects on the sky, including the access to the photometry associated with specific input images, moving objects associated with -specific OTA images. Detailed requirements for the IOD are described -in the IOD subsystem specification document xxx-xxx-xxxx. +specific chips. Detailed requirements for the IOD are described in +the IOD subsystem specification document xxx-xxx-xxxx. Reference Astrometry Catalogs: @@ -960,16 +974,16 @@ \begin{tabular}{l} \hline -\multicolumn{1}{l}{\bf Object DB Tables} \\ -Images \\ -Objects \\ -Detections \\ -NonDetections \\ -Filters \\ -Photcodes \\ -Bright Objects \\ -Region Tables \\ -Average Magnitudes \\ -USNO Objects \\ -Reference Objects \\ +\multicolumn{2}{l}{\bf Object DB Tables} \\ +Images & The images that have objects in the DB. \\ +Objects & The objects --- average properties of multiple detections of the same object. \\ +Detections & Detections of sources in an image. \\ +Non-Detections & Non-detections of objects in an image. \\ +Filters & Filters understood by the system. \\ +Photcodes & \tbd{Transformations between different photometric systems?} \\ +Bright Objects & \tbd{Links to postage stamp images of bright objects.} \\ +Region Tables & \tbd{???} \\ +Average Magnitudes & \tbd{How is this different from an `object'?} \\ +USNO Objects & Objects from the USNO database. \\ +Reference Objects & The reference catalogs for astrometry and photometry. \\ \hline \end{tabular} @@ -987,5 +1001,5 @@ IMD. The Controller must be able to manage more than a single processing thread to make maximum use of available processor -resources. Some analysis jobs, such as operations on the OTAs, must +resources. Some analysis jobs, such as operations on the chips, must be allocated preferentially to specified processors, while others must be distributed to the available machines in the cluster. @@ -1150,14 +1164,14 @@ The input to this analysis is the list of guide-star pixel centroids and their celestial coordinates as saved in the metadata database, as -well as the FPA and OTA organization and geometry, and the basic +well as the FPA and chip organization and geometry, and the basic optical distortion for the camera. For the sky-cell / detector-cell overlaps, the sky tiling scheme is required. The output consists of calculated astrometric parameters (linear -transformation + static distortion) for each of the FPA OTAs. On the -basis of this astrometry, the overlap between the OTAs and the +transformation + static distortion) for each of the FPA chips. On the +basis of this astrometry, the overlap between the detectors and the sky-cells is calculated. The output of this calculation is a list of -sky-cell / OTA links in a database table. This list of links can be -used by the later stages to initiate the analyses. +sky-cell / chip links in a database table. This list of links can be +used by the later stages to initiate the analyses. The phase 1 analysis is performed on an FPA basis to ensure that @@ -1489,8 +1503,8 @@ In the Phase 2 analysis, the astrometric solutions were determined -independently for each OTA. These solutions are limited by the +independently for each chip. These solutions are limited by the assumption of a static distortion and \tbd{by the accuracy of the astrometric reference}. In the phase 3 analysis, the astrometric -solutions of the N FPA images are improved by \tbd{???}. +solutions of the $N$ FPA images are improved by \tbd{???}. \tbd{what is the expected accuracy of the relative astrometric @@ -1502,8 +1516,8 @@ In the Phase 2 analysis, the background is determined based only on -the available sky in a single OTA image. However, the background +the available sky in a single chip. However, the background structures are normally correlated on the scale of the FPA, so an improved background solution can be determined by combining the -information from many OTA images. \tbd{is the background correlated +information from many chips. \tbd{is the background correlated between FPAs?} @@ -1517,5 +1531,5 @@ rejection. This combination requires the calculation of a set of PSF kernels to convert each of the input images to a single, common PSF. -These PSF kernels are determined from the per-OTA PSFs measured in +These PSF kernels are determined from the per-chip PSFs measured in Phase 2. @@ -1853,10 +1867,10 @@ demands in terms of data I/O throughput on the network. Phase 2 and Phase 4 also present the most significant CPU demands. In this -discusion, Phase 2 refers to the per-OTA image pre-processing in which -the instrumental signature is removed and a first pass object -detection is performed. Phase 4 refers to the multiple OTA -combination in which the pre-processed images are merged and combined, -in both addition and subtraction, with the static sky image, and up to -three object detection passes are performed. +discusion, Phase 2 refers to the per-chip pre-processing in which the +instrumental signature is removed and a first pass object detection is +performed. Phase 4 refers to the multiple chip combination in which +the pre-processed images are merged and combined, in both addition and +subtraction, with the static sky image, and up to three object +detection passes are performed. This document does not address the hardware requirements implied by @@ -1892,6 +1906,6 @@ computational resources. The IPP requires relativley large amounts of data storage space, primarily for the image data. Image data is -organized in two categories. First, there is the per-OTA data -- data -associated with specific OTAs, including the raw images, the +organized in two categories. First, there is the per-chip data -- +data associated with specific chips, including the raw images, the calibration images, and temporary processed images at various stages. Second, there is the data associated with the static sky imagery, @@ -1900,12 +1914,13 @@ provide both data storage and computational resources. The second assumption we make is that the data storage nodes are divided into two -classes: those which deal with the per-OTA data and those that provide -the static sky storage. In addition, we assume that the computational -tasks related to Phase 2 take place on the per-OTA storage nodes and -the Phase 4 computation takes place on the static sky storage nodes. +classes: those which deal with the per-chip data and those that +provide the static sky storage. In addition, we assume that the +computational tasks related to Phase 2 take place on the per-chip +storage nodes and the Phase 4 computation takes place on the static +sky storage nodes. Figure~\ref{hardware} shows our basic concept for the hardware organization for the IPP. This diagram shows the two types of compute -nodes: OTA-level processing and storage nodes (dominated by Phase 2) +nodes: chip-level processing and storage nodes (dominated by Phase 2) and static sky processing and storage nodes (mostly Phase 4). Also shown are two switches used in this configuration; although it is @@ -1923,10 +1938,10 @@ this document, we explore two extreme-case options: \begin{itemize} -\item Random Data Distribution - OTA \& Sky data is randomly - distributed within the compute node of a given type (ie, OTA data is - randomly distributed among the OTA compute nodes). -\item Optimal Data Distribution - OTA \& Sky data is optimally - distributed to compute OTA/Sky nodes (OTA processing is always on a - machine with local OTA data). +\item Random Data Distribution --- Detector \& Sky data is randomly + distributed within the compute node of a given type (ie, chip data + is randomly distributed among the detector compute nodes). +\item Optimal Data Distribution --- Detector \& Sky data is optimally + distributed to compute Detector/Sky nodes (chip processing is always + on a machine with local chip data). \end{itemize} A second factor which will have a significant impact on the I/O @@ -2096,17 +2111,17 @@ The image metadata consists of values associated with the FPA (4), the -OTAs (240), and the Cells (15360). Aside from the guide star history, -the total data requirements for each of these entries will be scaled -by the number of bytes required for the metadata from each data level. -Clearly, if the Cell entry is allowed to be large, it will dominate -the total Metadata data volume. If we suggest an expected number of -64 bytes per Cell, 256 B per OTA, and 1k per FPA, we find a total -metadata volume per exposure of roughly 1 MB, completely dominated by -the Cell metadata. With the exposure rates above, we find a total of -metadata volume of 1.8 TB over the lifetime of the project. For PS-1, -the total volume is reduced by a factor of 2.5 (for the shorter -lifetime) and another factor of 4 (for the lone telescope). Neither -data quantity is affected by the minimal vs standard data volume -choice. +chips (240), and the Cells (15360). Aside from the guide star +history, the total data requirements for each of these entries will be +scaled by the number of bytes required for the metadata from each data +level. Clearly, if the Cell entry is allowed to be large, it will +dominate the total Metadata data volume. If we suggest an expected +number of 64~bytes per Cell, 256~B per chips, and 1~kB per FPA, we find a +total metadata volume per exposure of roughly 1~MB, completely +dominated by the Cell metadata. With the exposure rates above, we +find a total of metadata volume of 1.8~TB over the lifetime of the +project. For PS-1, the total volume is reduced by a factor of 2.5 +(for the shorter lifetime) and another factor of 4 (for the lone +telescope). Neither data quantity is affected by the minimal vs +standard data volume choice. \paragraph{Object Database Storage} @@ -2142,13 +2157,13 @@ needed by the object detection, deblending, and analysis. Experiments with the FFTW package show that FFTs may be performed on Intel -processors at rates of approximately 0.25 GHz-sec / Mpix for data sets +processors at rates of approximately 0.25~GHz-sec / Mpix for data sets of order 1 Megapixel. The FFTs required for the Phase 2 analysis are performed on the 512$^2$ pixel cells, so these numbers may roughly be -scaled linearly to determine the total time required for OTA -processing. A single FFT on a full OTA, with 64 Cells, therefore -requires roughly 4 GHz-sec. For the full Phase 2 analysis, there are +scaled linearly to determine the total time required for chip +processing. A single FFT on a full chip, with 64 cells, therefore +requires roughly 4~GHz-sec. For the full Phase 2 analysis, there are roughly 4 single direction FFTs required excluding those associated with object detection; thus the total processing time for these FFTs -is approximately 16 GHz-sec. The addtional analysis steps, excluding +is approximately 16~GHz-sec. The addtional analysis steps, excluding object detection and characterization, account for a small fraction of this compute time, which we estimate at 10\%. The object detection @@ -2156,17 +2171,17 @@ performed, and the number of measurements made per object. Typical analysis performed by the Sextractor routine, which performs a -substantial number of per-object analyses, requires 27 GHz-sec for a -full OTA, including the FFTs used for smoothing. We can therefore -assume a total of 50 GHz-sec per OTA for the Phase 2 processing. This -converts to a total of 12000 GHz-sec for a complete major frame. +substantial number of per-object analyses, requires 27~GHz-sec for a +full chip, including the FFTs used for smoothing. We can therefore +assume a total of 50~GHz-sec per chip for the Phase 2 processing. +This converts to a total of 12,000~GHz-sec for a complete major frame. For Phase 4, the main computational tasks are combining the multiple images, with cosmic-ray rejection, and performing the object detection tasks. Nick Kaiser has done tests of the Phase 4 image combine and -rejection stages, and finds a total processing time of roughly 96 -GHz-sec for a full stack of 4 OTA images. If we add in an additional -34 GHz-sec for detailed object detection and image differencing, we -find a conservative estimage of 130 GHz-sec for a 4-image OTA stack, -equivalent to 7800 GHz-sec for a major frame. +rejection stages, and finds a total processing time of roughly +96~GHz-sec for a full stack of 4 chips. If we add in an additional +34~GHz-sec for detailed object detection and image differencing, we +find a conservative estimage of 130~GHz-sec for a 4-image chip stack, +equivalent to 7800~GHz-sec for a major frame. For PS-1, the data processing will clearly require a smaller amount of @@ -2181,5 +2196,5 @@ \begin{table} \begin{center} -\caption{Data Scenarios (MB per OTA or Sky-cell) \label{scenarios}} +\caption{Data Scenarios (MB per Chip or Sky-cell) \label{scenarios}} \begin{tabular}{lrrrr} \hline @@ -2244,9 +2259,9 @@ In the Random Data Distribution scenario, there is a single CPU -allocated to each OTA in the OTA farm and a single CPU for each Sky -cell process. The OTA data are stored across random machines in the -OTA farm, with the result that every Phase 2 processing requires -network access to the data. For each science OTA image which is -observed, each OTA node will read from the network a total of 560 MB +allocated to each chip in the detector farm and a single CPU for each Sky +cell process. The chip data are stored across random machines in the +detector farm, with the result that every Phase 2 processing requires +network access to the data. For each science chip which is +observed, each detector node will read from the network a total of 560 MB (the 2 raw images for data storage and the 7 calibration frames, along with one mask and one raw input image) and write a total of 200 MB @@ -2259,15 +2274,15 @@ \paragraph{Random / Minimal Data Scenario} -In the Random-Minimal, there is a single CPU allocated to each OTA in -the OTA farm and a single CPU for each Sky cell process, and the OTA -data are stored across random machines in the OTA farm. However, the -calibration and the processed science images are stored at 2 bytes per -pixel, the mask is set at 4 bits per pixel, and only 4 calibration -images are assumed. For each science OTA image which is observed, -each OTA node will read from the network a total of 232 MB (the 2 raw -images for data storage and the 4 calibration frames, along with one -mask and one raw input image) and write a total of 100 MB (one -processed image and the mask along with the 1.5 processed images for -the Phase 4 analysis). Given the assumption of 50 MB/s from the +In the Random-Minimal, there is a single CPU allocated to each chip in +the detector farm and a single CPU for each Sky cell process, and the +chip data are stored across random machines in the detector farm. +However, the calibration and the processed science images are stored +at 2 bytes per pixel, the mask is set at 4 bits per pixel, and only 4 +calibration images are assumed. For each science chip which is +observed, each detector node will read from the network a total of 232 MB +(the 2 raw images for data storage and the 4 calibration frames, along +with one mask and one raw input image) and write a total of 100 MB +(one processed image and the mask along with the 1.5 processed images +for the Phase 4 analysis). Given the assumption of 50 MB/s from the network adapter, the total data volume implies an I/O period of 6.6 seconds. Again, note that the disk I/O is parallel with the network @@ -2277,20 +2292,20 @@ In the Optimal Data Distribution scenario, there is a single CPU -allocated to each OTA in the OTA farm and a single CPU for each Sky -cell process. In addition, all data for the specified OTA are stored -on local disks attached to the same computer as the CPU, with the -result that all Phase 2 I/O is made to a local disk. For each science -OTA image which is observed, each OTA node will read from the network -a total of 2 raw images (one for the original image, one for the -backup copy) and write an average of roughly 1.5 processed images and -masks to the Phase 4 machines for a total of 184 MB of network I/O. -During the processing stage, the OTA node will read from disk a total -of 496 MB (7 calibration frames at 64 MB each, one 16 MB mask, and one -raw science image at 32 MB) and write a total of 80 MB (one processed -image at 64 MB and one mask at 8 MB). Given the assumptions for the -network and disk bandwidths (50 MB/s and 100 MB/s respectively), the -data volumes imply a total I/O period of 9.5 seconds. In this -instance, the network I/O is presumed to be sequential with the disk -I/O. +allocated to each chip in the detector farm and a single CPU for each +Sky cell process. In addition, all data for the specified chip are +stored on local disks attached to the same computer as the CPU, with +the result that all Phase 2 I/O is made to a local disk. For each +science chip which is observed, each detector node will read from the +network a total of 2 raw images (one for the original image, one for +the backup copy) and write an average of roughly 1.5 processed images +and masks to the Phase 4 machines for a total of 184 MB of network +I/O. During the processing stage, the detector node will read from +disk a total of 496 MB (7 calibration frames at 64 MB each, one 16 MB +mask, and one raw science image at 32 MB) and write a total of 80 MB +(one processed image at 64 MB and one mask at 8 MB). Given the +assumptions for the network and disk bandwidths (50 MB/s and 100 MB/s +respectively), the data volumes imply a total I/O period of 9.5 +seconds. In this instance, the network I/O is presumed to be +sequential with the disk I/O. \paragraph{Optimal / Minimal Data Scenario} @@ -2305,21 +2320,21 @@ \paragraph{Phase 4 Node I/O Requirements / Standard Data Volume} -Although it is easy to arrange the OTA data in such a way that the -majority of I/O is performed locally, it is not as easy to arrange +Although it is easy to arrange the detector data in such a way that +the majority of I/O is performed locally, it is not as easy to arrange this for the Static Sky data used by the Phase 4 analysis. We therefore make the assumption that the Phase 4 analysis will require -all input OTA data to be loaded across the network, as well as all -Static Sky data. This is somewhat of an overestimate as some of the -Static Sky data will be processed by machines with the data stored +all input detector data to be loaded across the network, as well as +all Static Sky data. This is somewhat of an overestimate as some of +the Static Sky data will be processed by machines with the data stored locally, and clever Static-Sky data organization schemes can enhance -this chance. +this chance. In the Phase 4 analysis, the images from the 4 separate telescopes are combined into a single image, confronted with the appropriate segment of the static sky, with output difference image and updated static sky -image. If we restrict input access to the individual OTA cells, the +image. If we restrict input access to the individual chip cells, the maximum read overhead is 50\% (need to read a 10x10 set of cells for an 8x8 input image). If the processing is performed on Static Sky -segments equivalent in size to the OTAs, the input data is 608 MB (384 +segments equivalent in size to the chips, the input data is 608 MB (384 MB of processed science image, 96 MB of mask images, 64 MB of static sky image and 64 MB of static sky weight map) while the output data is @@ -2376,15 +2391,15 @@ \paragraph{Random / Standard Data Scenario} -In the Random Data Distribution scenario, each OTA node needs to read -a total of 560 MB from the network and write a total of 200 MB every -30 seconds. With 240 OTA nodes, this corresponds to a total bandwidth -of 6080 MB/sec, or 49 Gb/sec. Note that this includes the bandwidth -needed to copy data from the summit and make two copies on the OTA -machines, as well as the bandwidth to send the processed image -portions to the Phase 4 machines. The Phase 4 processing adds an -additional 320 MB of network I/O per Sky-Cell group, and there are +In the Random Data Distribution scenario, each detector node needs to +read a total of 560 MB from the network and write a total of 200 MB +every 30 seconds. With 240 detector nodes, this corresponds to a +total bandwidth of 6080 MB/sec, or 49 Gb/sec. Note that this includes +the bandwidth needed to copy data from the summit and make two copies +on the detector machines, as well as the bandwidth to send the processed +image portions to the Phase 4 machines. The Phase 4 processing adds +an additional 320 MB of network I/O per Sky-Cell group, and there are roughly 60-70 Sky-cells per exposure set. Thus the Phase 4 processing adds an additional 750 MB/sec network bandwidth. In the architecture -defined in Figure \tbd{NN}, the Sky nodes and the OTA nodes are each +defined in Figure \tbd{NN}, the Sky nodes and the detector nodes are each attached to separate switches. An additional bandwidth requirement is derived by the need to exchange data between these switches in for @@ -2412,5 +2427,5 @@ In the Optimal Data Distribution, the Phase 2 network bandwidth is -reduced significantly to 184 MB per OTA node, for a total of +reduced significantly to 184 MB per detector node, for a total of 1.5GB/sec, while the Phase 4 network bandwidth remains unchanged at 750 MB/sec. The inter-switch communication also remains the same at @@ -2420,5 +2435,5 @@ In the Optimal / Minimal Scenario, the total Phase 2 network bandwidth -drops to 124 MB per OTA node, for a total of 1.0GB/sec, while the +drops to 124 MB per detector node, for a total of 1.0GB/sec, while the Phase 4 network bandwidth is 552 MB/sec. The inter-switch communication remains the same as the Random/Minimal Scenario at 560 @@ -2480,5 +2495,5 @@ bandwidths. The important conclusion in this analysis is the implied number of GHz per processor, given the assumptions laid out. -Phase 2 is specified to have 240 OTA nodes, while Phase 4 is specified +Phase 2 is specified to have 240 detector nodes, while Phase 4 is specified to have roughly 60 static sky nodes. The range of Phase 2 CPU requirements implies that each CPU needs to have speeds in the range @@ -2513,15 +2528,16 @@ \section{Notes} -\subsection{Cell vs OTA vs Mosaic vs Major Frame} - -There are several levels of input data pixel groups: Cell, OTA, -Mosaic, and Major Frame. It is necessary to make the association -between the data of one level and that of the next in a way that is -reliable and robust to missing elements. If a specific cell is -missing from an OTA, that information is known by the controller an -needs to be represented in the metadata. Similarly if an OTA is -missing from a mosaic camera, that information is also known and must -be carried though the metadata. A more difficult association is that -between the telescopes to define the major frame. Some possibilities: +\subsection{Cell vs Chip vs FPA vs Major Frame} + +There are several levels of input data pixel groups: Cell, Chip, Focal +Plane Array (FPA), and Major Frame. It is necessary to make the +association between the data of one level and that of the next in a +way that is reliable and robust to missing elements. If a specific +cell is missing from a chip, that information is known by the +controller an needs to be represented in the metadata. Similarly if a +chip is missing from a mosaic camera, that information is also known +and must be carried though the metadata. A more difficult association +is that between the telescopes to define the major frame. Some +possibilities: \begin{enumerate} @@ -2572,7 +2588,7 @@ Phase 4. -\subsection{Pending Sky-cell / OTA table} - -Define a pending sky-cell / OTA table to define the overlaps and to +\subsection{Pending Sky-cell / Detector table} + +Define a pending sky-cell / detector table to define the overlaps and to give something which the scheduler can query to decide when to initiate phase 4.