Changeset 508
- Timestamp:
- Apr 22, 2004, 4:44:14 PM (22 years ago)
- File:
-
- 1 edited
-
trunk/doc/design/design.tex (modified) (45 diffs)
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/design/design.tex
r507 r508 1 %%% $Id: design.tex,v 1. 6 2004-04-23 01:15:45price Exp $1 %%% $Id: design.tex,v 1.7 2004-04-23 02:44:14 price Exp $ 2 2 \documentclass[panstarrs]{panstarrs} 3 3 … … 49 49 \subsection{System Overview} 50 50 51 \tbd{description of the Pan-STARRS System and PS-1.} 51 \PS{} is a survey telescope system being developed by the University 52 of Hawaii Institute for Astronomy (IfA), the Maui High Performance 53 Computing Center (MHPCC), Science Applications International 54 Corporation (SAIC), and Massachusetts Institute of Technology (MIT) 55 Lincoln Laboratory. The baseline system will consist of four 1.8m 56 telescopes, each with a 1 gigapixel camera capable of sustained image 57 rates of 2 per minute. A single initial test telescope (PS-1) will 58 be constructed on Haleakala and will see first light at the beginning 59 of 2006. The full four-telescope system (PS-4) will follow PS-1 by 60 roughly 2 years. 52 61 53 62 \subsection{Document Overview} … … 59 68 60 69 Open Issues and TBDs in this document are marked \tbd{in bold, red 61 with surrounding square brackets}.70 type with surrounding square brackets}. 62 71 63 72 \section{Referenced Documents} … … 68 77 69 78 \DocumentsInternalSection 70 PSDC- 430-xxx& PS-1 Design Reference Mission \\ \hline79 PSDC-130-001 & PS-1 Design Reference Mission \\ \hline 71 80 PSDC-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline 72 81 PSDC-430-006 & Pan-STARRS IPP ADD \\ \hline … … 79 88 80 89 \section{System Design Decisions} 81 82 \PS{} is a survey telescope system being developed by the83 University of Hawaii Institute for Astronomy (IfA), the Maui High84 Performance Computing Center (MHPCC), Science Applications85 International Corporation (SAIC), and Massachusetts Institute of86 Technology (MIT) Lincoln Laboratory. The baseline system will consist87 of four 1.8m telescopes, each with a 1 gigapixel camera capable of88 sustained image rates of 2 per minute. An single initial test89 telescope (PS-1) will be constructed on Haleakala and will see first90 light at the beginning of 2006. The full four-telescope system (PS-4)91 will follow PS-1 by roughly 2 years.92 90 93 91 Since \PS{} is a survey project, all data from the telescopes … … 112 110 Processing System (MOPS), and potentially other client science 113 111 pipelines. 112 113 \subsection{System Overview} 114 114 115 115 The \PS{} Image Processing Pipeline (IPP) consists of a … … 152 152 requirements. 153 153 154 \subsection{System Overview}155 154 \subsection{System Architecture} 156 155 \subsubsection{Architectural Components} … … 191 190 \begin{center} 192 191 \resizebox{8cm}{!}{\includegraphics{pics/overview}} 193 \caption{ \label{overview} IPP System Overview} 192 \caption{ \label{overview} IPP System Overview. \tbd{``Processing 193 Jobs'' should be renamed ``Analysis Pipelines''.} } 194 194 \end{center} 195 195 \end{figure} 196 196 197 \subsubsection{Analysis Stages} 198 199 We now consider the collection of analysis tasks which are performed 200 by the IPP. Depending on the task, they may be performed on 201 individual images, collections of images, or on derived data products. 202 Because of the nature of the image data, many of the analysis tasks 203 can be performed in parallel because, for example, the analysis of an 204 OTA in one image does not depend on the results from another OTA. We 205 define the analysis pipelines to be the largest complete analysis task 206 which may be performed on a single data item. The data analysis 207 pipelines are divided into three categories, and further subdivided as 208 follows: 209 210 \begin{enumerate} 211 \item Science Image Pipelines 212 \begin{enumerate} 213 \item Phase 1 : image processing preparation 214 \item Phase 2 : image reduction 215 \item Phase 3 : exposure analysis 216 \item Phase 4 : image combination 217 \end{enumerate} 218 \item Calibration Image Pipelines 219 \begin{enumerate} 220 \item Calibration 1 : basic master-detrend creation 221 \item Calibration 2 : Sky-model/fringe-mode generation 222 \item Calibration 3 : Flat-field correction image Creation 223 \end{enumerate} 224 \item Reference Catalog Pipelines 225 \begin{enumerate} 197 \subsubsection{Analysis Pipelines} 198 199 We now consider the collection of IPP analysis pipelines. Depending 200 on the particular pipeline, they may be run on individual images, 201 collections of images, or on derived data products. Because of the 202 nature of the image data, many of the analysis pipelines can be run in 203 parallel because, for example, the analysis of a chip in one image 204 does not depend on the results from another chip. We define the 205 analysis pipelines to be the largest complete analysis task which may 206 be performed on a single data item. The data analysis tasks are 207 divided into three categories, and further subdivided as follows: 208 209 \begin{enumerate} 210 \item Science Image Pipelines 211 \begin{enumerate} 212 \item Phase 1: image processing preparation 213 \item Phase 2: image reduction 214 \item Phase 3: exposure analysis 215 \item Phase 4: image combination 216 \end{enumerate} 217 \item Calibration Image Pipelines 218 \begin{enumerate} 219 \item Calibration 1: basic master-detrend creation 220 \item Calibration 2: Sky-model/fringe-mode generation 221 \item Calibration 3: Flat-field correction image Creation 222 \end{enumerate} 223 \item Reference Catalog Pipelines 224 \begin{enumerate} 226 225 \item Astrometry reference catalog generation 227 226 \item Photometry reference catalog generation 228 \end{enumerate}227 \end{enumerate} 229 228 \end{enumerate} 230 229 … … 241 240 \begin{center} 242 241 \resizebox{8cm}{!}{\includegraphics{pics/pipelines}} 243 \caption{ \label{pipelines} IPP System Overview} 242 \caption{ \label{pipelines} IPP System Overview. \tbd{Small part at 243 top is missing.} } 244 244 \end{center} 245 245 \end{figure} … … 248 248 249 249 The basic IPP hardware organization is shown in Figure~\ref{hardware}. 250 The overall hardware organization, with a n OTAsubcluster and a251 Static -Sky subcluster, is largely chosen to reduce the I/O load during250 The overall hardware organization, with a Detector subcluster and a 251 Static Sky subcluster, is largely chosen to reduce the I/O load during 252 252 the pre-reduction analysis of the raw science images. In addition, we 253 253 have specified distinct machines to maintain the object and metadata … … 255 255 defined the details of these databases; it may be more appropriate 256 256 depending on the eventual solutions to distribute these database 257 elements across the OTAand Static Sky subclusters.257 elements across the Detector and Static Sky subclusters. 258 258 259 259 \begin{figure} … … 358 358 requirements, the IPS may maintain the pixel data distributed across 359 359 the processor nodes in an organized fashion, i.e.\ associating 360 specific machines with specific OTAs. The IPS interacts with the IPP361 Metadata Database to allow other systems or subsystems to identify the 362 available images meeting specified criteria. IPS specifications are 363 described in the IPS subsystem specification.360 specific machines with specific detectors. The IPS interacts with the 361 IPP Metadata Database to allow other systems or subsystems to identify 362 the available images meeting specified criteria. IPS specifications 363 are described in the IPS subsystem specification. 364 364 365 365 In addition to storing the pixel data, the IPS is responsible for … … 572 572 \begin{tabular}{l} 573 573 \hline 574 \multicolumn{ 1}{l}{\bf Metadata Tables} \\575 Weather \\576 SkyProbe \\577 LRProbe \\578 DIMM \\579 NIR \\580 Dome Status \\581 Telescope Status \\582 Raw FPAs \\583 Raw OTAs\\584 Raw Cells \\585 Observation Group \\586 OTA Guide Stars \\587 Science OTA stats\\588 Science Cell stats \\589 Science FPA stats 590 Sky- OTA overlaps\\591 Processed Sky-Cell stats \\592 Calibration 1 input OTA stats\\593 Calibration 1 output OTA stats\\594 Calibration 2 input OTA stats\\595 Calibration 2 output OTA stats\\596 Calibration 3 input stats \\597 Calibration 3 output stats \\574 \multicolumn{2}{l}{\bf Metadata Tables} \\ 575 Weather & Details on the weather, including internal temperatures. \\ 576 SkyProbe & Analysis of SkyProbe data. \\ 577 LRProbe & Analysis of LRProbe data. \\ 578 DIMM & Analysis of DIMM data. \\ 579 NIR & Analysis of NIR data. \\ 580 Dome Status & The status of the dome. \\ 581 Telescope Status & The status of the telescope. \\ 582 Raw FPAs & Details on raw FPA exposures. \\ 583 Raw Chips & Details on raw chips. \\ 584 Raw Cells & Details on raw cells. \\ 585 Observation Group & Details on a group of observations to be processed. \\ 586 Chip Guide Stars & Details on guide stars \\ 587 Science Chip stats & Details on processed chips. \\ 588 Science Cell stats & Details on processed cells. \\ 589 Science FPA stats & Details on processed FPAs. 590 Sky-Detector overlaps & List of overlaps between sky cells and detectors. \\ 591 Processed Sky-Cell stats & Details on sky cells. \\ 592 Calibration 1 input stats & Details on input images for Cal 1. \\ 593 Calibration 1 output stats & Details on output detrend images from Cal 1. \\ 594 Calibration 2 input stats & Details on input images for Cal 2. \\ 595 Calibration 2 output stats & Details on output detrend images from Cal 2. \\ 596 Calibration 3 input stats & Details on input images for Cal 3. \\ 597 Calibration 3 output stats & Details on output detrend images from Cal 3. \\ 598 598 \hline 599 599 \end{tabular} … … 781 781 Reference catalog & The reference catalog that was used for the photometry. \\ 782 782 PSF stats & Summary statistics of the PSF. \\ 783 OTA state & \tbd{The state of the OTA?} \\783 Chip state & \tbd{The state of the chip?} \\ 784 784 Software versions & Versions of each of the modules used in the processing. \\ 785 785 \hline … … 822 822 \begin{tabular}{ll} 823 823 \hline 824 \multicolumn{2}{l}{\bf Sky- Chipoverlaps} \\824 \multicolumn{2}{l}{\bf Sky-Detector overlaps} \\ 825 825 Chip ID & The identification number of the chip. \\ 826 826 Sky Cell ID & The identification number of the sky cell. \\ … … 849 849 \begin{tabular}{ll} 850 850 \hline 851 \multicolumn{2}{l}{\bf Calibration 1 input Chipstats} \\851 \multicolumn{2}{l}{\bf Calibration 1 input stats} \\ 852 852 Input ID & The input chip identification number. \\ 853 853 Output ID & The identification number of the output detrend image. \\ … … 861 861 \begin{tabular}{ll} 862 862 \hline 863 \multicolumn{2}{l}{\bf Calibration 1 output Chipstats} \\863 \multicolumn{2}{l}{\bf Calibration 1 output stats} \\ 864 864 Output ID & The identification number of the output detrend image. \\ 865 865 Data type & The type of the detrend image (bias | dark | flat) \\ … … 875 875 \begin{tabular}{ll} 876 876 \hline 877 \multicolumn{2}{l}{\bf Calibration 2 input Chipstats} \\877 \multicolumn{2}{l}{\bf Calibration 2 input stats} \\ 878 878 Input ID & The input chip identification number. \\ 879 879 Output ID & The identification number of the output detrend image. \\ … … 889 889 \begin{tabular}{ll} 890 890 \hline 891 \multicolumn{2}{l}{\bf Calibration 2 output OTAstats } \\891 \multicolumn{2}{l}{\bf Calibration 2 output stats } \\ 892 892 Output ID & The identification number of the output detrend image. \\ 893 893 Data type & The type of the detrend image (bias | dark | flat) \\ … … 903 903 \begin{tabular}{ll} 904 904 \hline 905 \multicolumn{2}{l}{\bf Calibration 3 input stats} \\ 906 Input ID & The input chip identification number. \\ 907 Output ID & The identification number of the output detrend image. \\ 908 State & \tbd{State of the processing?} \\ 909 Accepted? & Is the detrend image of acceptable quality? \\ 910 Image stats & Assorted image statistics (mean flux, exposure time, airmass). \\ 911 Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ 912 Applied reduction & \tbd{Reduction method used?} \\ 913 Applied params & Parameters of reduction. \\ 914 \hline 915 \end{tabular} 916 917 \begin{tabular}{ll} 918 \hline 905 919 \multicolumn{1}{l}{\bf Calibration 3 output metadata } \\ 906 920 Input images & Identification numbers of the input chips. \\ … … 946 960 access to objects on the sky, including the access to the photometry 947 961 associated with specific input images, moving objects associated with 948 specific OTA images. Detailed requirements for the IOD are described949 inthe IOD subsystem specification document xxx-xxx-xxxx.962 specific chips. Detailed requirements for the IOD are described in 963 the IOD subsystem specification document xxx-xxx-xxxx. 950 964 951 965 Reference Astrometry Catalogs: … … 960 974 \begin{tabular}{l} 961 975 \hline 962 \multicolumn{ 1}{l}{\bf Object DB Tables} \\963 Images \\964 Objects \\965 Detections \\966 Non Detections\\967 Filters \\968 Photcodes \\969 Bright Objects \\970 Region Tables \\971 Average Magnitudes \\972 USNO Objects \\973 Reference Objects \\976 \multicolumn{2}{l}{\bf Object DB Tables} \\ 977 Images & The images that have objects in the DB. \\ 978 Objects & The objects --- average properties of multiple detections of the same object. \\ 979 Detections & Detections of sources in an image. \\ 980 Non-Detections & Non-detections of objects in an image. \\ 981 Filters & Filters understood by the system. \\ 982 Photcodes & \tbd{Transformations between different photometric systems?} \\ 983 Bright Objects & \tbd{Links to postage stamp images of bright objects.} \\ 984 Region Tables & \tbd{???} \\ 985 Average Magnitudes & \tbd{How is this different from an `object'?} \\ 986 USNO Objects & Objects from the USNO database. \\ 987 Reference Objects & The reference catalogs for astrometry and photometry. \\ 974 988 \hline 975 989 \end{tabular} … … 987 1001 IMD. The Controller must be able to manage more than a single 988 1002 processing thread to make maximum use of available processor 989 resources. Some analysis jobs, such as operations on the OTAs, must1003 resources. Some analysis jobs, such as operations on the chips, must 990 1004 be allocated preferentially to specified processors, while others must 991 1005 be distributed to the available machines in the cluster. … … 1150 1164 The input to this analysis is the list of guide-star pixel centroids 1151 1165 and their celestial coordinates as saved in the metadata database, as 1152 well as the FPA and OTAorganization and geometry, and the basic1166 well as the FPA and chip organization and geometry, and the basic 1153 1167 optical distortion for the camera. For the sky-cell / detector-cell 1154 1168 overlaps, the sky tiling scheme is required. 1155 1169 1156 1170 The output consists of calculated astrometric parameters (linear 1157 transformation + static distortion) for each of the FPA OTAs. On the1158 basis of this astrometry, the overlap between the OTAs and the1171 transformation + static distortion) for each of the FPA chips. On the 1172 basis of this astrometry, the overlap between the detectors and the 1159 1173 sky-cells is calculated. The output of this calculation is a list of 1160 sky-cell / OTAlinks in a database table. This list of links can be1161 used by the later stages to initiate the analyses. 1174 sky-cell / chip links in a database table. This list of links can be 1175 used by the later stages to initiate the analyses. 1162 1176 1163 1177 The phase 1 analysis is performed on an FPA basis to ensure that … … 1489 1503 1490 1504 In the Phase 2 analysis, the astrometric solutions were determined 1491 independently for each OTA. These solutions are limited by the1505 independently for each chip. These solutions are limited by the 1492 1506 assumption of a static distortion and \tbd{by the accuracy of the 1493 1507 astrometric reference}. In the phase 3 analysis, the astrometric 1494 solutions of the NFPA images are improved by \tbd{???}.1508 solutions of the $N$ FPA images are improved by \tbd{???}. 1495 1509 1496 1510 \tbd{what is the expected accuracy of the relative astrometric … … 1502 1516 1503 1517 In the Phase 2 analysis, the background is determined based only on 1504 the available sky in a single OTA image. However, the background1518 the available sky in a single chip. However, the background 1505 1519 structures are normally correlated on the scale of the FPA, so an 1506 1520 improved background solution can be determined by combining the 1507 information from many OTA images. \tbd{is the background correlated1521 information from many chips. \tbd{is the background correlated 1508 1522 between FPAs?} 1509 1523 … … 1517 1531 rejection. This combination requires the calculation of a set of PSF 1518 1532 kernels to convert each of the input images to a single, common PSF. 1519 These PSF kernels are determined from the per- OTAPSFs measured in1533 These PSF kernels are determined from the per-chip PSFs measured in 1520 1534 Phase 2. 1521 1535 … … 1853 1867 demands in terms of data I/O throughput on the network. Phase 2 and 1854 1868 Phase 4 also present the most significant CPU demands. In this 1855 discusion, Phase 2 refers to the per- OTA image pre-processing in which1856 the instrumental signature is removed and a first pass object 1857 detection is performed. Phase 4 refers to the multiple OTA 1858 combination in which the pre-processed images are merged and combined, 1859 in both addition and subtraction, with the static sky image, and up to 1860 three objectdetection passes are performed.1869 discusion, Phase 2 refers to the per-chip pre-processing in which the 1870 instrumental signature is removed and a first pass object detection is 1871 performed. Phase 4 refers to the multiple chip combination in which 1872 the pre-processed images are merged and combined, in both addition and 1873 subtraction, with the static sky image, and up to three object 1874 detection passes are performed. 1861 1875 1862 1876 This document does not address the hardware requirements implied by … … 1892 1906 computational resources. The IPP requires relativley large amounts of 1893 1907 data storage space, primarily for the image data. Image data is 1894 organized in two categories. First, there is the per- OTA data -- data1895 associated with specific OTAs, including the raw images, the1908 organized in two categories. First, there is the per-chip data -- 1909 data associated with specific chips, including the raw images, the 1896 1910 calibration images, and temporary processed images at various stages. 1897 1911 Second, there is the data associated with the static sky imagery, … … 1900 1914 provide both data storage and computational resources. The second 1901 1915 assumption we make is that the data storage nodes are divided into two 1902 classes: those which deal with the per-OTA data and those that provide 1903 the static sky storage. In addition, we assume that the computational 1904 tasks related to Phase 2 take place on the per-OTA storage nodes and 1905 the Phase 4 computation takes place on the static sky storage nodes. 1916 classes: those which deal with the per-chip data and those that 1917 provide the static sky storage. In addition, we assume that the 1918 computational tasks related to Phase 2 take place on the per-chip 1919 storage nodes and the Phase 4 computation takes place on the static 1920 sky storage nodes. 1906 1921 1907 1922 Figure~\ref{hardware} shows our basic concept for the hardware 1908 1923 organization for the IPP. This diagram shows the two types of compute 1909 nodes: OTA-level processing and storage nodes (dominated by Phase 2)1924 nodes: chip-level processing and storage nodes (dominated by Phase 2) 1910 1925 and static sky processing and storage nodes (mostly Phase 4). Also 1911 1926 shown are two switches used in this configuration; although it is … … 1923 1938 this document, we explore two extreme-case options: 1924 1939 \begin{itemize} 1925 \item Random Data Distribution - OTA\& Sky data is randomly1926 distributed within the compute node of a given type (ie, OTA data is1927 randomly distributed among the OTAcompute nodes).1928 \item Optimal Data Distribution - OTA\& Sky data is optimally1929 distributed to compute OTA/Sky nodes (OTA processing is always on a1930 machine with local OTAdata).1940 \item Random Data Distribution --- Detector \& Sky data is randomly 1941 distributed within the compute node of a given type (ie, chip data 1942 is randomly distributed among the detector compute nodes). 1943 \item Optimal Data Distribution --- Detector \& Sky data is optimally 1944 distributed to compute Detector/Sky nodes (chip processing is always 1945 on a machine with local chip data). 1931 1946 \end{itemize} 1932 1947 A second factor which will have a significant impact on the I/O … … 2096 2111 2097 2112 The image metadata consists of values associated with the FPA (4), the 2098 OTAs (240), and the Cells (15360). Aside from the guide star history, 2099 the total data requirements for each of these entries will be scaled 2100 by the number of bytes required for the metadata from each data level. 2101 Clearly, if the Cell entry is allowed to be large, it will dominate 2102 the total Metadata data volume. If we suggest an expected number of 2103 64 bytes per Cell, 256 B per OTA, and 1k per FPA, we find a total 2104 metadata volume per exposure of roughly 1 MB, completely dominated by2105 the Cell metadata. With the exposure rates above, we find a total of 2106 metadata volume of 1.8 TB over the lifetime of the project. For PS-1, 2107 the total volume is reduced by a factor of 2.5 (for the shorter 2108 lifetime) and another factor of 4 (for the lone telescope). Neither 2109 data quantity is affected by the minimal vs standard data volume 2110 choice.2113 chips (240), and the Cells (15360). Aside from the guide star 2114 history, the total data requirements for each of these entries will be 2115 scaled by the number of bytes required for the metadata from each data 2116 level. Clearly, if the Cell entry is allowed to be large, it will 2117 dominate the total Metadata data volume. If we suggest an expected 2118 number of 64~bytes per Cell, 256~B per chips, and 1~kB per FPA, we find a 2119 total metadata volume per exposure of roughly 1~MB, completely 2120 dominated by the Cell metadata. With the exposure rates above, we 2121 find a total of metadata volume of 1.8~TB over the lifetime of the 2122 project. For PS-1, the total volume is reduced by a factor of 2.5 2123 (for the shorter lifetime) and another factor of 4 (for the lone 2124 telescope). Neither data quantity is affected by the minimal vs 2125 standard data volume choice. 2111 2126 2112 2127 \paragraph{Object Database Storage} … … 2142 2157 needed by the object detection, deblending, and analysis. Experiments 2143 2158 with the FFTW package show that FFTs may be performed on Intel 2144 processors at rates of approximately 0.25 2159 processors at rates of approximately 0.25~GHz-sec / Mpix for data sets 2145 2160 of order 1 Megapixel. The FFTs required for the Phase 2 analysis are 2146 2161 performed on the 512$^2$ pixel cells, so these numbers may roughly be 2147 scaled linearly to determine the total time required for OTA2148 processing. A single FFT on a full OTA, with 64 Cells, therefore2149 requires roughly 4 2162 scaled linearly to determine the total time required for chip 2163 processing. A single FFT on a full chip, with 64 cells, therefore 2164 requires roughly 4~GHz-sec. For the full Phase 2 analysis, there are 2150 2165 roughly 4 single direction FFTs required excluding those associated 2151 2166 with object detection; thus the total processing time for these FFTs 2152 is approximately 16 2167 is approximately 16~GHz-sec. The addtional analysis steps, excluding 2153 2168 object detection and characterization, account for a small fraction of 2154 2169 this compute time, which we estimate at 10\%. The object detection … … 2156 2171 performed, and the number of measurements made per object. Typical 2157 2172 analysis performed by the Sextractor routine, which performs a 2158 substantial number of per-object analyses, requires 27 2159 full OTA, including the FFTs used for smoothing. We can therefore2160 assume a total of 50 GHz-sec per OTA for the Phase 2 processing. This2161 converts to a total of 12000GHz-sec for a complete major frame.2173 substantial number of per-object analyses, requires 27~GHz-sec for a 2174 full chip, including the FFTs used for smoothing. We can therefore 2175 assume a total of 50~GHz-sec per chip for the Phase 2 processing. 2176 This converts to a total of 12,000~GHz-sec for a complete major frame. 2162 2177 2163 2178 For Phase 4, the main computational tasks are combining the multiple 2164 2179 images, with cosmic-ray rejection, and performing the object detection 2165 2180 tasks. Nick Kaiser has done tests of the Phase 4 image combine and 2166 rejection stages, and finds a total processing time of roughly 962167 GHz-sec for a full stack of 4 OTA images. If we add in an additional2168 34 2169 find a conservative estimage of 130 GHz-sec for a 4-image OTAstack,2170 equivalent to 7800 2181 rejection stages, and finds a total processing time of roughly 2182 96~GHz-sec for a full stack of 4 chips. If we add in an additional 2183 34~GHz-sec for detailed object detection and image differencing, we 2184 find a conservative estimage of 130~GHz-sec for a 4-image chip stack, 2185 equivalent to 7800~GHz-sec for a major frame. 2171 2186 2172 2187 For PS-1, the data processing will clearly require a smaller amount of … … 2181 2196 \begin{table} 2182 2197 \begin{center} 2183 \caption{Data Scenarios (MB per OTAor Sky-cell) \label{scenarios}}2198 \caption{Data Scenarios (MB per Chip or Sky-cell) \label{scenarios}} 2184 2199 \begin{tabular}{lrrrr} 2185 2200 \hline … … 2244 2259 2245 2260 In the Random Data Distribution scenario, there is a single CPU 2246 allocated to each OTA in the OTAfarm and a single CPU for each Sky2247 cell process. The OTAdata are stored across random machines in the2248 OTAfarm, with the result that every Phase 2 processing requires2249 network access to the data. For each science OTA imagewhich is2250 observed, each OTAnode will read from the network a total of 560 MB2261 allocated to each chip in the detector farm and a single CPU for each Sky 2262 cell process. The chip data are stored across random machines in the 2263 detector farm, with the result that every Phase 2 processing requires 2264 network access to the data. For each science chip which is 2265 observed, each detector node will read from the network a total of 560 MB 2251 2266 (the 2 raw images for data storage and the 7 calibration frames, along 2252 2267 with one mask and one raw input image) and write a total of 200 MB … … 2259 2274 \paragraph{Random / Minimal Data Scenario} 2260 2275 2261 In the Random-Minimal, there is a single CPU allocated to each OTAin2262 the OTA farm and a single CPU for each Sky cell process, and the OTA2263 data are stored across random machines in the OTA farm. However, the 2264 calibration and the processed science images are stored at 2 bytes per 2265 pixel, the mask is set at 4 bits per pixel, and only 4 calibration 2266 images are assumed. For each science OTA image which is observed, 2267 each OTA node will read from the network a total of 232 MB (the 2 raw 2268 images for data storage and the 4 calibration frames, along with one 2269 mask and one raw input image) and write a total of 100 MB (one 2270 processed image and the mask along with the 1.5 processed images for 2271 the Phase 4 analysis). Given the assumption of 50 MB/s from the2276 In the Random-Minimal, there is a single CPU allocated to each chip in 2277 the detector farm and a single CPU for each Sky cell process, and the 2278 chip data are stored across random machines in the detector farm. 2279 However, the calibration and the processed science images are stored 2280 at 2 bytes per pixel, the mask is set at 4 bits per pixel, and only 4 2281 calibration images are assumed. For each science chip which is 2282 observed, each detector node will read from the network a total of 232 MB 2283 (the 2 raw images for data storage and the 4 calibration frames, along 2284 with one mask and one raw input image) and write a total of 100 MB 2285 (one processed image and the mask along with the 1.5 processed images 2286 for the Phase 4 analysis). Given the assumption of 50 MB/s from the 2272 2287 network adapter, the total data volume implies an I/O period of 6.6 2273 2288 seconds. Again, note that the disk I/O is parallel with the network … … 2277 2292 2278 2293 In the Optimal Data Distribution scenario, there is a single CPU 2279 allocated to each OTA in the OTA farm and a single CPU for each Sky2280 cell process. In addition, all data for the specified OTA are stored 2281 on local disks attached to the same computer as the CPU, with the 2282 result that all Phase 2 I/O is made to a local disk. For each science 2283 OTA image which is observed, each OTA node will read from the network 2284 a total of 2 raw images (one for the original image, one for the 2285 backup copy) and write an average of roughly 1.5 processed images and 2286 masks to the Phase 4 machines for a total of 184 MB of network I/O. 2287 During the processing stage, the OTA node will read from disk a total 2288 of 496 MB (7 calibration frames at 64 MB each, one 16 MB mask, and one 2289 raw science image at 32 MB) and write a total of 80 MB (one processed 2290 image at 64 MB and one mask at 8 MB). Given the assumptions forthe2291 network and disk bandwidths (50 MB/s and 100 MB/s respectively), the 2292 data volumes imply a total I/O period of 9.5 seconds. In this 2293 instance, the network I/O is presumed to be sequential with the disk 2294 I/O.2294 allocated to each chip in the detector farm and a single CPU for each 2295 Sky cell process. In addition, all data for the specified chip are 2296 stored on local disks attached to the same computer as the CPU, with 2297 the result that all Phase 2 I/O is made to a local disk. For each 2298 science chip which is observed, each detector node will read from the 2299 network a total of 2 raw images (one for the original image, one for 2300 the backup copy) and write an average of roughly 1.5 processed images 2301 and masks to the Phase 4 machines for a total of 184 MB of network 2302 I/O. During the processing stage, the detector node will read from 2303 disk a total of 496 MB (7 calibration frames at 64 MB each, one 16 MB 2304 mask, and one raw science image at 32 MB) and write a total of 80 MB 2305 (one processed image at 64 MB and one mask at 8 MB). Given the 2306 assumptions for the network and disk bandwidths (50 MB/s and 100 MB/s 2307 respectively), the data volumes imply a total I/O period of 9.5 2308 seconds. In this instance, the network I/O is presumed to be 2309 sequential with the disk I/O. 2295 2310 2296 2311 \paragraph{Optimal / Minimal Data Scenario} … … 2305 2320 \paragraph{Phase 4 Node I/O Requirements / Standard Data Volume} 2306 2321 2307 Although it is easy to arrange the OTA data in such a way that the2308 majority of I/O is performed locally, it is not as easy to arrange2322 Although it is easy to arrange the detector data in such a way that 2323 the majority of I/O is performed locally, it is not as easy to arrange 2309 2324 this for the Static Sky data used by the Phase 4 analysis. We 2310 2325 therefore make the assumption that the Phase 4 analysis will require 2311 all input OTA data to be loaded across the network, as well as all2312 Static Sky data. This is somewhat of an overestimate as some of the 2313 Static Sky data will be processed by machines with the data stored2326 all input detector data to be loaded across the network, as well as 2327 all Static Sky data. This is somewhat of an overestimate as some of 2328 the Static Sky data will be processed by machines with the data stored 2314 2329 locally, and clever Static-Sky data organization schemes can enhance 2315 this chance. 2330 this chance. 2316 2331 2317 2332 In the Phase 4 analysis, the images from the 4 separate telescopes are 2318 2333 combined into a single image, confronted with the appropriate segment 2319 2334 of the static sky, with output difference image and updated static sky 2320 image. If we restrict input access to the individual OTAcells, the2335 image. If we restrict input access to the individual chip cells, the 2321 2336 maximum read overhead is 50\% (need to read a 10x10 set of cells for 2322 2337 an 8x8 input image). If the processing is performed on Static Sky 2323 segments equivalent in size to the OTAs, the input data is 608 MB (3842338 segments equivalent in size to the chips, the input data is 608 MB (384 2324 2339 MB of processed science image, 96 MB of mask images, 64 MB of static 2325 2340 sky image and 64 MB of static sky weight map) while the output data is … … 2376 2391 \paragraph{Random / Standard Data Scenario} 2377 2392 2378 In the Random Data Distribution scenario, each OTA node needs to read2379 a total of 560 MB from the network and write a total of 200 MB every 2380 30 seconds. With 240 OTA nodes, this corresponds to a total bandwidth 2381 of 6080 MB/sec, or 49 Gb/sec. Note that this includes the bandwidth 2382 needed to copy data from the summit and make two copies on the OTA 2383 machines, as well as the bandwidth to send the processed image 2384 portions to the Phase 4 machines. The Phase 4 processing adds an 2385 a dditional 320 MB of network I/O per Sky-Cell group, and there are2393 In the Random Data Distribution scenario, each detector node needs to 2394 read a total of 560 MB from the network and write a total of 200 MB 2395 every 30 seconds. With 240 detector nodes, this corresponds to a 2396 total bandwidth of 6080 MB/sec, or 49 Gb/sec. Note that this includes 2397 the bandwidth needed to copy data from the summit and make two copies 2398 on the detector machines, as well as the bandwidth to send the processed 2399 image portions to the Phase 4 machines. The Phase 4 processing adds 2400 an additional 320 MB of network I/O per Sky-Cell group, and there are 2386 2401 roughly 60-70 Sky-cells per exposure set. Thus the Phase 4 processing 2387 2402 adds an additional 750 MB/sec network bandwidth. In the architecture 2388 defined in Figure \tbd{NN}, the Sky nodes and the OTAnodes are each2403 defined in Figure \tbd{NN}, the Sky nodes and the detector nodes are each 2389 2404 attached to separate switches. An additional bandwidth requirement is 2390 2405 derived by the need to exchange data between these switches in for … … 2412 2427 2413 2428 In the Optimal Data Distribution, the Phase 2 network bandwidth is 2414 reduced significantly to 184 MB per OTAnode, for a total of2429 reduced significantly to 184 MB per detector node, for a total of 2415 2430 1.5GB/sec, while the Phase 4 network bandwidth remains unchanged at 2416 2431 750 MB/sec. The inter-switch communication also remains the same at … … 2420 2435 2421 2436 In the Optimal / Minimal Scenario, the total Phase 2 network bandwidth 2422 drops to 124 MB per OTAnode, for a total of 1.0GB/sec, while the2437 drops to 124 MB per detector node, for a total of 1.0GB/sec, while the 2423 2438 Phase 4 network bandwidth is 552 MB/sec. The inter-switch 2424 2439 communication remains the same as the Random/Minimal Scenario at 560 … … 2480 2495 bandwidths. The important conclusion in this analysis is the implied 2481 2496 number of GHz per processor, given the assumptions laid out. 2482 Phase 2 is specified to have 240 OTAnodes, while Phase 4 is specified2497 Phase 2 is specified to have 240 detector nodes, while Phase 4 is specified 2483 2498 to have roughly 60 static sky nodes. The range of Phase 2 CPU 2484 2499 requirements implies that each CPU needs to have speeds in the range … … 2513 2528 \section{Notes} 2514 2529 2515 \subsection{Cell vs OTA vs Mosaic vs Major Frame} 2516 2517 There are several levels of input data pixel groups: Cell, OTA, 2518 Mosaic, and Major Frame. It is necessary to make the association 2519 between the data of one level and that of the next in a way that is 2520 reliable and robust to missing elements. If a specific cell is 2521 missing from an OTA, that information is known by the controller an 2522 needs to be represented in the metadata. Similarly if an OTA is 2523 missing from a mosaic camera, that information is also known and must 2524 be carried though the metadata. A more difficult association is that 2525 between the telescopes to define the major frame. Some possibilities: 2530 \subsection{Cell vs Chip vs FPA vs Major Frame} 2531 2532 There are several levels of input data pixel groups: Cell, Chip, Focal 2533 Plane Array (FPA), and Major Frame. It is necessary to make the 2534 association between the data of one level and that of the next in a 2535 way that is reliable and robust to missing elements. If a specific 2536 cell is missing from a chip, that information is known by the 2537 controller an needs to be represented in the metadata. Similarly if a 2538 chip is missing from a mosaic camera, that information is also known 2539 and must be carried though the metadata. A more difficult association 2540 is that between the telescopes to define the major frame. Some 2541 possibilities: 2526 2542 2527 2543 \begin{enumerate} … … 2572 2588 Phase 4. 2573 2589 2574 \subsection{Pending Sky-cell / OTAtable}2575 2576 Define a pending sky-cell / OTAtable to define the overlaps and to2590 \subsection{Pending Sky-cell / Detector table} 2591 2592 Define a pending sky-cell / detector table to define the overlaps and to 2577 2593 give something which the scheduler can query to decide when to 2578 2594 initiate phase 4.
Note:
See TracChangeset
for help on using the changeset viewer.
