Changeset 421
- Timestamp:
- Apr 12, 2004, 6:05:15 PM (22 years ago)
- File:
-
- 1 edited
-
trunk/doc/design/design.tex (modified) (76 diffs)
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/design/design.tex
r410 r421 1 %%% $Id: design.tex,v 1. 2 2004-04-09 02:27:09 eugene Exp $1 %%% $Id: design.tex,v 1.3 2004-04-13 04:05:15 price Exp $ 2 2 %\documentclass[panstarrs,psreport]{panstarrs} 3 3 \documentclass[panstarrs]{panstarrs} 4 4 5 5 % basic document variables 6 \title{Pan-STARRS Image Processing Pipeline Supplementary Design Requirements} 7 \shorttitle{IPP SSDD} 6 \title{Pan-STARRS Image Processing Pipeline} 7 \subtitle{Supplementary Design Requirements Specification} 8 \shorttitle{IPP SDRS} 8 9 \author{Eugene Magnier, Paul Price, Josh Hoblitt} 9 \group{ Pan-STARRSAlgorithm Group}10 \project{ Pan-STARRSImage Processing Pipeline}10 \group{\PS{} Algorithm Group} 11 \project{\PS{} Image Processing Pipeline} 11 12 \organization{Institute for Astronomy} 12 13 \version{DR} … … 39 40 40 41 This document establishes the design, performance, development, and 41 verification requirements for the Pan-STARRSImage Processing Pipeline42 (IPP) for both the full four-telescope Pan-STARRSdeployment (PS-4)42 verification requirements for the \PS{} Image Processing Pipeline 43 (IPP) for both the full four-telescope \PS{} deployment (PS-4) 43 44 and the initial single-telescope demonstration deployment (PS-1). 44 45 … … 49 50 \subsection{Document Overview} 50 51 51 Open Issues and TBDs in this document are marked in bold with52 surrounding square brackets.52 Open Issues and TBDs in this document are marked in bold red type, 53 with surrounding square brackets, \tbd{like this}. 53 54 54 55 \section{Referenced Documents} … … 59 60 \hline 60 61 \multicolumn{2}{l}{\bf Internal Documents} \\ 61 xxx-xxx-xxx & Pan-STARRSTelescope Scheduler specification document \\62 xxx-xxx-xxx & \PS{} Telescope Scheduler specification document \\ 62 63 xxx-xxx-xxx & Telescope Control System specification document \\ 63 64 xxx-xxx-xxx & Summit Pixel Server specification document \\ … … 66 67 xxx-xxx-xxx & Camera Readout specification document \\ 67 68 xxx-xxx-xxx & PS-1 Design Reference Mission \\ 68 xxx-xxx-xxx & Pan-STARRSC Code Conventions \\69 xxx-xxx-xxx & \PS{} C Code Conventions \\ 69 70 \hline 70 71 \multicolumn{2}{l}{\bf External Documents} \\ … … 77 78 \section{System Design Decisions} 78 79 79 Pan-STARRSis a survey telescope system being developed by the80 \PS{} is a survey telescope system being developed by the 80 81 University of Hawaii Institute for Astronomy (IfA), the Maui High 81 82 Performance Computing Center (MHPCC), Science Applications 82 International Corporation (SAIC), and \note{Massachusetts Institute of83 Technology (MIT) Lincoln Laboratory }. The baseline system will84 consist of 4 1.8m telescopes, each with a 1 gigapixel camera capable 85 ofsustained image rates of 2 per minute. An single initial test83 International Corporation (SAIC), and Massachusetts Institute of 84 Technology (MIT) Lincoln Laboratory. The baseline system will consist 85 of four 1.8m telescopes, each with a 1 gigapixel camera capable of 86 sustained image rates of 2 per minute. An single initial test 86 87 telescope (PS-1) will be constructed on Haleakala and will see first 87 88 light at the beginning of 2006. The full four-telescope system (PS-4) 88 89 will follow PS-1 by roughly 2 years. 89 90 90 Since Pan-STARRSis a survey project, all data from the telescopes91 will be uniformly analysed by the Pan-STARRSImage Processing Pipeline91 Since \PS{} is a survey project, all data from the telescopes 92 will be uniformly analysed by the \PS{} Image Processing Pipeline 92 93 (IPP) and the appropriate resulting data products made available to 93 94 internal and external science analysis systems as they become … … 95 96 will consist of detrending and object detection for the individual 96 97 images, combination of multiple overlapping images and further object 97 detection, subtraction of a reference (static-sky) image and detection o98 f residual objects, update of the static sky images, and detailed98 detection, subtraction of a reference (static-sky) image and detection 99 of residual objects, update of the static sky images, and detailed 99 100 object analysis of the static sky images. In addition, the IPP will 100 101 produce improved astrometric and photometric reference catalogs on an … … 104 105 object photometry, and reference astrometry and photometry. 105 106 106 The IPP interacts closely with other Pan-STARRS systems responsible 107 for other aspects of the Pan-STARRS operation, including the summit 108 systems (OATS), the science object database, the Moving/Transient 109 Object Pipeline, and potentially other client science pipelines. 110 111 The Pan-STARRS Image Processing Pipeline (IPP) consists of a 107 The IPP interacts closely with other \PS{} systems responsible 108 for other aspects of the \PS{} operation, including the summit 109 systems (OATS), the science object database, the Moving Object 110 Processing System (MOPS), and potentially other client science 111 pipelines. 112 113 The \PS{} Image Processing Pipeline (IPP) consists of a 112 114 collection of computer hardware and software organized to perform the 113 tasks required to process images from the Pan-STARRStelescopes. The115 tasks required to process images from the \PS{} telescopes. The 114 116 primary goal of the IPP is to process the science images from the 115 Pan-STARRStelescopes and make the results available to other systems116 within Pan-STARRS. To achieve this goal, the IPP must also perform117 \PS{} telescopes and make the results available to other systems 118 within \PS{}. To achieve this goal, the IPP must also perform 117 119 other analysis functions to generate the calibrations needed in the 118 120 science image processing and to occasionally use the derived data to … … 120 122 121 123 In order to meet these broad goals, the IPP must have the following 122 capabilities. First, the IPP must have the ability to store a large 123 amount of image data, and other derived data products (metadata \& 124 extracted objects), to provice access mechanisms to these data 125 products (both to the subsystems of the IPP and in some cases to 126 external users), and to continuously accept new image data and 127 metadata from the telescope system, 2) to execute various analysis 128 processes using these data products, 3) to provide the decision-making 129 logic needed to guide the data processing, and to automatically launch 130 the data processing tasks on an appropriate timescale. The IPP 131 therefore includes subsystems which provide the data storage 124 capabilities: 125 \begin{itemize} 126 \item Store a large amount of image data, and other derived data 127 products (metadata and extracted objects); 128 \item Provide access mechanisms to these data products (both to the 129 subsystems of the IPP and in some cases to external users); 130 \item Continuously accept new image data and 131 metadata from the telescope system; 132 \item Execute various analysis processes using these data products; 133 and 134 \item Provide the decision-making logic needed to guide the data 135 processing, and to automatically launch the data processing tasks on 136 an appropriate timescale. 137 \end{itemize} 138 The IPP therefore includes subsystems which provide the data storage 132 139 framework, the data analysis framework, and the scheduling of the 133 140 analysis processes. The data storage subsystems also provide 134 interface mechanisms to the external Pan-STARRSsystems.141 interface mechanisms to the external \PS{} systems. 135 142 136 143 The IPP architecture can be viewed in several possible ways. We first … … 147 154 \subsubsection{Architectural Components} 148 155 149 The IPP is organised into several different software elements, listed156 The IPP is organised into several different architectural components, 150 157 as follows: 151 158 152 159 \begin{enumerate} 153 \item Pixel Server 154 \item Object Database 155 \item Metadata Database 156 \item Analysis Pipelines 157 \item Controller 158 \item Scheduler 160 \item IPP Pixel Server (IPS) --- a respository for all image pixel 161 data, including the raw images from the telescope, the master 162 calibration images, the reference static-sky images, and any temporary 163 image data products produced by the IPP. 164 \item IPP Object Database (IOD) --- a facility to store all of the 165 information about astronomical objects, including individual 166 measurements of objects on the images, the summary information about 167 those objects, and reference object data\footnote{Note that this is 168 (possibly) a separate entity from the object database being developed 169 by SAIC.}. 170 \item IPP Metadata Database (IMD) --- a storage element for all data 171 which is neither image pixel data or astronomical object data. 172 \item Analysis Pipelines --- all of the top-level analysis processes 173 which are performed on images or collections of object data. 174 \item Controller --- a system which manages the process of executing 175 in parallel analysis pipelines on specific datasets on the cluster of 176 computers. 177 \item Scheduler --- a system which evaluates the current state of data 178 in the various repositories and makes decisions about which analysis 179 processes should be executed at any given time. 159 180 \end{enumerate} 160 181 161 182 The relationship between these software elements is shown in 162 183 Figure~\ref{overview}. This figure also shows the interactions 163 between the IPP and other Pan-STARRS systems. The Pixel Server is a 164 respository for all image pixel data, including the raw images from 165 the telescope, the master calibration images, the reference static-sky 166 images, and any temporary image data products produced by the IPP. 167 The Object Database is a facility to store all of the information 168 about astronomical objects, including individual measurements of 169 objects on the images, the summary information about those objects, 170 and reference object data. The Metadata Database is a storage element 171 for all data which is neither image pixel data or astronomical object 172 data. The analysis pipelines are all of the top-level analysis 173 processes which are performed on images or collections of object data. 174 The Controller is a system which manages the process of executing in 175 parallel analysis pipelines on specific datasets on the cluster of 176 computers. The Scheduler is a system which evaluates the current 177 state of data in the various repositories and makes decisions about 178 which analysis processes should be executed at any given time. 184 between the IPP and other \PS{} systems. 185 186 The IPP team will develop and have responsibility for these systems. 179 187 180 188 \begin{figure} 181 189 \begin{center} 182 \resizebox{8cm}{!}{\includegraphics{pics/overview .ps}}190 \resizebox{8cm}{!}{\includegraphics{pics/overview}} 183 191 \caption{ \label{overview} IPP System Overview} 184 192 \end{center} … … 194 202 OTA in one image does not depend on the results from another OTA. We 195 203 define the analysis pipelines to be the largest complete analysis task 196 which may be performed on a single data item. {\bf drop the word197 'pipeline' and use something else?}. The data analysis pipelines are 198 divided into three categories, and further subdivided asfollows:204 which may be performed on a single data item. The data analysis 205 pipelines are divided into three categories, and further subdivided as 206 follows: 199 207 200 208 \begin{enumerate} … … 223 231 controller. The thick lines represent the flow of pixel data, the 224 232 thin lines represent the flow of metadata and object data, and the 225 grey lines represent the flow of commands. {\bf All subsystem 226 interactions, except that between the scheduler and controller, are in 227 the form of updates to and queries from the databases}. The hatched 228 systems represent external PanSTARRS systems (OATS, the Sky Server, 229 the SAIC Object Database, the Moving/Transient Object Pipeline, and 230 other Client Science Pipelines. 233 grey lines represent the flow of commands. The hatched systems 234 represent external \PS{} systems (OATS, the Sky Server, the SAIC 235 Object Database, the Moving Object Processing System, and other Client 236 Science Pipelines). 231 237 232 238 \begin{figure} 233 239 \begin{center} 234 \resizebox{8cm}{!}{\includegraphics{pics/pipelines .ps}}240 \resizebox{8cm}{!}{\includegraphics{pics/pipelines}} 235 241 \caption{ \label{pipelines} IPP System Overview} 236 242 \end{center} … … 246 252 databases. This last aspect is largely theoretical until we have 247 253 defined the details of these databases; it may be more appropriate 248 depending on the eventual solutions to distribut ionthese database254 depending on the eventual solutions to distribute these database 249 255 elements across the OTA and Static Sky subclusters. 250 256 251 257 \begin{figure} 252 258 \begin{center} 253 \resizebox{8cm}{!}{\includegraphics{pics/hardware .ps}}259 \resizebox{8cm}{!}{\includegraphics{pics/hardware}} 254 260 \caption{ \label{hardware} IPP Hardware Organization} 255 261 \end{center} … … 258 264 \subsection{Software Hierarchy} 259 265 260 \subsubsection{External Data Libraries}261 262 \subsubsection{Pan-STARRS Data Library}263 264 266 In order to facilitate testing and development, and to encourage 265 267 flexibility, the IPP will be built in a layered fashion. The lowest 266 268 level functions will be written in C and collected together into a 267 Pan-STARRS library. These library functions canbe used to write more269 \PS{} library. These library functions will be used to write more 268 270 complex modules. The modules will be written in C but will make use 269 271 of the SWIG tool to make their functionality available within other 270 272 frameworks. In particular, the modules can be tied together with a 271 simple framework ( 'theengine') or with detailed flow-control through272 the use of a high-level language such as Perl, Python, or T CL. For273 simple framework (an `engine') or with detailed flow-control through 274 the use of a high-level language such as Perl, Python, or Tcl. For 273 275 the high-level functions in the operational system, the IPP will make 274 276 use of \tbd{Python} as the scripting language to tie the modules 275 together. Note that a subset of the library functions will be 276 provided with SWIG interfaces as well to allow for their use the in 277 creation of the top-level functions. 278 279 The Pan-STARRS Data Library consists of C structures describing the 280 basic data types needed by the IPP and C functions which perform the 281 basic data manipulation operations. The library is organized into NN 282 topics. 277 together. 278 279 This approach satisfies the requirement that complicated low-level 280 analysis steps run fast, while preserving flexibility for coding the 281 high-level wrappers for which the speed requirements are not so 282 stringent. 283 284 \subsubsection{External Libraries} 285 286 \PS{} will employ several external libraries to save duplicating 287 functionality that is already available. These external libraries 288 will be wrapped by the \PS{} Library, insulating the project from the 289 implementation details of the external libraries. Examples of the 290 external libraries are FFTW and SLALib. 291 292 \subsubsection{\PS{} Library} 293 294 The \PS{} Library will consist of C structures describing the basic 295 data types needed by the IPP and C functions which perform the basic 296 data manipulation operations. Note that a subset of the library 297 functions will be provided with SWIG interfaces as well to allow for 298 their use in the creation of the processing stages. Examples of the 299 \PS{} Library are fourier transforms and transforming between pixel 300 and celestial coordinates. 283 301 284 302 \subsubsection{Modules} … … 286 304 The IPP analysis tasks are broken down into modules which represent 287 305 specific functional operations. The modules will be written in C 288 using the Pan-STARRS Data Library functions and will be grouped into a 289 Pan-STARRS Module Library. The modules will be provided with SWIG 290 interfaces to all for their use in top-level functions. 306 using the \PS{} Library functions and will be grouped into a \PS{} 307 Module Library. The modules will be provided with SWIG interfaces to 308 all public APIs for their use in processing stages. Examples of modules 309 are overscan subtraction and image combination. 291 310 292 311 \subsubsection{Stages} 293 312 294 The major IPP tasks are organized into stages. Each stage represents 295 a collection of complex operations performed on a single data entity. 296 Each stage therefore represents the maximum amount of effort which can 297 be performed in serial without interaction between parallel threads. 313 The major IPP tasks are organized into stages, which consist of 314 multiple modules. Each stage represents a collection of complex 315 operations performed on a single data entity. Each stage therefore 316 represents the maximum amount of effort which can be performed in 317 serial without interaction between parallel threads. The stages will 318 be written in \tbd{Python}, linking the modules together. Examples of 319 stages are Phase 2 (detrend images) and Phase 4 (combine images from 320 multiple telescopes and search for transients). 321 322 \subsubsection{Controllers} 323 324 The stages are parallelized by a controller, which initiates the 325 stages on separate machines and monitors their progress. An example 326 of the controller functionality is ``Run the phase 2 processing on 327 exposure number 1234''. 328 329 \subsubsection{Scheduler} 330 331 The scheduler is responsible for interacting with \PS{} systems 332 external to the IPP, and for initiating the reduction appropriate for 333 images as they are received. An example of the scheduler 334 functionality is ``I've just received exposure number 1234; run phase 335 1--4 controllers on these''. 298 336 299 337 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%% … … 316 354 they are needed, up to the lifetime of the project. In order to 317 355 achieve the I/O requirements, the IPS may maintain the pixel data 318 distributed across the processor nodes in an organized fashion, i e356 distributed across the processor nodes in an organized fashion, i.e.\ 319 357 associating specific machines with specific OTAs. The IPS interacts 320 with the IPP InternalDatabase to allow other systems or subsystems to358 with the IPP Metadata Database to allow other systems or subsystems to 321 359 identify the available images meeting specified criteria. IPS 322 specifications are described in the IPS subsystem specification. 323 324 In addition t he IPS is responsible for acquiring new image data and325 meta-data from the Summit Pixel Server and making it available for 326 processing by the IPP System. 360 specifications are described in the IPS subsystem specification. 361 362 In addition to storing the pixel data, the IPS is responsible for 363 acquiring new image data and metadata from the Summit Pixel Server and 364 making it available for processing by the IPP System. 327 365 328 366 \paragraph{Pixel Server Components} 329 367 330 The Pixel Server consists of the following components:368 The IPP Pixel Server consists of the following components: 331 369 332 370 \begin{enumerate} … … 343 381 The IPP Pixel Data Scheduler coordinates the movement of image data 344 382 onto {\em local} storage for processing by the IPP System and executes 345 batch image data management tasks. 346 347 The IPP Pixel Data Scheduler has four basic modes of operation. 383 batch image data management tasks. By ``local storage'' is meant 384 storage accessible from a particular local machine (i.e.\ either on a 385 disk physically connected to the machine, or a disk mounted over the 386 network). 387 388 The IPP Pixel Data Scheduler has four basic modes of operation: 348 389 349 390 \begin{itemize} 350 \item The Summit Pixel Server sends a new data available message to the 351 IPP-PDS. The IPP-PDS generates a {\em retrieve data} task which is passed 352 through 0 or more registered filters. The task is then sent to the IPP Controller. 353 \item The IPP-PDS receives a clean stale data message. \tbd{The source of 354 which is TBD}. A list of {\em delete data} tasks are generated 355 which is passed to the IPP Pixel Data Locality Optimizer for assignment 356 to specific the data storage locations. The list of tasks is then sent 357 to the IPP Controller. 358 \item The IPP-PDS receives a data replication message. \tbd{The source of 359 which is TDB}. A list of {\em retrieve data} tasks are generated to 360 copy the data. The list of tasks is then sent to the IPP Controller. 361 \item The IPP-PDS receives a move data message. \tbd{The source of 362 which is TDB}. A list of {\em retrieve data} tasks are generated to copy the 363 data to it's new destination. The list of tasks is then sent to the IPP 364 Controller.l Upon receiving task completed notification from the IPP 365 Controller a list of {\em delete data} tasks are generated to remove the data 366 from it's original storage location. This list of tasks is then sent to the 367 IPP Controller. 391 \item Copy external data: The IPP-PDS generates {\em retrieve data} 392 tasks which are executed on nodes specified by the IPP-DLO. This 393 mode will be used frequently to copy data from the Summit Pixel 394 Server to the IPP nodes for processing. 395 \item Delete data: The IPP-PDS looks up the location of the data in 396 the IPP Pixel Data Database and generates {\em delete data} tasks 397 which are executed on the appropriate nodes. This mode will be used 398 on a regular basis to clean old data that is no longer required. 399 \item Replicate data: The IPP-PDS generates {\em retrieve data} tasks 400 which are executed on nodes specified either by the ``replicate 401 data'' command, or by the IPP-DLO. This mode differs from the 402 ``copy external data'' mode in that it copies data already within 403 the IPP-PDS. This mode will be used to backup and rearrange data. 404 \item Move data: the IPP-PDS executes a replication followed by a 405 deletion. This mode will be used to reorganise the storage. 368 406 \end{itemize} 369 407 408 It is not intended that the IPP-PDS will be used by the nodes in the 409 course of processing --- it is only for bulk data management. ``Copy 410 external data'' mode will be used frequently to retrieve data from the 411 Summit Pixel Server. ``Delete data'' mode will be used on a regular 412 basis to flush the system of stale files. It is expected that the 413 other modes will be used only occassionally, and initiated by a human 414 operator. 415 416 370 417 \subparagraph{IPP Pixel Data Locality Optimizer (IPP-PDLO)} 371 418 372 The IPP Pixel Data Locality Optimizer is a data task filter that registers with373 t he IPP Pixel Data Scheduler. Data tasks generated by the IPP Pixel Data374 Scheduler are passed through the IPP Pixel Data Locality Optimizer which may 375 assign tasks to specific nodes. This component is a merely a plug-in and maybe 376 bypassed dependingon the operating mode of the IPP Pixel Data Scheduler.419 The IPP Pixel Data Locality Optimizer is a data task filter. Data 420 tasks generated by the IPP Pixel Data Scheduler are passed through the 421 IPP Pixel Data Locality Optimizer which may assign tasks to specific 422 nodes. This component is a merely a plug-in and may be bypassed 423 depending upon the operating mode of the IPP Pixel Data Scheduler. 377 424 378 425 \subparagraph{IPP Pixel Data Database (IPP-PDD)} 379 426 380 The IPP Pixel Data Database contains image data locations and the associated381 meta-data. 427 The IPP Pixel Data Database contains image data locations \tbd{and the 428 associated metadata}. 382 429 383 430 The IPP-PDD will contain at least: 384 431 385 432 \begin{itemize} 386 \item The location of image data and it 's associated meta-data that is433 \item The location of image data and its associated metadata that is 387 434 available for retrieval from the Summit Pixel Server. 388 \item The location of image data and it 's associated meta-data that is available389 for processing withinthe IPP System.390 \item The location of calibration data and it 's associated meta-data for435 \item The location of image data and its associated metadata that is 436 yet to be processed by the IPP System. 437 \item The location of calibration data and its associated metadata for 391 438 processing within the IPP System. 392 \item The location of reduced image data and it 's associated meta-data as439 \item The location of reduced image data and its associated metadata as 393 440 generated by the IPP System. 394 \item The location of difference image data and it 's associated meta-data as441 \item The location of difference image data and its associated metadata as 395 442 generated by the IPP System. 396 \item The location of stacked image data and it 's associated meta-data as443 \item The location of stacked image data and its associated metadata as 397 444 generated by the IPP System. 398 445 \item A history of data management commands and actions. … … 401 448 \subparagraph{IPP Pixel Data Retrieval Agent (IPP-PDRA)} 402 449 403 The IPP Pixel Data Retrieval Agent acquires image data from a specified location,404 possibly the Summit Pixel Server(s), and stores it at a specified location. 405 The IPP-PDRA attempts to be independent of the underlying storage medium by 406 u sing the IPP Pixel Data I/O Library.407 408 \subparagraph{IPP Pixel Data Query Library (IPP-PDQL)} 409 410 The IPP Pixel Data Query Library provides an interface to the IPP Pixel Data 411 Database while hiding the implementation details (ie. the SQL queries). 412 413 It will be able to: 414 450 The IPP Pixel Data Retrieval Agent acquires image data from a 451 specified location, possibly the Summit Pixel Server(s), and stores it 452 at a specified location. The IPP-PDRA is independent of the 453 underlying storage medium by using the IPP Pixel Data I/O Library. 454 455 456 \subparagraph{IPP Pixel Data I/O Library (IPP-PDIOL)} 457 458 The PDIOL is the workhorse of the Pixel Server system. It is a 459 library for retrieving files from and storing files to Uniform 460 Resource Identifiers (URIs), which can be used on the nodes to access 461 the pixel data. It will be able to: 415 462 \begin{itemize} 416 \item Locate new and reduced data for a sky cell. 417 \item Locale the latest calibration data for sky cell. 418 \item Add the storage location and meta-data of new data. 419 \item Update the storage location and/or meta-data of any data. 420 \item Remove the storage location of data and meta-data that has been deleted. 463 \item Locate new and reduced data for an exposure. 464 \item Locate the appropriate calibration data for an exposure. 465 \item Add the storage location and metadata of new data. 466 \item Update the storage location and/or metadata of any data. 467 \item Remove the storage location of data and metadata that has been 468 deleted. 421 469 \end{itemize} 422 470 423 \subparagraph{IPP Pixel Data I/O Library (IPP-PDIOL)} 424 425 A library for retrieving files from and storing files to URIs. 471 426 472 427 473 \paragraph{Pixel Data Flow} … … 430 476 431 477 \begin{enumerate} 432 \item The Summit Pixel Server sends a new data notificationto the478 \item The Summit Pixel Server sends a ``new data notification'' to the 433 479 IPP Pixel Data Data Scheduler. 434 \item The IPP Pixel Data Data Scheduler generates a {\em retrieve data} task 435 which is passed to the IPP Pixel Data Locality Optimizer. 436 \item The IPP Pixel Data Locality Optimizer possibly assigns the task 437 to a specific node or group of nodes and passes it on to the IPP Controller. 438 \item The IPP Controller passes the task to a \tbd{IPP Node Agent}. 439 \item The \tbd{IPP Node Agent} spawns a IPP Pixel Data Retrieval Agent 440 and passes it the task. 441 \item The IPP Pixel Data Retrieval Agent downloads the image data from the 442 Summit Pixel Server. 443 \item The IPP Pixel Data Retrieval Agent reports successful task completion 444 to the \tbd{IPP Node Agent}. 445 \item The \tbd{IPP Node Agent} reports the finished task to the IPP Controller. 446 \item The IPP Controller reports the finished task to the IPP Pixel Data Scheduler. 480 \item The IPP Pixel Data Data Scheduler generates a {\em retrieve 481 data} task which is filtered through the IPP Pixel Data Locality 482 Optimizer, which possibly assigns the task to a specific node or group 483 of nodes. 484 \item The IPP Pixel Data Scheduler farms out the various copy tasks to 485 the nodes, which spawn IPP Pixel Data Retrieval Agents. 486 \item The IPP Pixel Data Retrieval Agents downloads the image data 487 from the Summit Pixel Server to the disk physically mounted on the 488 node. 489 \item The node reports the finished task to the IPP Pixel Data Scheduler. 447 490 \item The IPP Pixel Data Scheduler updates the IPP Pixel Data Database to 448 491 the new storage location. … … 517 560 additional analysis. The Metadata Database may potentially be used in 518 561 close coupling with the analysis pipelines to store temporary data 519 either within stages of the analysis or between pipeline stages. In 520 this scenario, the analysis pipeline will interact directly with the 521 database. However, database latency may make this scenario 522 impractical, in which case the database may be used for long-term 523 storage only. In this scenario, the data produced by analysis 524 pipelines which is destined for the Metadata Database may be collected 525 and inserted by a separate, dedicated process or analysis pipeline 526 collection of processes. 562 either within or between stages of the analysis. In this scenario, 563 the analysis pipeline will interact directly with the database. 564 However, database latency may make this scenario impractical, in which 565 case the database may be used for long-term storage only. In this 566 scenario, the data produced by analysis pipelines which is destined 567 for the Metadata Database may be collected and inserted by a separate, 568 dedicated process or analysis pipeline collection of processes. 527 569 528 570 \paragraph{Metadata Tables} 529 571 530 Table NNlists the Metadata tables identified for the Metadata572 Table \tbd{NN} lists the Metadata tables identified for the Metadata 531 573 Database. 532 574 … … 562 604 \paragraph{Metadata Table Contents} 563 605 564 Tables NN -- NNlist the basic contents of each of the Metadata tables606 Tables \tbd{NN} -- \tbd{NN} list the basic contents of each of the Metadata tables 565 607 listed above. 566 608 … … 938 980 \subsubsection{Controller} 939 981 940 The IPP Controller is responsible for executing the connecting the941 low-level functions together to define the various processing 942 subsystems. The Controller manages the parallel processing of these 943 subsystems in the IPP computer hardware environment and reportsthe944 processing status to the IID. The Controller must be able to manage945 more than a single processing thread to make maximum use of available 946 processor resources. Some analysis jobs, such as operations on the 947 OTAs, must be allocated preferentially to specified processors, while 948 others mustbe distributed to the available machines in the cluster.982 The IPP Controller is responsible for connecting the low-level modules 983 together to define the various processing subsystems. The Controller 984 manages the parallel processing of these subsystems in the IPP 985 computer hardware environment and reports the processing status to the 986 IMD. The Controller must be able to manage more than a single 987 processing thread to make maximum use of available processor 988 resources. Some analysis jobs, such as operations on the OTAs, must 989 be allocated preferentially to specified processors, while others must 990 be distributed to the available machines in the cluster. 949 991 950 992 \paragraph{Components} 951 993 952 The Controller consists of N components: the Controller daemon, the953 remote clients, and the user clients. 994 The Controller consists of the following components: the Controller 995 daemon, the remote clients, and the user clients. 954 996 955 997 The Controller daemon maintains a table of processing nodes available … … 978 1020 The commands include: 979 1021 980 {\bf \em report status} return the state of the client (idle, busy, 981 done), the state of the current job (none, busy, crash, done), and the 982 exit status of the current job (none, 0-256). The three states of the 983 client indicate that the client has no current job (idle), that it has 984 a job which is still running (busy), and that it has a job which has 985 completed. The job states indicate the there is no current job 986 (none), that the current job is running (busy), that the current job 987 has crashed (crash), and that the current job has exited gracefully 988 (done). The exit state is the exit state reported by the job (0-256 989 with 0 indicating a successful completion) or is an indication that 990 there is no current job (none). 991 992 {\bf \em report stdout} Send and flush the current stdout buffer. The 1022 {\bf \em report status}: Return the state of the client (idle, busy, 1023 done), the state of the current job\footnote{Note that a job is 1024 considered ``current'' until it is cleared with {\em clear job} --- 1025 even if it has crashed or completed.} (`none', `busy', `crash', 1026 `done'), and the exit status of the current job (`none', 0--256). The 1027 three states of the client indicate that the client has no current job 1028 (`idle'), that it has a job which is still running (`busy'), and that 1029 it has a job which has completed. The job states indicate the there 1030 is no current job (`none'), that the current job is running (`busy'), 1031 that the current job has crashed (`crash'), and that the current job 1032 has exited gracefully (`done'). The exit state is the exit state 1033 reported by the job (0--256 with 0 indicating a successful completion) 1034 or is an indication that there is no current job (`none'). 1035 1036 {\bf \em report stdout}: Send and flush the current stdout buffer. The 993 1037 remote client will return the complete contents of the stdout buffer 994 1038 via a buffered write and flush the buffer when it is finished. The … … 997 1041 daemon must accept all of the buffer output. 998 1042 999 {\bf \em report stderr} Identical to 'report stdout' for stderr.1000 1001 {\bf \em kill job} remote client should send a kill signal to the1043 {\bf \em report stderr}: Identical to `report stdout' for stderr. 1044 1045 {\bf \em kill job}: remote client should send a kill signal to the 1002 1046 current job. When the job has exited, the remote client should set 1003 the job status to crash and the client status to done.1004 1005 {\bf \em clear job} The remote client should set the current job state1006 to 'none' and the client state to 'idle'. If a job is currently1047 the job status to `crash' and the client status to `done'. 1048 1049 {\bf \em clear job}: The remote client should set the current job state 1050 to `none' and the client state to `idle'. If a job is currently 1007 1051 running, it should be killed before the job is cleared. 1008 1052 1009 {\bf \em start job [command]} execute the given command. The command1053 {\bf \em start job [command]}: execute the given command. The command 1010 1054 should be a standard unix command without command line redirection or 1011 1055 backgrounding. … … 1041 1085 and for initiating the various processing systems, executed by the IPP 1042 1086 Controller, based on the state of the survey as reflected by the IPP 1043 Internal Database (IID). The Scheduler must send calibration data1087 Metadata Database (IMD). The Scheduler must send calibration data 1044 1088 requests to the PTS, including required flat-field images, flat-field 1045 1089 correction observations, or other specialized observations needed to … … 1048 1092 timely manner given the capabilities of the science pipelines. 1049 1093 1050 The scheduler is a subsystem which defines the tasks that the pipeline1051 needs to perform at any given time. The scheduler takes input1052 information which describes the collection of all tasks which may need1053 to be performed, along with information about their requirements in1054 terms of specific data (images / entries in database tables). The1055 scheduler decides which tasks to perform at any moment based on the1056 current state of the pixel and metadata databases, by confronting the1057 task descriptions and task requirements with the existence of data in1058 the databases.1059 1060 1094 \tbd{how are the schedules defined? how are dependencies between jobs 1061 1062 1095 defined? scheduler must communicate with the controller (as a user 1096 client) to send new jobs}. 1063 1097 1064 1098 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% … … 1072 1106 The IPP science image pipelines perform analyses on the night-sky 1073 1107 science images to extract the science data from these images. These 1074 consist of: Phase 0, the night preparation stage; Phase 1, the image1075 processing preparation stage; Phase 2, the image reduction stage; 1076 Phase 3, the exposure analysis stage; and Phase 4, the image1077 combination stage. These pipelines must process the images in a 1078 timely manner so that the incoming data stream will not overload the 1079 IPS. The decision to execute a specific pipeline for a specific 1080 dataset is made by the Scheduler, which sends the infomation to the1081 Controller. The Controller executes the pipeline for the data on an 1082 appropriate machine and monitors the success orfailure of the job.1108 consist of: Phase 1, the image processing preparation stage; Phase 2, 1109 the image reduction stage; Phase 3, the exposure analysis stage; and 1110 Phase 4, the image combination stage. These pipelines must process 1111 the images in a timely manner so that the incoming data stream will 1112 not overload the IPS. The decision to execute a specific pipeline for 1113 a specific dataset is made by the Scheduler, which sends the 1114 infomation to the Controller. The Controller executes the pipeline 1115 for the data on an appropriate machine and monitors the success or 1116 failure of the job. 1083 1117 1084 1118 \paragraph{Calibration Image Pipelines} … … 1095 1129 \paragraph{Reference Catalog Pipelines} 1096 1130 1097 The IPP reference catalog pipelines use the data in the IPP Internal1131 The IPP reference catalog pipelines use the data in the IPP Metadata 1098 1132 Database and the IPP Object Database to determined improved 1099 1133 astrometric and photometric calibration references. … … 1126 1160 used by the later stages to initiate the analyses. 1127 1161 1128 The phase 1 analysis is performed on a FPA basis to ensure that enough1129 reference stars are available for the astrometry calculation. Phase 1 1130 cannot be usefully calculated on the basis of a major frame since the1131 telescope positions are independent; no additional information is 1132 available by combining stars from different FPAs. This analysis does1133 not restrict the definition of a major frame in any way.1134 1135 \ note{Phase 1 command: P1 (exposure)}1136 1137 \ note{Megacam: P1 654321o}1162 The phase 1 analysis is performed on an FPA basis to ensure that 1163 enough reference stars are available for the astrometry calculation. 1164 Phase 1 cannot be usefully calculated on the basis of a major frame 1165 since the telescope positions are independent; no additional 1166 information is available by combining stars from different FPAs. This 1167 analysis does not restrict the definition of a major frame in any way. 1168 1169 \tbd{Phase 1 command: P1 (exposure)} 1170 1171 \tbd{Megacam: P1 654321o} 1138 1172 1139 1173 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% … … 1151 1185 \begin{figure} 1152 1186 \begin{center} 1153 \resizebox{8cm}{!}{\includegraphics{pics/phase2 .ps}}1187 \resizebox{8cm}{!}{\includegraphics{pics/phase2}} 1154 1188 \caption{ \label{phase2} Phase 2 dataflow} 1155 1189 \end{center} … … 1158 1192 \paragraph{Phase 2 Concept} 1159 1193 1160 Phase~2 processing within the Pan-STARRSimage processing pipeline is1194 Phase~2 processing within the \PS{} image processing pipeline is 1161 1195 the de-trend stage, where the images from the detector are processed 1162 1196 to remove instrumental signatures. Phase~2 processing is purely serial, … … 1167 1201 the guide stars and initial masking of ghost reflections. 1168 1202 1169 Phase~2 consists of the following tasks:1203 Phase~2 consists of the following modules: 1170 1204 \begin{enumerate} 1171 1205 \item Form OT kernel; 1172 1206 \item Convolve de-trend images with the OT kernel; 1173 \item bias / dark / Overscan subtraction; 1174 \item Trim; 1207 \item Mask bad pixels 1208 \item Mask diffraction spikes and optical ghosts; 1209 \item Bias/dark/overscan subtraction; 1210 \item Trim overscan; 1175 1211 \item Non-linearity correction; 1176 1212 \item Flat-field; 1177 1213 \item Subtract sky; 1178 1214 \item Identify CRs by morphology; 1179 \item Find objects in the image; and 1215 \item Determine PSF model; 1216 \item Find and photometer objects in the image; 1217 \item Improved astrometry; and 1180 1218 \item Bright object postage stamps. 1181 \item {\em from old version:} 1182 \item mask bad pixels 1183 \item remove diffraction spikes 1184 \item remove ghosts 1185 \item remove cosmic rays 1186 \item estimate foreground 1187 \item subtract foreground 1188 \item extract objects, photometry 1189 \item determine PSF model 1190 \item improved astrometry based on comparison with references. 1191 \end{enumerate} 1192 These tasks are each explained below. 1219 \end{enumerate} 1220 These modules are each explained below. 1193 1221 1194 1222 \paragraph{Form OT Kernel} 1195 1223 1196 The first taskfor Phase~2 is to form the OT kernel from the image1224 The first module for Phase~2 is to form the OT kernel from the image 1197 1225 metadata of pixel shifts made during the exposure. This involves 1198 1226 decoding the metadata and converting it to a data type that can be … … 1202 1230 \paragraph{Convolve de-trend images} 1203 1231 1204 This taskconvolves the de-trend images with the OT convolution kernel1232 This module convolves the de-trend images with the OT convolution kernel 1205 1233 so that they can be used to de-trend the object image. The inputs 1206 1234 are: 1207 1235 \begin{enumerate} 1208 \item The OT convolution kernel --- from the previous task;1236 \item The OT convolution kernel --- from the previous module; 1209 1237 \item The appropriate dark frame --- from the IPP Pixel Server; 1210 1238 \item The appropriate flat-field --- from the IPP Pixel Server; … … 1213 1241 \end{enumerate} 1214 1242 1215 The taskconvolves each of the dark frame, flat-field, and the fringe1243 The module convolves each of the dark frame, flat-field, and the fringe 1216 1244 frame(s) by the OT convolution kernel. Specific flags in the static 1217 1245 bad pixel mask are grown by the outline of the OT convolution kernel 1218 (see Appendix\ref{ap:masks}). The output results are:1246 (see Section \ref{ap:masks}). The output results are: 1219 1247 \begin{enumerate} 1220 1248 \item The convolved flat-field; … … 1222 1250 \item The updated pixel mask. 1223 1251 \end{enumerate} 1224 Each of these will be used for a later task.1252 Each of these will be used for a later module. 1225 1253 1226 1254 1227 1255 \paragraph{Overscan Subtraction} 1228 1256 1229 This taskcorrects the object exposures for the electronic pedestal1257 This module corrects the object exposures for the electronic pedestal 1230 1258 introduced by the readout electronics. The inputs are: 1231 1259 \begin{enumerate} 1232 1260 \item The object image --- from the IPP Pixel Server; 1233 \item The pixel mask --- from the previous task;1261 \item The pixel mask --- from the previous module; 1234 1262 \item The overscan and physical detector regions --- from the 1235 1263 Metadata; and … … 1242 1270 Overscan rows having a standard deviation which exceeds a threshold of 1243 1271 twice (configurable) the detector read noise should be masked. Pixels 1244 saturated in the A/D converter should also be masked, and these regions 1245 grown by an additional pixel. The output is: 1272 saturated in the A/D converter should also be masked, and these 1273 regions grown by an additional pixel to counter CCD ``blooming''. The 1274 output is: 1246 1275 \begin{enumerate} 1247 1276 \item The overscan-subtracted object image; and 1248 1277 \item The updated pixel mask. 1249 1278 \end{enumerate} 1250 These will be used for a subsequent task.1279 These will be used for a subsequent module. 1251 1280 1252 1281 \paragraph{Trim} 1253 1282 1254 This tasktrims the object image and each of the calibration frames to1283 This module trims the object image and each of the calibration frames to 1255 1284 remove the outer edge which was affected by the OT during the 1256 exposure. The inputs, each from previous tasks, are:1285 exposure. The inputs, each from previous modules, are: 1257 1286 \begin{enumerate} 1258 1287 \item The overscan-subtracted object image; 1259 1288 \item The corresponding pixel mask; 1260 \item The convolved dark frame;1261 1289 \item The convolved flat-field; 1262 1290 \item The convolved fringe frame(s); and … … 1264 1292 \end{enumerate} 1265 1293 1266 Each of the input frames (object image, dark frame, flat-field, fringe1267 frame(s) and pixel mask) are trimmed by the extent of the OT 1268 convolution kernel in each direction ($+x$, $-x$, $+y$, $-y$). The 1269 outputs are trimmed images for each of the input images, which will be 1270 used in later tasks.1294 Each of the input frames (object image, flat-field, fringe frame(s) 1295 and pixel mask) are trimmed by the extent of the OT convolution kernel 1296 in each direction ($+x$, $-x$, $+y$, $-y$). The outputs are trimmed 1297 images for each of the input images, which will be used in later 1298 modules. 1271 1299 1272 1300 \paragraph{Non-Linearity Correction} 1273 1301 1274 This taskcorrects images for non-linearity in the detector. The1302 This module corrects images for non-linearity in the detector. The 1275 1303 inputs are: 1276 1304 \begin{enumerate} 1277 \item The trimmed object image --- from a previous task; and1305 \item The trimmed object image --- from a previous module; and 1278 1306 \item The detector non-linearity correction coefficient(s) --- from 1279 1307 the Metadata. 1280 1308 \end{enumerate} 1281 1309 1282 The taskcorrects the flux in each pixel for non-linearity by applying1310 The module corrects the flux in each pixel for non-linearity by applying 1283 1311 a polynomial correction, with the specified coefficients. The output 1284 is the corrected object image, which is used for a later task.1312 is the corrected object image, which is used for a later module. 1285 1313 1286 1314 \paragraph{Flat field} 1287 1315 1288 This taskcorrects the object image for variations in sensitivity over1316 This module corrects the object image for variations in sensitivity over 1289 1317 the image. The inputs are: 1290 1318 \begin{enumerate} … … 1293 1321 \item The convolved, trimmed flat-field. 1294 1322 \end{enumerate} 1295 Each of these comes from a previous task.1296 1297 The taskdivides the object image by the flat-field, masking pixels1323 Each of these comes from a previous module. 1324 1325 The module divides the object image by the flat-field, masking pixels 1298 1326 that are non-positive in the flat-field. The outputs are: 1299 1327 \begin{enumerate} … … 1301 1329 \item The updated pixel mask. 1302 1330 \end{enumerate} 1303 Both of these will be used in later tasks.1331 Both of these will be used in later modules. 1304 1332 1305 1333 \paragraph{Subtract sky} 1306 1334 1307 This tasksubtracts the sky background from the object image. The1335 This module subtracts the sky background from the object image. The 1308 1336 inputs are: 1309 1337 \begin{enumerate} 1310 \item The object image --- from the previous task;1338 \item The object image --- from the previous module; 1311 1339 \item The list of objects on the image --- from the object database; and 1312 \item The convolved, trimmed fringe frame(s) --- from a previous task.1313 \end{enumerate} 1314 1315 The taskmasks (though {\em not} in the ``official'' pixel mask) all1340 \item The convolved, trimmed fringe frame(s) --- from a previous module. 1341 \end{enumerate} 1342 1343 The module masks (though {\em not} in the ``official'' pixel mask) all 1316 1344 objects on the image using the astrometric solution from the 1317 1345 boresight, and fits for the sky background, consisting of a polynomial … … 1320 1348 is too high to reliably fit the sky background, the background 1321 1349 solution from an exposure close in time and airmass to the current 1322 object image. The output is the sky-subtracted object image, which is 1323 sent to the IPP pixel server for use in Phase~3, and also used for the 1324 next task. 1350 object image is used. The output is the sky-subtracted object image, 1351 which is used for the next module. 1325 1352 1326 1353 \paragraph{Identify CRs by morphology} 1327 1354 1328 This taskidentifies cosmic rays (or other hot pixels missed in the1355 This module identifies cosmic rays (or other hot pixels missed in the 1329 1356 static bad pixel mask) on the basis of their morphology. The inputs 1330 1357 are: … … 1333 1360 \item The corresponding pixel mask. 1334 1361 \end{enumerate} 1335 Both of these come from a previous task.1336 1337 The taskidentifies CRs, the pixels of which are masked in the pixel1362 Both of these come from a previous module. 1363 1364 The module identifies CRs, the pixels of which are masked in the pixel 1338 1365 mask. The pixels flagged as CRs are then grown by an additional pixel 1339 in each direction. The output is the updated pixel mask, which is 1340 sent to the IPP pixel server for use in Phase~3, and is also used for 1341 the next task. 1366 in each direction. Masked pixels are interpolated over. The outputs 1367 are the updated pixel mask, which is sent to the IPP pixel server for 1368 use in Phase~3, and is also used for the next module; and the object image, 1369 which is sent to the IPP Pixel Server. 1342 1370 1343 1371 \paragraph{Find objects} 1344 1372 1345 This taskfinds objects on the object image. The inputs are:1373 This module finds objects on the object image. The inputs are: 1346 1374 \begin{enumerate} 1347 1375 \item The sky-subtracted object image; and 1348 1376 \item The corresponding pixel mask. 1349 1377 \end{enumerate} 1350 Both of these come from a previous task.1351 1352 The taskidentifies objects on the image, which will be later used to1378 Both of these come from a previous module. 1379 1380 The module identifies objects on the image, which will be later used to 1353 1381 register images from different focal planes. The output is the 1354 catalog ue of objects (see Appendix~\ref{ap:catalogues}) identified on1382 catalog of objects (see Appendix~\ref{ap:catalogs}) identified on 1355 1383 the image, which is sent to the metadata database, associated with the 1356 1384 object image. … … 1358 1386 \paragraph{Bright object postage stamps} 1359 1387 1360 This tasksaves postage stamps of bright objects, so that extra care1388 This module saves postage stamps of bright objects, so that extra care 1361 1389 with regard to astrometry and photometry can be taken with them at a 1362 later stage. The inputs, each from a previous task, are:1390 later stage. The inputs, each from a previous module, are: 1363 1391 \begin{enumerate} 1364 1392 \item The sky-subtracted object image; 1365 1393 \item The corresponding pixel mask; and 1366 \item The catalog ueof objects.1367 \end{enumerate} 1368 1369 The taskmakes postage stamps of all objects brighter than a given1394 \item The catalog of objects. 1395 \end{enumerate} 1396 1397 The module makes postage stamps of all objects brighter than a given 1370 1398 instrumental magnitude, along with corresponding pixel masks. The 1371 1399 outputs are these postage stamps and pixel masks, which are sent to … … 1385 1413 detrend images; 1386 1414 \item Exposure time --- for the photometric calibration; 1387 \item Detector gain --- for calculating photometric errors and 1388 determining the quality of the overscan; 1415 \item Detector gain --- for calculating photometric errors; and 1389 1416 \item Detector read noise --- for calculating photometric errors and 1390 1417 determining the quality of the overscan; … … 1395 1422 1396 1423 This section describes the requirements on Bad Pixel Masks (BPMs). 1397 These will consist inof bit masks for each pixel. For Phase 2, flags1424 These will consist of bit masks for each pixel. For Phase 2, flags 1398 1425 are required for at least each of the following pixel attributes: 1399 1426 \begin{enumerate} … … 1412 1439 affect the flux in neighbouring pixels 1413 1440 1414 \paragraph{Object Catalog ues}1415 \label{ap:catalog ues}1416 1417 Object catalog ues from Phase 2 shall consist of at least the1441 \paragraph{Object Catalogs} 1442 \label{ap:catalogs} 1443 1444 Object catalogs from Phase 2 shall consist of at least the 1418 1445 following elements for each object: 1419 1446 \begin{enumerate} … … 1426 1453 \end{enumerate} 1427 1454 1428 Though further details may be required for catalog ues in Phase~4,1429 the above details are minimum requirements for Phase~2 catalog ues.1430 1431 \ note{Phase 2 command: P2 (exposure.ota.fits)}1432 \ note{Megacam: P2 654321o.fits[ccd00] - what are output names?}1433 \ note{PS FPA is saved as a collection of MEF files. Megacam FPA is1455 Though further details may be required for catalogs in Phase~4, 1456 the above details are minimum requirements for Phase~2 catalogs. 1457 1458 \tbd{Phase 2 command: P2 (exposure.ota.fits)} 1459 \tbd{Megacam: P2 654321o.fits[ccd00] - what are output names?} 1460 \tbd{PS FPA is saved as a collection of MEF files. Megacam FPA is 1434 1461 saved as a single MEF file. how to handle this difference?} 1435 1462 … … 1439 1466 \begin{figure} 1440 1467 \begin{center} 1441 \resizebox{8cm}{!}{\includegraphics{pics/phase3 .ps}}1468 \resizebox{8cm}{!}{\includegraphics{pics/phase3}} 1442 1469 \caption{ \label{phase3} Phase 3 dataflow} 1443 1470 \end{center} 1444 1471 \end{figure} 1445 1472 1446 Phase 3 : image processing preparation 1447 1448 The Phase 3 system operates on the combined Phase 2 results from a 1449 collection of FPA images to determine improved solutions for the image 1450 calibrations and to provide the parameters needed by Phase 4. The 1451 Phase 3 output is saved by the IID, and consists largely of improved 1452 values of the calibrations already determined by Phase 2. The 1453 analysis performed by this pipeline consists of: 1473 The Phase 3 system operates on the combined Phase 2 results from an 1474 FPA to determine improved solutions for the image calibrations and to 1475 provide the parameters needed by Phase 4. The Phase 3 output is saved 1476 by the IMD, and consists largely of improved values of the 1477 calibrations already determined by Phase 2. The analysis performed by 1478 this pipeline consists of: 1454 1479 1455 1480 \begin{itemize} … … 1460 1485 \item photometric solution based on comparison to photometric 1461 1486 standards 1462 \item PSF convolution kernels to transform images to a common PSF.1463 1487 \end{itemize} 1464 1488 … … 1466 1490 independently for each OTA. These solutions are limited by the 1467 1491 assumption of a static distortion and \tbd{by the accuracy of the 1468 1469 solutions of the N FPA images are improved by ???1492 astrometric reference}. In the phase 3 analysis, the astrometric 1493 solutions of the N FPA images are improved by \tbd{???}. 1470 1494 1471 1495 \tbd{what is the expected accuracy of the relative astrometric … … 1488 1512 absolute photometry solution? (probably)} 1489 1513 1490 In the Phase 4 analysis, N FPA images are optimally combined to create 1491 a single image of the sky with bad-pixel and cosmic-ray rejection. 1492 This combination requires the calculation of a set of PSF kernels to 1493 convert each of the input images to a single, common PSF. These PSF 1494 kernels are determined from the per-OTA PSFs measured in Phase 2. 1514 In the Phase 4 analysis, the $N$ FPA images are optimally combined to 1515 create a single image of the sky with bad-pixel and cosmic-ray 1516 rejection. This combination requires the calculation of a set of PSF 1517 kernels to convert each of the input images to a single, common PSF. 1518 These PSF kernels are determined from the per-OTA PSFs measured in 1519 Phase 2. 1495 1520 1496 1521 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% … … 1499 1524 \begin{figure} 1500 1525 \begin{center} 1501 \resizebox{8cm}{!}{\includegraphics{pics/phase4 .ps}}1526 \resizebox{8cm}{!}{\includegraphics{pics/phase4}} 1502 1527 \caption{ \label{phase4} Phase 4 dataflow} 1503 1528 \end{center} … … 1506 1531 \paragraph{Phase 4 Concept} 1507 1532 1508 Phase 4 processing within the Pan-STARRSimage processing pipeline is1533 Phase 4 processing within the \PS{} image processing pipeline is 1509 1534 the final stage of processing for a science image. It operates on 1510 1535 each sky cell that has overlapping imaging data from the exposure(s) 1511 1536 being processed, and produces the main output image data products of 1512 1537 the pipeline --- the difference images and a deep static sky image --- 1513 along with the associated catalog ues of static and variable sources.1538 along with the associated catalogs of static and variable sources. 1514 1539 1515 1540 Prior to Phase 4, the Phase 3 process produces the following products: … … 1518 1543 \item photometric calibration; 1519 1544 \item astrometric calibration with mapping to sky cells; and 1520 \item PSF models for the images.1521 1545 \end{itemize} 1522 These will each be used by the Phase 4 tasks:1546 These will each be used by the Phase 4 modules: 1523 1547 \begin{enumerate} 1524 1548 \item Combine Images; … … 1527 1551 \item Add to Static Sky. 1528 1552 \end{enumerate} 1529 These tasks are each explained below.1553 These modules are each explained below. 1530 1554 1531 1555 \paragraph{Combine Images} 1532 1556 1533 The first taskfor Phase 4 is to combine the images from each1557 The first module for Phase 4 is to combine the images from each 1534 1558 telescope, rejecting artifacts such as cosmic rays and low altitude 1535 streaks. The inputs to this taskare:1559 streaks. The inputs to this module are: 1536 1560 \begin{enumerate} 1537 1561 \item the sky-subtracted images that overlap the sky cell (or portions 1538 1562 thereof) --- from the IPP Pixel Server (or directly from Phase 3); 1539 \item a (linear) map for the image pixels of each detector to the sky1540 cell pixels --- from Phase 3;1563 \item a \tbd{linear} map for the image pixels of each detector to the 1564 sky cell pixels --- from Phase 3; 1541 1565 \item photometric calibration (zeropoint) for each image --- from 1542 1566 Phase 3; and … … 1546 1570 \end{enumerate} 1547 1571 1548 The taskmaps the detector images to the sky cell using the specified1572 The module maps the detector images to the sky cell using the specified 1549 1573 linear transformations, combines the images with strong rejection 1550 1574 criteria and uses the combined sky cell image to identify artifacts in 1551 1575 the original detector images. It is desirable that the artifacts are 1552 1576 masked in the detector plane (i.e.\ before mapping to the sky cell) so 1553 that they are not smeared out by the mapping. The masked detector 1554 images are then mapped to the sky cell and optimally combined using 1555 the specified weighting. Both sets of combinations use the 1556 photometric calibration for the images to set the relative scales of 1557 the input images. The final combination should have the adopted 1558 Universal zeropoint (25 mag, configurable). 1559 1560 A PSF model for the combined sky cell image should be made by 1561 identifying point sources in the combined image, scaling and stacking 1562 them to achieve high signal-to-noise, and fitting with an analytic 1563 functional form (e.g. Gaussian, Moffat, Waussian). The limiting 1564 magnitude for the combined sky cell image should also be estimated. 1565 1566 The outputs from this task are: 1577 that they are not smeared out by the mapping; alternatively, the CR 1578 mask needs to be grown by an additional pixel (which is likely 1579 faster). The mapped and masked detector images are then optimally 1580 combined using the specified weighting. Both sets of combinations use 1581 the photometric calibration for the images to set the relative scales 1582 of the input images. The final combination should have the adopted 1583 Universal zeropoint (25 mag, configurable). The limiting magnitude 1584 for the combined sky cell image should also be estimated. 1585 1586 The outputs from this module are: 1567 1587 \begin{enumerate} 1568 1588 \item The combined sky cell image --- sent to the IPP Pixel Server 1569 and/or the next task; 1570 \item PSF model for the combined sky cell image --- metadata 1571 associated with the combined sky cell image, and used for the other 1572 tasks in Phase 4; 1589 and/or the next module; 1573 1590 \item Limiting magnitude of the combined sky cell image --- metadata 1574 associated with the combined sky cell image, and used for a later task1591 associated with the combined sky cell image, and used for a later module 1575 1592 in Phase 4; and 1576 \item Catalog ueof sources on the combined sky cell image --- sent to1593 \item Catalog of sources on the combined sky cell image --- sent to 1577 1594 the IPP Object Database. 1578 1595 \end{enumerate} … … 1581 1598 \paragraph{Identify Sources} 1582 1599 1583 This task identifies sources in the combined sky cell image. The 1584 inputs are: 1585 \begin{enumerate} 1586 \item The combined sky cell image --- from the IPP Pixel Server 1587 or the previous task; 1588 \item PSF model for the combined sky cell image --- metadata 1589 associated with the combined sky cell image, from the previous task; 1590 \end{enumerate} 1600 This module identifies sources in the combined sky cell image. The 1601 input is the combined sky cell image, which is obtained from the IPP Pixel Server 1602 or the previous module. 1591 1603 1592 1604 Sources are identified on the combined sky cell image by convolving 1593 with the PSF model and searching for peaks above the noise. The output 1594 is: 1595 \begin{enumerate} 1596 \item Catalogue of sources on the combined sky cell image --- sent to 1605 with the PSF and searching for peaks above the noise. The output 1606 is the catalog of sources on the combined sky cell image, which is to 1597 1607 the IPP Object Database. 1598 \end{enumerate}1599 1608 1600 1609 1601 1610 \paragraph{Transient Identification} 1602 1611 1603 This task identifies variable/moving sources. The inputs are: 1604 \begin{enumerate} 1605 \item The combined sky cell image --- from the previous task or the 1606 IPP Pixel Server; 1607 \item The PSF model for the combined sky cell image from the previous 1608 task --- from the Metadata database, or the previous task; 1609 \item The current static sky image --- from the Sky Image Server; and 1610 \item The PSF model for the static sky image --- from the metadata or 1611 the Sky Image Server. 1612 \end{enumerate} 1613 1614 The task subtracts the current static sky image from the combined sky 1612 This module identifies variable/moving sources. The inputs are: 1613 \begin{enumerate} 1614 \item The combined sky cell image --- from the previous module or the 1615 IPP Pixel Server; and 1616 \item The current static sky image --- from the Sky Image Server. 1617 \end{enumerate} 1618 1619 The module subtracts the current static sky image from the combined sky 1615 1620 cell image. In order to do so, the PSFs need to be matched. This is 1616 1621 done by convolving the image that has the narrower PSF with the 1617 1622 kernel, which is the ratio of the two PSFs (this should be done with a 1618 fit to the PSFs instead of just using the data). It should be 1619 sufficient to assume that the kernel is constant over the sky cell. 1623 fit to the kernel instead of just using the data). It should be 1624 sufficient to assume that the kernel is constant over the sky cell 1625 (otherwise, the sky cell can be broken into smaller sections). 1620 1626 1621 1627 The subtracted image is scoured for point sources above the noise … … 1639 1645 per day. 1640 1646 1641 The taskoutputs:1647 The module outputs: 1642 1648 \begin{enumerate} 1643 1649 \item Combined sky cell image, with all variable sources masked --- 1644 used for the next task;1650 used for the next module; 1645 1651 \item Subtracted image, with long trails masked --- sent to the IPP 1646 1652 Pixel Server; and 1647 \item Catalog ueof variable sources --- sent to the IPP Object1653 \item Catalog of variable sources --- sent to the IPP Object 1648 1654 Database. 1649 1655 \end{enumerate} … … 1652 1658 \paragraph{Add to Static Sky} 1653 1659 1654 This task adds the combined sky cell image into the static sky, so 1655 that a deep image of the sky may be formed. The inputs are: 1660 This module adds the combined sky cell image into the static sky, so 1661 that a deep image of the sky may be formed. This step should only be 1662 performed if the new data is of sufficient quality that it will not 1663 degrade the static sky image. The inputs are: 1656 1664 \begin{enumerate} 1657 1665 \item The combined sky cell image with variable sources masked --- 1658 from a previous task;1659 \item The current version of the static sky --- from a previous task,1666 from a previous module; 1667 \item The current version of the static sky --- from a previous module, 1660 1668 or the IPP Pixel Server; and 1661 1669 \item Relative weightings, based on the relative signal-to-noise in … … 1674 1682 \begin{enumerate} 1675 1683 \item The new static sky image --- sent to the Sky Image Server; 1676 \item The Catalog ueof sources on the new static sky image --- sent to the IPP Object Database; and1684 \item The Catalog of sources on the new static sky image --- sent to the IPP Object Database; and 1677 1685 \item The estimated limiting magnitude for the new static sky --- 1678 1686 metadata associated with the the new static sky image. … … 1682 1690 1683 1691 \begin{itemize} 1684 \item Catalog ues should include positional information ($x,y$, with1692 \item Catalogs should include positional information ($x,y$, with 1685 1693 associated errors), photometry (with associated error), and shape 1686 1694 parameters (FWHM, major and minor axes, position angle). … … 1693 1701 1694 1702 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1695 \subsubsection{ basic detrend image creation}1703 \subsubsection{Basic detrend image creation} 1696 1704 1697 1705 The basic detrend image creation pipeline collects the appropriate 1698 1706 input detrend images (bias, dark, flat, etc?) and generates a master 1699 image by combining the input images in some optimal way (median /1700 sigma-clipping / etc). The master image is used to determine input 1701 image residuals so that poor input images can be iteratively 1702 rejected. 1707 image by combining the input images in some optimal way 1708 \tbd{median/sigma-clipping/etc}. The master image is used to 1709 determine input image residuals so that poor input images can be 1710 iteratively rejected. 1703 1711 1704 1712 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1705 \subsubsection{ fringe pattern and sky foreground model creation}1713 \subsubsection{Fringe pattern and sky foreground model creation} 1706 1714 1707 1715 The fringe model creation and sky foreground model creation pipelines … … 1715 1723 1716 1724 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1717 \subsubsection{ photometric flat correction image creation}1725 \subsubsection{Photometric flat correction image creation} 1718 1726 1719 1727 The photometric flat-field correction uses images which have been … … 1727 1735 \subsubsection{Astrometric Reference Catalog} 1728 1736 1737 For PS1, this shall be UCAC. 1738 1739 For PS4, this shall be the PS1 catalog. 1740 1729 1741 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1730 1742 \subsubsection{Photometric Reference Catalog} 1731 1743 1744 For PS1, absolute photometry will not be available until the master 1745 fit which will be performed when all data is taken. For purposes of 1746 relative photometric extinction, the guide star brightnesses should be 1747 sufficient. 1748 1749 For PS4, the PS1 catalogue shall be used. 1750 1751 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1732 1752 \subsection{Modules} 1733 1753 1734 \subsection{ PanSTARRSLibrary}1754 \subsection{\PS{} Library} 1735 1755 1736 1756 \subsection{Internal Interfaces} 1737 1757 1738 Internal interfaces consist of queries to the I ID or IPS, insertion of1739 data in the I ID, IPS, or IOD, or processing configuration files. The1758 Internal interfaces consist of queries to the IMD or IPS, insertion of 1759 data in the IMD, IPS, or IOD, or processing configuration files. The 1740 1760 science and calibration image processing pipelines make requests for 1741 images from the IPS, meta -data from the IID, and push their results1742 back onto the IPS and I ID. The reference catalog pipelines make1743 requests on the I ID and the IOD and push their results back to the1761 images from the IPS, metadata from the IMD, and push their results 1762 back onto the IPS and IMD. The reference catalog pipelines make 1763 requests on the IMD and the IOD and push their results back to the 1744 1764 IOD. The scheduler creates input processing configuration files for 1745 the processing pipelines and queries the I ID and IPS and pushes1765 the processing pipelines and queries the IMD and IPS and pushes 1746 1766 results back to the IIS. 1747 1767 … … 1761 1781 1762 1782 This subsection describes the interfaces between the IPP and other 1763 Pan-STARRSsystems and the external clients. The interfaces are1764 illustrated in Figure NN. Incoming data is received by either the IPS1765 (pixels), the IID (meta-data), or the IOD (objects). Requests for 1766 data by external clients are also made to these three databases.1767 Requests for data made by the IPP are generated by the IPP Sch deduler1768 or the science processing pipelines. 1783 \PS{} systems and the external clients. The interfaces are 1784 illustrated in Figure \tbd{NN}. Incoming data is received by either 1785 the IPS (pixels), the IMD (metadata), or the IOD (objects). Requests 1786 for data by external clients are also made to these three databases. 1787 Requests for data made by the IPP are generated by the IPP Scheduler 1788 or the science processing pipelines. 1769 1789 1770 1790 \subsubsection{OATS} 1771 1791 1772 The Summit Pixel Server (SPS) sends raw image data, image meta -data,1773 and enviromental meta -data to the IPP. The IPP provides an interface1792 The Summit Pixel Server (SPS) sends raw image data, image metadata, 1793 and enviromental metadata to the IPP. The IPP provides an interface 1774 1794 mechanism by which the SPS can register new images with the IPP, which 1775 1795 sends them to the appropiate subsystem: The image pixel data is sent 1776 to the IPS while the metadata is sent to the I ID.1777 1778 The Pan-STARRSTelescope Scheduler (PTS) sends information about the1779 telescope sche lude to the IPP: observing plan for the night, or longer1796 to the IPS while the metadata is sent to the IMD. 1797 1798 The \PS{} Telescope Scheduler (PTS) sends information about the 1799 telescope schedule to the IPP: observing plan for the night, or longer 1780 1800 time scales. The IPP scheduler sends telescope schedule requests to 1781 the PTS .1801 the PTS (i.e.\ calibration needs). 1782 1802 1783 1803 \subsubsection{Published Static Sky Server} … … 1788 1808 provides updated static sky images to the SIS when available. 1789 1809 1790 \subsubsection{ PublishedObject Database}1810 \subsubsection{Object Database} 1791 1811 1792 1812 The Master Science Object Database receives new object photometry from … … 1795 1815 timescale. Is this a function of the IOD?} 1796 1816 1797 \subsubsection{Moving Object Pipeline} 1798 1799 The Moving Object Pipeline interfaces with the IPP to receive the 1800 objects detected in the difference images. \tbd{Does the IPP IOD push 1801 the objects out or respond to requests for new objects?} The MOP 1802 sends the IPP the current set of known ephemerids for objects as 1803 requested. The MOP may interface with the IID as needed. 1817 \subsubsection{Moving Object Processing System} 1818 1819 The Moving Object Processing System interfaces with the IPP to receive 1820 the objects detected in the difference images via queries to the IOD. 1821 The MOPS may interface with the IMD as needed. 1804 1822 1805 1823 \subsubsection{Other Client Science Pipelines} 1806 1824 1807 1825 The client science pipelines may interface with the IPP via requests 1808 for data from the I ID, IOD, or IPS. \tbd{how many clients max? / how1826 for data from the IMD, IOD, or IPS. \tbd{how many clients max? / how 1809 1827 much data?} 1810 1828 … … 1813 1831 \subsubsection{Overview} 1814 1832 1815 This document discusses the likely range of the Pan-STARRSImage1833 This document discusses the likely range of the \PS{} Image 1816 1834 Processing Pipeline (IPP) hardware requirements. The hardware 1817 1835 requirements addressed in this document consist of: … … 1866 1884 organization scenario, which will require the software to track the 1867 1885 location of data products more carefully. In addition, this document 1868 will address the data requirements of the complete Pan-STARRSpipeline1869 with 4 telescopes as well as the single-telescope Pan-STARRS-1 scenario1886 will address the data requirements of the complete \PS{} pipeline 1887 with 4 telescopes as well as the single-telescope \PS{}-1 scenario 1870 1888 based on the Design Reference Mission [REF]. 1871 1889 … … 1893 1911 currently possible to buy a single switch which would have a 1894 1912 sufficient number of GigE ports for both sections of the PS-1 system, 1895 such a two-switch organization may be needed for the full Pan-STARRS1913 such a two-switch organization may be needed for the full \PS{} 1896 1914 system. In such a case, the interswitch communication must also meet 1897 1915 the required throughput needs. We discuss the hardware requirements … … 1989 2007 \subsubsection{Data Storage Requirements} 1990 2008 1991 The Pan-STARRSIPP data storage requirements may be divided into five2009 The \PS{} IPP data storage requirements may be divided into five 1992 2010 principal areas: raw image data, static sky image data, master 1993 2011 calibration images, the metadata database, and the object database. … … 2367 2385 roughly 60-70 Sky-cells per exposure set. Thus the Phase 4 processing 2368 2386 adds an additional 750 MB/sec network bandwidth. In the architecture 2369 defined in Figure NN, the Sky nodes and the OTA nodes are each2387 defined in Figure \tbd{NN}, the Sky nodes and the OTA nodes are each 2370 2388 attached to separate switches. An additional bandwidth requirement is 2371 2389 derived by the need to exchange data between these switches in for … … 2501 2519 reliable and robust to missing elements. If a specific cell is 2502 2520 missing from an OTA, that information is known by the controller an 2503 needs to be represented in the meta -data. Similarly if an OTA is2521 needs to be represented in the metadata. Similarly if an OTA is 2504 2522 missing from a mosaic camera, that information is also known and must 2505 be carried though the meta -data. A more difficult association is that2523 be carried though the metadata. A more difficult association is that 2506 2524 between the telescopes to define the major frame. Some possibilities: 2507 2525 … … 2517 2535 appropriate, some varient is required). 2518 2536 \item exposure links are defined more generally on the basis of the 2519 resulting image meta -data. The telescopes may have images requested2537 resulting image metadata. The telescopes may have images requested 2520 2538 at the same coordinates and time, and are defined as a major frame on 2521 2539 the basis of the observed time and coordinates. The TCS or PTS might … … 2612 2630 and their orbital elements, and the time range for the calculation. 2613 2631 If the calculation is slow, Phase 0 could be paralellized by object. 2614 If Phase 0 is fast enough ( {\bfminutes?}), the process need not be2632 If Phase 0 is fast enough (\tbd{minutes?}), the process need not be 2615 2633 parallel. The {\tt lifetime} and {\tt date of calculation} allow old 2616 Phase 0 entries to be removed when they are not needed. {\bf [This2617 cleaning phase could be a function of Phase 0. ]}.Phase 0 need not be2634 Phase 0 entries to be removed when they are not needed. \tbd{This 2635 cleaning phase could be a function of Phase 0.} Phase 0 need not be 2618 2636 run only for the current night. Any time a specific set of data is to 2619 2637 be analysed by the later stages, phase 0 should be run for the 2620 appropriate time period. {\bf [Does there need to be a database table2638 appropriate time period. \tbd{Does there need to be a database table 2621 2639 with phase 0 runs and time periods defined? this could be the 2622 2640 reference used by later phases to decide if phase 0 has been run. they 2623 2641 could also trigger the phase 0 run if they notice it has not been run 2624 (a job of the scheduler). ]}2625 2626 {\bf TBD: what is the orbit calculation speed? does it scale with 2627 Npts?what is the number of known objects now? in 5 years?}2642 (a job of the scheduler).} 2643 2644 \tbd{what is the orbit calculation speed? does it scale with Npts? 2645 what is the number of known objects now? in 5 years?} 2628 2646 2629 2647 … … 2670 2688 affect the flux in neighbouring pixels 2671 2689 2672 \milsection{Object Catalog ues}2673 \label{ap:catalog ues}2674 2675 Object catalog ues from Phase 2 shall consist of at least the2690 \milsection{Object Catalogs} 2691 \label{ap:catalogs} 2692 2693 Object catalogs from Phase 2 shall consist of at least the 2676 2694 following elements for each object: 2677 2695 \begin{enumerate} … … 2684 2702 \end{enumerate} 2685 2703 2686 Though further details may be required for catalog ues in Phase~4,2687 the above details are minimum requirements for Phase~2 catalog ues.2688 2704 Though further details may be required for catalogs in Phase~4, 2705 the above details are minimum requirements for Phase~2 catalogs. 2706
Note:
See TracChangeset
for help on using the changeset viewer.
