Changeset 1084
- Timestamp:
- Jun 24, 2004, 10:24:27 AM (22 years ago)
- File:
-
- 1 edited
-
trunk/doc/design/ippSRS.tex (modified) (45 diffs)
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/design/ippSRS.tex
r1067 r1084 1 %%% $Id: ippSRS.tex,v 1. 5 2004-06-22 08:35:17 eugene Exp $1 %%% $Id: ippSRS.tex,v 1.6 2004-06-24 20:24:27 eugene Exp $ 2 2 \documentclass[panstarrs]{panstarrs} 3 3 … … 14 14 15 15 % allow paragraphs to be listed in TOC for now 16 \setcounter{tocdepth}{ 4}16 \setcounter{tocdepth}{3} 17 17 18 18 \begin{document} … … 28 28 DR.05 & 2003.05.24 & Incorporating comments from internal review \\ \hline 29 29 \RevisionsEnd 30 31 \TBDsStart 32 % section page TBR number Description 33 section & page & TBR & description \\ \hline 34 \TBDsEnd 30 35 31 36 \tableofcontents … … 99 104 \subsection{Science Requirements} 100 105 \label{req:system-capabilities} 101 102 \tbd{distinguish data products in commissioning, during AP survey,103 after AP survey}104 105 \tbd{define astronomy concepts: cleaned, pre-processed, significant,106 outlier}107 106 108 107 The IPP must perform the following tasks: … … 171 170 variations in the atmosphere across the field, and the reference 172 171 catalogs. The photometric scatter in photometric weather must be 173 better than \tb d{5 millimag} for relative photometry (relative to the174 internal filter system) and \tb d{10 millimag} for absolute photometry172 better than \tbr{5 millimag} for relative photometry (relative to the 173 internal filter system) and \tbr{10 millimag} for absolute photometry 175 174 (relative to other filter systems such as the SDSS filters). 176 175 … … 218 217 \label{req:active-state} 219 218 220 In active state, the IPP must accept images and metadata from the 221 external sources (i.e., the summit) and automatically perform the 222 complete set of image processing tasks, including both calibration and 223 science image processing. The IPP must respond to requests for data 224 from client science pipelines. In the active state, the IPP must 225 respond to analysis priority requests issued by the IPP users. 219 In active state, the IPP must: 220 221 \begin{enumerate} 222 \item Accept images and metadata from the external sources (i.e., the 223 summit) 224 225 \item Automatically perform the complete set of image processing 226 tasks, including both calibration and science image processing. 227 228 \item Respond to requests for data from client science pipelines. 229 230 \item Respond to analysis priority requests issued by the IPP users. 231 \end{enumerate} 226 232 227 233 \subsubsection{Paused State} … … 234 240 \label{req:interactive-state} 235 241 236 In interactive state, the IPP must accept incoming data and metadata, 237 but must not automatically process the data. The IPP must respond to 238 user commands to initiate portions of the data analysis. 242 In interactive state, the IPP must: 243 244 \begin{enumerate} 245 \item Accept incoming data and metadata from the external sources. 246 \item Not automatically process the data 247 \item Respond to user commands to initiate portions of the data 248 analysis. 249 \end{enumerate} 239 250 240 251 \subsection{Software Coding Requirements} … … 243 254 \label{req:languages} 244 255 245 Source code must be in C. All source code must be compiled with `gcc' 246 version v2.95 or higher, and the tested compiler version must be 247 defined for the delivered software product. 248 249 Scripting language must be \tbd{Python, version TBD}. 256 \begin{enumerate} 257 \item Source code must be in C. 258 \item All source code must be compiled with `gcc' version v2.95 or higher. 259 \item The tested compiler version must be defined for the delivered software product. 260 \item Scripting language must be \tbd{Python}, version X.X. 261 \end{enumerate} 250 262 251 263 \subsubsection{Interfaces} 252 We require the following types of interfaces:253 264 \begin{enumerate} 254 265 \item Access to low-level Library functions must be provided via C … … 278 289 \subsubsection{Naming Conventions} 279 290 280 Header files must have names starting \code{ps} or \code{p_ps} for 281 private interface definitions. The latter must appear in a 291 \begin{enumerate} 292 \item Header files must have names starting \code{ps} or \code{p_ps} 293 for private interface definitions. The latter must appear in a 282 294 subdirectory \code{private} of whichever directory is being searched 283 295 for the public header files. 284 296 285 Functions visible at global scope that are part of the public API must 286 have names beginning with \code{ps} and follow the naming conventions 287 in the coding standard. Functions visible at global scope but which 288 are not part of the public interface must have names beginning with 289 \ code{p_ps}. Functions that are local to a file must \textit{not}290 start with \code{ps} or\code{p_ps}.297 \item Functions visible at global scope that are part of the public 298 API must have names beginning with \code{ps} and follow the naming 299 conventions in the coding standard. 300 301 \item Functions visible at global scope but which are not part of the 302 public interface must have names beginning with \code{p_ps}. 291 303 292 Variables visible at global scope which are part of the public API 293 must have names beginning with \code{ps}, and follow the naming 294 conventions in the coding standard. Variables that are visible at 295 global scope but which are not part of the public interface must have 296 names beginning with \code{p_ps}. Variables that are local to a file 297 must \textit{not} start with \code{ps} (or \code{p_ps}). 298 299 The names of all enumerated types and C-preprocessor symbols (but not 300 variables declared \code{const}) must start with \code{PS_}, in the 301 case of public symbols, or \code{P_PS_}, for private symbols. The 302 rest of the name must be uppercase with words separated by underscores 303 (\code{_}). An exception is the case of system utilities implemented 304 as macros, in which case the names must conform to the convention for 305 function names. 306 307 When defining a function to convert from one type to another, the name 308 must be of the form \code{psOldToNew}, e.g.\hfil\break 309 \code{psEquatorialToEcliptic} (\emph{not} 304 \item Functions that are local to a file must \textit{not} start with 305 \code{ps} or \code{p_ps}. 306 307 \item Variables visible at global scope which are part of the public 308 API must have names beginning with \code{ps}, and follow the naming 309 conventions in the coding standard. 310 311 \item Variables that are visible at global scope but which are not 312 part of the public interface must have names beginning with 313 \code{p_ps}. 314 315 \item Variables that are local to a file must \textit{not} start with 316 \code{ps} (or \code{p_ps}). 317 318 \item The names of all enumerated types and C-preprocessor symbols 319 (but not variables declared \code{const}) must start with \code{PS_}, 320 in the case of public symbols, or \code{P_PS_}, for private symbols. 321 The rest of the name must be uppercase with words separated by 322 underscores (\code{_}). An exception is the case of system utilities 323 implemented as macros, in which case the names must conform to the 324 convention for function names. 325 326 \item When defining a function to convert from one type to another, 327 the name must be of the form \code{psOldToNew}, 328 e.g.\code{psEquatorialToEcliptic} (\emph{not} 310 329 \code{psEquatorial2Ecliptic}). 330 \end{enumerate} 311 331 312 332 \subsubsection{C Programming Guidelines} 313 333 314 Functions that assign to a variable must list that argument 315 \textit{first}, following the pattern of \code{strcpy}; e.g. 316 \begin{verbatim} 317 void psVectorCopy(restrict psVector *out, const restrict psVector *in); 318 \end{verbatim} 319 320 Type definitions should always be accompanied by prototypes for their 321 constructors and destructors, following these guidelines: 322 323 \begin{itemize} 324 \item The constructor name should consist of the type name followed by 325 \code{Alloc}; e.g. a type \code{psImage} would be created by a 326 function \code{psImage *psImageAlloc();}. 327 328 \item The type should be freed with a destructor named 329 \code{typeFree}, e.g. \code{void psImageFree(psImage *image);}. 330 331 \item The constructor must never return \code{NULL}, and no code calling the 332 constructor should ever check the return value. 333 334 \item The destructor must not return a value. 335 336 \item The destructor must handle being passed \code{NULL} by simply 337 returning immediately. This must not be treated as an error 338 condition. 339 340 \item Constructors and Destructors should use the memory reference 334 \begin{enumerate} 335 \item Functions that assign to a variable must list that argument 336 \textit{first}, following the pattern of \code{strcpy}. For 337 example: 338 \begin{verbatim} 339 void psVectorCopy(restrict psVector *out, const restrict psVector *in); 340 \end{verbatim} 341 342 \item Type definitions should always be accompanied by prototypes for 343 their constructors and destructors, following these guidelines: 344 345 \begin{enumerate} 346 \item The constructor name should consist of the type name followed 347 by \code{Alloc}; e.g. a type \code{psImage} would be created by a 348 function \code{psImage *psImageAlloc();}. 349 350 \item The type should be freed with a destructor named 351 \code{typeFree}, e.g. \code{void psImageFree(psImage *image);}. 352 353 \item The constructor must never return \code{NULL}, and no code 354 calling the constructor should ever check the return value. 355 356 \item The destructor must not return a value. 357 358 \item The destructor must handle being passed \code{NULL} by simply 359 returning immediately. This must not be treated as an error 360 condition. 361 362 \item Constructors and Destructors should use the memory reference 341 363 counter facilities of the PSLib memory management system. 342 343 \end{ itemize}364 \end{enumerate} 365 \end{enumerate} 344 366 345 367 \subsubsection{Commenting and Documentation} 346 368 347 Commenting of delivered C and Python code must follow the C and 348 Python coding standards and must provide tags for Doxygen 349 interpretation of the comments and program structures. 350 351 Documentation for the IPP consists of source code documentation and 352 user documentation. Source code documentation must be generated with 353 Doxygen from the in-line comments and must be provided as HTML, 354 Latex, and man pages. User documentation includes the API usage for 355 the modules and library functions as well as user interface 356 description for the higher-level architectural systems. User 357 documentation must be delivered as PDF documents. 369 \begin{enumerate} 370 \item Commenting of delivered C code must follow the C coding 371 standards and provide tags for Doxygen interpretation of the 372 comments and program structures. 373 374 \item Commenting of delivered Python code must follow the Python 375 coding standards. 376 377 \item Source code documentation must be generated with Doxygen from 378 the in-line comments and must be provided as HTML, Latex, and man 379 pages. 380 381 \item User documentation includes the API usage for the modules and 382 library functions as well as user interface description for the 383 higher-level architectural systems. User documentation must be 384 delivered as PDF documents. 385 \end{enumerate} 358 386 359 387 \subsubsection{Version Control} … … 409 437 The architectural components consist of: 410 438 411 \begin{ enumerate}439 \begin{itemize} 412 440 413 441 \item {\bf Image Server:} This component is a large data store for all … … 444 472 to assign the analysis tasks to the Controller. 445 473 446 \end{ enumerate}474 \end{itemize} 447 475 448 476 The relationship between these software elements is shown in … … 461 489 \subsubsection{Image Server} 462 490 463 The IPP Image Server must store images on a distributed collection of 464 computer disks. Individual instances of a file are only required to 465 be stored on a single machine (striping across computers is not a 466 requirement). 467 468 The IPP Image Server must be capable of honoring requests to store an 469 image on a specific machine. If such a request cannot be honored (ie, 470 the machine is down), the IPP Image Server must select an appropriate 471 machine and notify the requesting agent of the new location. 472 473 The IPP Image Server must store multiple copies of each image upon 474 request, the number of copies specified independently for each file by 475 the user. 476 477 The IPP Image Server must maintain a record of all image copies 478 currently available in the repository. This record must include the 479 image name, location (which machine), the image size, and the state of 480 the image (available, locked, deleted). 481 482 The IPP Image Server must lock images in the repository on request. 483 Both read (shared) and write (exclusive) locks must be provided. A 484 read lock must prevent write access to the file; a write lock must 485 prevent both read and write access. 486 487 The IPP Image Server must return the image location (the computer on 488 which it resides) upon request. 489 490 The IPP Image Server must provide a specified image upon request. 491 492 The IPP Image Server must delete images in the repository on request. 493 494 The IPP Image Server must accept images from the summit at the maximum 495 rate of 1 full-camera image every 30 seconds. The IPP Image Server 496 must therefore accept new images into the repository at a rate of 64 497 raw OTAs in 30 seconds and a total input data volume rate of 75 498 MB/sec. 491 \begin{enumerate} 492 \item The IPP Image Server must store images on a distributed 493 collection of computer disks. Individual instances of a file are 494 only required to be stored on a single machine (striping across 495 computers is not a requirement). 496 497 \item The IPP Image Server must be capable of honoring requests to 498 store an image on a specific machine. 499 500 \item If such a request cannot be honored (ie, the machine is down), 501 the IPP Image Server must select an appropriate machine and notify 502 the requesting agent of the new location. 503 504 \item The IPP Image Server must store multiple copies of each image 505 upon request, the number of copies specified independently for each 506 file by the user. 507 508 \item The IPP Image Server must maintain a record of all image copies 509 currently available in the repository. This record must include the 510 image name, location (which machine), the image size, and the state 511 of the image (available, locked, deleted). 512 513 \item The IPP Image Server must lock images in the repository on 514 request. Both read (shared) and write (exclusive) locks must be 515 provided. A read lock must prevent write access to the file; a 516 write lock must prevent both read and write access. 517 518 \item The IPP Image Server must return the image location (the 519 computer on which it resides) upon request. 520 521 \item The IPP Image Server must provide a specified image upon request. 522 523 \item The IPP Image Server must delete images in the repository on request. 524 525 \item The IPP Image Server must accept images from the summit at the 526 maximum rate of 1 full-camera image every 30 seconds. The IPP Image 527 Server must therefore accept new images into the repository at a 528 rate of 64 raw OTAs in 30 seconds and a total input data volume rate 529 of 75 MB/sec. 530 \end{enumerate} 499 531 500 532 \tbd{archive lifetime} … … 532 564 \end{table} 533 565 534 The AP Database must accept and store individual detections and 535 collections of detections along with information about the image which 536 provided the detections. 537 538 Detections must be saved as one of several detection classes (P2, P4S, 539 P4D, SS) and the AP Database must store the appropriate parameters, 540 listed in Table~\ref{APdetections}, for each class. 541 542 The AP Database must identify the image which provided the detection, 543 or in the case of external references, an identifier specific to the 544 reference source. 545 546 The AP Database must group detections into objects and measure average 547 parameters of those objects. 548 549 The AP Database must store parallax and proper motion parameters for a 550 subset of the average objects. 551 552 The AP Database must store image and filter calibration information 553 necessary to convert between instrumental magnitudes and calibrated 554 magnitudes in standard systems. 555 556 The AP Database must perform at least the follow queries, with 557 constraints on the output based on at least time ranges, magnitude 558 limits, error limits: 559 \begin{enumerate} 560 \item given $(RA,DEC)$ and a Radius, return all objects and/or 561 detections in the region. 562 563 \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all objects and/or 564 detections in the region. 565 566 \item given $(RA,DEC)$, return closest object. 567 568 \item given object ID, return all detections 569 570 \item given detection, return source image data. 571 572 \item given detection, return object. 573 574 \item given $(RA,DEC)$, return all images overlapping coordinate. 575 576 \item given $(RA,DEC)$ and a Radius, return all images overlapping region. 577 578 \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping 579 region. 580 581 \item given detection instrumental magnitude, return derived 582 magnitudes based on calibration information. 583 584 \item given a collection of detections in a filter, determine the 585 object average magnitude in that filter. 586 587 \item given a collection of objects and detections, determine the 588 individual image zero-points. 589 590 \item given a region, return all possible combinations of the object 591 or detection magnitudes $(M_1 - M_2)$. 592 593 \item given a list of $(RA,DEC)$ entries, return all nearest objects. 594 595 \item given a filter, telescope, or detector, return all calibration 596 terms and history. 597 598 \item given a detection, return all non-detections from images which 599 overlapped the detection coordinates. 600 601 \end{enumerate} 602 603 The AP Database must accept detection IDs of moving objects and label 604 the detections with the identified object. 566 \begin{enumerate} 567 \item The AP Database must accept and store individual detections and 568 collections of detections along with information about the image 569 which provided the detections. 570 571 \item Detections must be saved as one of several detection classes 572 (P2, P4S, P4D, SS) and the AP Database must store the appropriate 573 parameters, listed in Table~\ref{APdetections}, for each class. 574 575 \item The AP Database must identify the image which provided the 576 detection, or in the case of external references, an identifier 577 specific to the reference source. 578 579 \item The AP Database must group detections into objects and measure 580 average parameters of those objects. 581 582 \item The AP Database must store parallax and proper motion parameters 583 for a subset of the average objects. 584 585 \item The AP Database must store image and filter calibration 586 information necessary to convert between instrumental magnitudes and 587 calibrated magnitudes in standard systems. 588 589 \item The AP Database must perform at least the follow queries, with 590 constraints on the output based on at least time ranges, magnitude 591 limits, error limits: 592 593 \begin{enumerate} 594 \item given $(RA,DEC)$ and a Radius, return all objects and/or 595 detections in the region. 596 597 \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all objects and/or 598 detections in the region. 599 600 \item given $(RA,DEC)$, return closest object. 601 602 \item given object ID, return all detections 603 604 \item given detection, return source image data. 605 606 \item given detection, return object. 607 608 \item given $(RA,DEC)$, return all images overlapping coordinate. 609 610 \item given $(RA,DEC)$ and a Radius, return all images overlapping region. 611 612 \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping 613 region. 614 615 \item given detection instrumental magnitude, return derived 616 magnitudes based on calibration information. 617 618 \item given a collection of detections in a filter, determine the 619 object average magnitude in that filter. 620 621 \item given a collection of objects and detections, determine the 622 individual image zero-points. 623 624 \item given a region, return all possible combinations of the object 625 or detection magnitudes $(M_1 - M_2)$. 626 627 \item given a list of $(RA,DEC)$ entries, return all nearest objects. 628 629 \item given a filter, telescope, or detector, return all calibration 630 terms and history. 631 632 \item given a detection, return all non-detections from images which 633 overlapped the detection coordinates. 634 \end{enumerate} 635 636 \item The AP Database must accept detection IDs of moving objects and 637 label the detections with the identified object. 638 639 \item The AP Database must accept new detections at the rate generated 640 by the telescope from the Phase 2 and Phase 4 analysis. Except 641 within 10 degrees of the galactic plane, the AP Database must keep 642 up with the incoming rates. The expected rates are listed in 643 Table~\ref{APrates}, along with the total data volume required for 644 storage space over the PS-1 lifetime. 645 646 \end{enumerate} 647 648 \tbd{archive lifetime} 649 650 \tbd{reliability} 651 652 \tbd{backups} 605 653 606 654 \begin{table} … … 625 673 \end{table} 626 674 627 The AP Database must accept new detections at the rate generated by628 the telescope from the Phase 2 and Phase 4 analysis. Except within 10629 degrees of the galactic plane, the AP Database must keep up with the630 incoming rates. The expected rates are listed in Table~\ref{APrates},631 along with the total data volume required for storage space over the632 PS-1 lifetime.633 634 \tbd{archive lifetime}635 636 \tbd{reliability}637 638 \tbd{backups}639 640 675 \subsubsection{Metadata Database} 641 676 642 677 \begin{table} 643 678 \begin{center} 644 \caption{Metadata Classes\label{, and the while 645 the metadata}} 679 \caption{Metadata Classes\label{metadata}} 646 680 \begin{tabular}{l} 647 681 \hline … … 673 707 stages 674 708 675 The Metadata Database must store and provide metadata for all raw 676 images, for processed images, for the calibration images (both raw and 677 master), for the extracted object lists. Metadata describing the 678 environmental conditions at the telescope must also be stored and 679 provided as needed. Table~\ref{metadata} lists the classes of 680 metadata which must be stored by the Metadata Database. 681 682 If analysis results are exchanged between analysis stages via the 683 Metadata Database, it must provide access to the queried data on 684 timescales of $<2$ seconds to avoid slowing down the analysis systems. 685 686 The Metadata Database must store the metadata for the lifetime of the 687 project. The Metadata Database must be capable of accepting a total 688 data volume after 2 years of operation of 128 GB. 689 690 The Metadata Database must respond to simple queries which return the 691 data in the categories listed in Table~\ref{metadata} based on the 692 primary data key and with basic constraints of time ranges and other 693 simple conditional constraints. 694 695 The Metadata must store descriptive information about the raw images 696 received from the summit and the current state of the data processing. 697 The Metadata must also store descriptive information for each of the 698 static sky images currently available. 699 700 The IPP requires configuration information defining the organization 701 and configuration of the IPP itself. The Metadata database must store 702 the configuration information with restricted access so that only 703 specific people may change the information. Examples of configuration 704 data include the default parameters for the various analysis programs, 705 the description of the computing environment, and the process status 706 information, etc. The Metadata Database must restrict access to the 707 scientific parameters to a different group from the software and 708 hardware configuration parameters. 709 \begin{enumerate} 710 \item The Metadata Database must store and provide metadata for all 711 raw images, for processed images, for the calibration images (both 712 raw and master), for the extracted object lists. Metadata 713 describing the environmental conditions at the telescope must also 714 be stored and provided as needed. Table~\ref{metadata} lists the 715 classes of metadata which must be stored by the Metadata Database. 716 717 \item If analysis results are exchanged between analysis stages via 718 the Metadata Database, it must provide access to the queried data on 719 timescales of $<2$ seconds to avoid slowing down the analysis 720 systems. 721 722 \item The Metadata Database must store the metadata for the lifetime 723 of the project. 724 725 \item The Metadata Database must be capable of accepting a total data 726 volume after 2 years of operation of 128 GB. 727 728 \item The Metadata Database must respond to simple queries which 729 return the data in the categories listed in Table~\ref{metadata} 730 based on the primary data key and with basic constraints of time 731 ranges and other simple conditional constraints. 732 733 \item The Metadata must store descriptive information about the raw 734 images received from the summit and the current state of the data 735 processing. 736 737 \item The Metadata must also store descriptive information for each of 738 the static sky images currently available. 739 740 \item The IPP requires configuration information defining the 741 organization and configuration of the IPP itself. The Metadata 742 database must store the configuration information with restricted 743 access so that only specific people may change the information. 744 Examples of configuration data include the default parameters for 745 the various analysis programs, the description of the computing 746 environment, and the process status information, etc. 747 748 \item The Metadata Database must restrict access to the scientific 749 parameters to a different group from the software and hardware 750 configuration parameters. 751 752 \item In the discussion of the Analysis Stages below, various steps 753 specify that the values are user-configurable parameters. These 754 parameters must be stored in and extracted from the Metadata 755 Database. 756 \end{enumerate} 709 757 710 758 \subsubsection{Controller} 711 712 The IPP Controller must manage tasks on a cluster of up to 128 713 computers. 714 715 On startup, the IPP Controller must attempt to establish communication 716 with all of its computers and set their state to be {\tt alive} or 717 {\tt dead} based on the success of the connection. 718 719 The IPP Controller must detect computers which crash or stop 720 responding and set their state to {\tt dead}. 721 722 The IPP Controller must attempt to re-establish communication with 723 {\tt dead} computers. 724 725 The IPP Controller must accept tasks from external users and systems, 726 which may specify a desired CPU (node) and priority in addition to the 727 task command. 728 729 The IPP Controller must attempt to run pending tasks on the desired 730 node, if available (not {\tt dead} or {\tt off}). If the node is 731 unavailable, the IPP Controller must attempt to run the task on 732 another node. If the node is available, the IPP Controller must 733 attempt to run a given task only if no higher-priority tasks are 734 available and no task is currently being executed. 735 736 The IPP Controller must monitor the output from the task and write it 737 to an associated log destination. 738 739 The IPP Controller must monitor the execution status of each task 740 currently executing on a node and perform the following actions: 741 \begin{enumerate} 742 \item identify the task as successful if it has a valid exit status. 743 \item identify the task as unsuccessful if it has an error exit 744 status. 745 \item identify the task as unattempted if the computer crashed. 746 \end{enumerate} 747 748 The IPP Controller must accept and perform the following external 749 commands: 750 \begin{enumerate} 751 \item add a task to the pending task list. 752 \item delete a specific task from the pending task list. 753 \item return the current status of a specific task. 754 \item return a list of all pending and non-pending tasks. 755 \item set a specified computer state to {\tt off} or {\tt dead}. 756 \item restrict a specified CPU to a class of tasks. 757 \item halt execution of a specified task. 758 \item set the IPP Controller state to {\tt finish}, {\tt abort}, or 759 {\tt stop}. 759 \begin{enumerate} 760 761 \item The IPP Controller must manage tasks on a cluster of up to 128 762 computers. 763 764 \item On startup, the IPP Controller must attempt to establish 765 communication with all of its computers and set their state to be 766 {\tt alive} or {\tt dead} based on the success of the connection. 767 768 \item The IPP Controller must detect computers which crash or stop 769 responding and set their state to {\tt dead}. 770 771 \item The IPP Controller must attempt to re-establish communication 772 with {\tt dead} computers. 773 774 \item The IPP Controller must accept tasks from external users and 775 systems, which may specify a desired CPU (node) and priority in 776 addition to the task command. 777 778 \item The IPP Controller must attempt to run pending tasks on the 779 desired node, if available (not {\tt dead} or {\tt off}). 780 781 \item If the node is unavailable, the IPP Controller must attempt to 782 run the task on another node. 783 784 \item If the node is available, the IPP Controller must attempt to run 785 a given task only if no higher-priority tasks are available and no 786 task is currently being executed. 787 788 \item The IPP Controller must monitor the output from the task and 789 write it to an associated log destination. 790 791 \item The IPP Controller must monitor the execution status of each 792 task currently executing on a node and perform the following 793 actions: 794 795 \begin{enumerate} 796 \item identify the task as successful if it has a valid exit status. 797 \item identify the task as unsuccessful if it has an error exit 798 status. 799 \item identify the task as unattempted if the computer crashed. 800 \end{enumerate} 801 802 \item The IPP Controller must accept and perform the following 803 external commands: 804 \begin{enumerate} 805 \item add a task to the pending task list. 806 \item delete a specific task from the pending task list. 807 \item return the current status of a specific task. 808 \item return a list of all pending and non-pending tasks. 809 \item set a specified computer state to {\tt off} or {\tt dead}. 810 \item restrict a specified CPU to a class of tasks. 811 \item halt execution of a specified task. 812 \item set the IPP Controller state to {\tt finish}, {\tt abort}, or 813 {\tt stop}. 814 \end{enumerate} 760 815 \end{enumerate} 761 816 762 817 \subsubsection{Scheduler} 763 764 The IPP Scheduler initiates analysis tasks which it must send to the 765 IPP Controller. 766 767 All analysis tasks sent by the IPP Scheduler must include a complete 768 UNIX command with necessary arguments, the priority of the task, and 769 optionally the desired processing node. 770 771 The IPP Scheduler must refer to several input data sources to decide 772 what tasks to initiate. These data sources include the IPP Metadata 773 Database, the Summit Metadata Database, and User requests. 774 775 The IPP Scheduler must query the Databases on a regular basis to check 776 for new input information. These queries must take place at least 777 once every \tbr{5 seconds}. 778 779 The IPP Scheduler must accept new User input in real-time (within 0.1 780 seconds of the request). 781 782 The IPP Scheduler must construct new tasks on the basis of the inputs 783 and a task dependency table. 784 785 When the IPP Scheduler is placed in the {\em paused state}, it must 786 only initiate User-requested tasks. 787 788 When the IPP Scheduler is placed in the {\em interactive state}, it 789 must initiate User-requested tasks as well as data transfer tasks. 790 791 When the IPP Scheduler is placed in the {\em automatic state}, it must 792 initiate the most appropriate task based on the inputs. 793 794 The IPP Scheduler must receive the exit status of tasks from the IPP 795 Controller. 796 797 The IPP Scheduler must send the exit status of the analysis tasks to 798 the appropriate destination as defined by the task dependency table. 818 \begin{enumerate} 819 \item The IPP Scheduler must send the analysis tasks which it 820 initiates to the IPP Controller. 821 822 \item All analysis tasks sent by the IPP Scheduler must include a 823 complete UNIX command with necessary arguments, the priority of the 824 task, and optionally the desired processing node. 825 826 \item The IPP Scheduler must refer to several input data sources to 827 decide what tasks to initiate. These data sources include the IPP 828 Metadata Database, the Summit Metadata Database, and User requests. 829 830 \item The IPP Scheduler must query the Databases on a regular basis to 831 check for new input information. These queries must take place at 832 least once every \tbr{5 seconds}. 833 834 \item The IPP Scheduler must accept new User input in real-time 835 (within 0.1 seconds of the request). 836 837 \item The IPP Scheduler must construct new tasks on the basis of the 838 inputs and a task dependency table. 839 840 \item When the IPP Scheduler is placed in the {\em paused state}, it 841 must only initiate User-requested tasks. 842 843 \item When the IPP Scheduler is placed in the {\em interactive state}, 844 it must initiate User-requested tasks as well as data transfer 845 tasks. 846 847 \item When the IPP Scheduler is placed in the {\em automatic state}, 848 it must initiate the most appropriate task based on the inputs. 849 850 \item The IPP Scheduler must receive the exit status of tasks from the 851 IPP Controller. 852 853 \item The IPP Scheduler must send the exit status of the analysis 854 tasks to the appropriate destination as defined by the task 855 dependency table. 856 \end{enumerate} 799 857 800 858 \subsection{Analysis Stages} … … 813 871 group. 814 872 815 The science image analysis stages must perform their analysis quickly 816 enough to keep up with the incoming data stream. The required 817 processing time is derived from the rate at which science images are 818 obtained by PS-1. At a minimum, the Science Image Analysis must keep 819 up with the average image rate over the course of 1 day. In order to 820 provide a sufficient buffer for variations in the processing speed, 821 the Science Image Analysis must be able to process all images from a 822 night within 12 hours. 823 824 The maximum latency between the acquisition of an image and the 825 completion of the science image analysis is set by the science 826 requirements of the fast transient recovery programs. The science 827 image analysis must process images from these observing programs 828 within \tbr{5 min} of their arrival time in the IPP Image Server. 829 830 The science image analysis stages must processes up to 1000 science 831 images per night. 873 \begin{enumerate} 874 \item The science image analysis stages must perform their analysis 875 quickly enough to keep up with the incoming data stream. The 876 required processing time is derived from the rate at which science 877 images are obtained by PS-1. 878 879 \item At a minimum, the Science Image Analysis must keep up with the 880 average image rate over the course of 1 day. 881 882 \item In order to provide a sufficient buffer for variations in the 883 processing speed, the Science Image Analysis must be able to process 884 all images from a night within 12 hours. 885 886 \item The maximum latency between the acquisition of an image and the 887 completion of the science image analysis is set by the science 888 requirements of the fast transient recovery programs. The science 889 image analysis must process images from these observing programs 890 within \tbr{5 min} of their arrival time in the IPP Image Server. 891 892 \item The science image analysis stages must processes up to 1000 893 science images per night. 894 895 \end{enumerate} 832 896 833 897 \subsubsection{Phase 1 : image processing preparation} 834 898 835 The Phase 1 analysis stage must determine the astrometric solution of 836 the complete camera (FPA image) with an accuracy of \tbr{1 arcsec} 837 peak-to-peak deviation. 838 839 The Phase 1 analysis stage must load the guide star pixel and 840 celestial coordinates from the \tbd{IPP Metadata Database}\comment{or 841 from the image header?}. 842 843 If guide stars are not available, the Phase 1 analysis stage must 844 extract bright stars from the image. This extraction must be done in 845 less than \tbr{1 second}. The total number of stars and size of the 846 bright-star acquisition box must be a user-configurable parameter. 847 848 In order for blind astrometry of an image to succeed, it is necessary 849 that approximate image coordinates be known. The Phase 1 analysis 850 must be able to succeed despite initial coordinate errors as large as 851 \tbr{20\arcsec}. 852 853 The Phase 1 analysis stage must construct a table of the overlaps 854 between the science image to be processed and the static sky images. 855 856 The overlaps must be overestimated by a small amount so that errors in 857 astrometry at Phase 1 will not cause any valid static sky / science 858 image pairs to be missed. The amount of overlap must be a 859 user-configurable parameter. 860 861 Sky cells which do not have sufficient science image overlap \tbd{$< 862 5\%$} must be excluded from the overlap table. 863 864 It is not unusual for an image to be obtained with invalid coordinates 865 or without any valid stars. For example, the telescope control system 866 may make an error and report the wrong time or coordinates. Or, the 867 image may be obtained in exceptionally poor conditions with no 868 detected stars. Phase 1 must return a descriptive error message in 869 these conditions. 899 \begin{enumerate} 900 \item The Phase 1 analysis stage must determine the astrometric 901 solution of the complete camera (FPA image) with an accuracy of 902 \tbr{1 arcsec} peak-to-peak deviation. 903 904 \item The Phase 1 analysis stage must load the guide star pixel and 905 celestial coordinates from the \tbd{IPP Metadata 906 Database}\comment{or from the image header?}. 907 908 \item If guide stars are not available, the Phase 1 analysis stage 909 must extract bright stars from the image. 910 911 \item This extraction must be done in less than \tbr{1 second}. 912 913 \item The total number of stars and size of the bright-star 914 acquisition box must be a user-configurable parameter. 915 916 \item In order for blind astrometry of an image to succeed, it is 917 necessary that approximate image coordinates be known. The Phase 1 918 analysis must be able to succeed despite initial coordinate errors 919 as large as \tbr{20\arcsec}. 920 921 \item The Phase 1 analysis stage must construct a table of the 922 overlaps between the science image to be processed and the static 923 sky images. 924 925 \item The overlaps must be overestimated by a small amount so that 926 errors in astrometry at Phase 1 will not cause any valid static sky 927 / science image pairs to be missed. 928 929 \item The amount of overlap must be a user-configurable parameter. 930 931 \item Sky cells which do not have sufficient science image overlap 932 \tbd{$< 5\%$} must be excluded from the overlap table. 933 934 \item It is not unusual for an image to be obtained with invalid 935 coordinates or without any valid stars. For example, the telescope 936 control system may make an error and report the wrong time or 937 coordinates. Or, the image may be obtained in exceptionally poor 938 conditions with no detected stars. Phase 1 must return a 939 descriptive error message in these conditions. 940 \end{enumerate} 870 941 871 942 \subsubsection{Phase 2 : image reduction} … … 874 945 the detector are processed to remove instrumental signatures. 875 946 876 The Phase 2 analysis stage must consult the processing recipe to 877 define the necessary analysis steps performed by the Phase 2 stage. 878 879 Phase 2 must perform the analysis steps only if required by the 880 processing recipe. The processing recipe must define the stages to be 881 executed with optional exposure time and background flux limits to 882 require or exclude select certain stages. 883 884 In the discussion below, various steps specify that the values are 885 user-configurable parameters. These parameters must be stored in and 886 extracted from the Metadata Database. 947 \paragraph{Processing Recipe} 948 \begin{enumerate} 949 \item The Phase 2 analysis stage must consult the processing recipe to 950 define the necessary analysis steps performed by the Phase 2 stage. 951 952 \item Phase 2 must perform the analysis steps only if required by the 953 processing recipe. 954 955 \item The processing recipe must define the stages to be executed with 956 optional exposure time and background flux limits to require or 957 exclude select certain stages. 958 \end{enumerate} 887 959 888 960 \paragraph{Detrend Image Convolutions} 889 890 The Phase 2 analysis stage must determine the OT kernel from the IPP 891 Metadata Database\comment{or image header}. 892 893 The Phase 2 analysis stage must convolve the flat-field and 894 high-spatial-frequency fringe images with the OT kernel. If no OT 895 kernel exists, this step must be silently skipped. 961 \begin{enumerate} 962 963 \item The Phase 2 analysis stage must determine the OT kernel from the 964 IPP Metadata Database\comment{or image header}. 965 966 \item The Phase 2 analysis stage must convolve the flat-field and 967 high-spatial-frequency fringe images with the OT kernel. 968 969 \item If no OT kernel exists, this step must be silently skipped. 970 \end{enumerate} 896 971 897 972 \paragraph{Flag bad and saturated pixels} 898 899 The Phase 2 analysis must load the basic bad pixel map appropriate to 900 the detector of interest. 901 902 The Phase 2 analysis must use the OT kernel to grow the traps in the 973 \begin{enumerate} 974 975 \item The Phase 2 analysis must load the basic bad pixel map appropriate to 976 the detector of interest. 977 978 \item The Phase 2 analysis must use the OT kernel to grow the traps in the 903 979 raw bad pixel map. 904 980 905 The Phase 2 analysis must mask saturated pixels and a user-specified981 \item The Phase 2 analysis must mask saturated pixels and a user-specified 906 982 number of surrounding pixels. 907 983 908 Different bits must be set to identify different reasons for masking984 \item Different bits must be set to identify different reasons for masking 909 985 the pixels. 986 \end{enumerate} 910 987 911 988 \paragraph{Bias correction via overscan subtraction} 912 913 Phase 2 must perform bias subtraction on the image. 914 915 Phase 2 must choose the bias subtraction method and analysis statistic 989 \begin{enumerate} 990 991 \item Phase 2 must perform bias subtraction on the image. 992 993 \item Phase 2 must choose the bias subtraction method and analysis statistic 916 994 based on the user-configured parameters. 917 995 918 The bias correction must be measured from the image overscan region.919 920 The overscan region must be determined from the image996 \item The bias correction must be measured from the image overscan region. 997 998 \item The overscan region must be determined from the image 921 999 header\comment{or Metadata DB}. 922 1000 923 The bias subtraction must apply one of the following bias corrections,1001 \item The bias subtraction must apply one of the following bias corrections, 924 1002 depending on the user parameters: 1003 925 1004 \begin{enumerate} 926 1005 \item subtract a single constant from the image. … … 934 1013 \end{enumerate} 935 1014 936 The statistic used to calculate the overscan constant or the inputs to 937 the spline and polynomial fits must be derived from groups of pixels 938 on the basis of one of several possible statistics, as specified by 939 the user parameters. The choice of statistics must include the sample 940 and robust mean, median, and modes. 941 942 In the case of a single constant, all of the overscan pixel values are 943 used in the calculation of this statistic. In the case of the 1D 944 functional representation, the input values to the fit must represent 945 the coordinate along the overscan, with the statistic derived from the 946 pixels in the perpendicular direction at each location. 947 948 If specified in the user parameters, sigma-clipping must be performed 949 on the input data values. 1015 \item The statistic used to calculate the overscan constant or the 1016 inputs to the spline and polynomial fits must be derived from groups 1017 of pixels on the basis of one of several possible statistics, as 1018 specified by the user parameters. 1019 1020 \item The choice of statistics must include the sample and robust 1021 mean, median, and modes. 1022 1023 \item In the case of a single constant, all of the overscan pixel 1024 values are used in the calculation of this statistic. 1025 1026 \item In the case of the 1D functional representation, the input 1027 values to the fit must represent the coordinate along the overscan, 1028 with the statistic derived from the pixels in the perpendicular 1029 direction at each location. 1030 1031 \item If specified in the user parameters, sigma-clipping must be 1032 performed on the input data values. 950 1033 951 1034 The bias subtraction must leave no residuals greater than \tbr{1 DN} 952 1035 peak-to-peak. 1036 \end{enumerate} 953 1037 954 1038 \paragraph{Trim object image} 955 956 The Phase 2 analysis must trim the non-imaging pixels from the image. 957 958 The definition of the imaging area must be determined from the 1039 \begin{enumerate} 1040 1041 \item The Phase 2 analysis must trim the non-imaging pixels from the 1042 image. 1043 1044 \item The definition of the imaging area must be determined from the 959 1045 Metadata Database\comment{or image header?}. 960 1046 961 Phase 2 must trim pixel near the edges that have been compromised due 962 to OT operation. 1047 \item Phase 2 must trim pixel near the edges that have been 1048 compromised due to OT operation. 1049 \end{enumerate} 963 1050 964 1051 \paragraph{Correct for non-linearity} 965 1052 966 If required , the science image must be corrected for the effects of967 non-linearity. The correction must be a function of chip.1053 If required by the recipe, each chip must be independently corrected for the 1054 effects of non-linearity. 968 1055 969 1056 \paragraph{Flat-field correction} 970 971 The Phase 2 analysis must divide the science image by the provided 972 flat-field image. 973 974 The division must handle zero-valued pixels in the flat-field image 975 without raising floating point exceptions, setting the corresponding 976 bit value in the mask. 977 978 The flat-field images must be appropriately normalized (see section 979 \ref{mkcal}). 980 981 The flat-fielded image must have a consistent photometric zero-point 982 across the chip, and across the full FPA, to within 0.2\% with 983 peak-to-peak deviations of \tbr{0.5\%}. 1057 \begin{enumerate} 1058 1059 \item The Phase 2 analysis must divide the science image by the 1060 provided flat-field image. 1061 1062 \item The division must handle zero-valued pixels in the flat-field 1063 image without raising floating point exceptions, setting the 1064 corresponding bit value in the mask. 1065 1066 \item The flat-field images must be appropriately normalized (see 1067 section \ref{mkcal}). 1068 1069 \item The flat-fielded image must have a consistent photometric 1070 zero-point across the chip, and across the full FPA, to within 0.2\% 1071 with peak-to-peak deviations of \tbr{0.5\%}. 1072 \end{enumerate} 984 1073 985 1074 \tbd{color of stars in flat-field correction?} 986 1075 987 1076 \paragraph{Sky \& Fringe subtraction} 988 989 The Phase 2 analysis must subtract the sky (and fringe where needed) 990 contributions from the images. 991 992 The residual after the background subtraction must have an average 993 offset of 0 counts, excluding the signal from astronomical features. 994 995 The background residuals must have peak-to-peak variations of less 996 than \tbr{1\%} of the input background amplitude. 997 998 The background residuals must have a scatter of less than \tbr{1\%} of 999 the input background amplitude for apertures of less than 1000 \tbr{10~arcsec}.\comment{derived from the need for systematic errors 1001 of better than 0.5\% and known background ranges.} 1077 \begin{enumerate} 1078 1079 \item The Phase 2 analysis must subtract the sky (and fringe where 1080 needed) contributions from the images. 1081 1082 \item The residual after the background subtraction must have an 1083 average offset of 0 counts, excluding the signal from astronomical 1084 features. 1085 1086 \item The background residuals must have peak-to-peak variations of 1087 less than \tbr{1\%} of the input background amplitude. 1088 1089 \item The background residuals must have a scatter of less than 1090 \tbr{1\%} of the input background amplitude for apertures of less 1091 than \tbr{10~arcsec}.\comment{derived from the need for systematic 1092 errors of better than 0.5\% and known background ranges.} 1093 \end{enumerate} 1002 1094 1003 1095 \paragraph{Identify `cosmic rays'} 1004 1005 The Phase 2 analysis must detect cosmic rays in single images which 1006 are brighter than a user-configurable threshold. 1007 1008 The Phase 2 analysis must mask detected cosmic rays with a unique 1009 bit value in the mask. 1010 1011 The Phase 2 analysis must extend the masked region by a 1012 user-configurable growth factor. 1013 1014 The Phase 2 analysis must perform the cosmic ray detection only if it 1015 is required by the analysis recipe. 1096 \begin{enumerate} 1097 1098 \item The Phase 2 analysis must detect cosmic rays in single images 1099 which are brighter than a user-configurable threshold. 1100 1101 \item The Phase 2 analysis must mask detected cosmic rays with a 1102 unique bit value in the mask. 1103 1104 \item The Phase 2 analysis must extend the masked region by a 1105 user-configurable growth factor. 1106 1107 \item The Phase 2 analysis must perform the cosmic ray detection only 1108 if it is required by the analysis recipe. 1109 \end{enumerate} 1016 1110 1017 1111 \paragraph{Find objects in the image} 1018 1019 The Phase 2 analysis must perform object detection on the processed 1020 images. 1021 1022 The object detection must detect all objects above a user-configured 1023 threshold. The threshold must be a positive value; negative values 1024 must invoke an error. The detection threshold must optionally be a 1025 function of the average background flux or the local noise level. 1026 1027 The object detection must measure the following object parameters: 1028 \begin{enumerate} 1029 \item object centroid and position errors 1030 \item an extended object position ($x_g, y_g$) 1031 \item instrumental PSF magnitude and error 1032 \item local background level and error 1033 \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) of the object 1034 and their covariance matrix 1035 \end{enumerate} 1036 1037 Minimal object classification must be performed to distinguish objects 1038 which are consistent with a single PSF, objects which are 1039 inconsistently large, objects which are inconsistently small, and 1040 objects which are saturated. 1041 1042 The resulting collection of detected objects must be saved along with 1043 the relevant image metadata (\ie filter, exposure time, etc). 1112 \begin{enumerate} 1113 1114 \item The Phase 2 analysis must perform object detection on the 1115 processed images. 1116 1117 \item The object detection process must detect all objects above a 1118 user-configured threshold. 1119 1120 \item The threshold must be a positive value; negative values must 1121 invoke an error. 1122 1123 \item The detection threshold must optionally be a function of the 1124 average background flux or the local noise level. 1125 1126 \item The object detection must measure the following object 1127 parameters: 1128 \begin{enumerate} 1129 \item object centroid and position errors 1130 \item an extended object position ($x_g, y_g$) 1131 \item instrumental PSF magnitude and error 1132 \item local background level and error 1133 \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) of the object 1134 and their covariance matrix 1135 \end{enumerate} 1136 1137 \item Minimal object classification must be performed to distinguish 1138 objects which are consistent with a single PSF, objects which are 1139 inconsistently large, objects which are inconsistently small, and 1140 objects which are saturated. 1141 1142 \item The resulting collection of detected objects must be saved along 1143 with the relevant image metadata (\ie filter, exposure time, etc). 1144 \end{enumerate} 1044 1145 1045 1146 \paragraph{Astrometry} 1046 1047 The Phase 2 analysis must match the detected objects with known 1048 astrometric reference objects. 1049 1050 The astrometric reference object coordinates must be adjusted for 1051 proper motion. 1052 1053 The reference and detected object coordinates must be fit to determine 1054 astrometric parameters for the individual OTAs. 1055 1056 The OTA astrometric parameters must include Chebychev polynomials of the 1057 coordinates up to 3rd order. 1058 1059 The fitted number of polynomial orders must be a user-configured 1060 parameter. 1061 1062 The Cell astrometric parameters must not be allowed to vary in the 1063 fit. 1064 1065 The fit must be robust, rejecting outlier matches (either stars with 1066 poorly determined proper motion or spurious matches). 1067 1068 The resulting astrometric solution must be consistent across the OTA 1069 field to within \tbr{300 milli-arcsec}. 1147 \begin{enumerate} 1148 1149 \item The Phase 2 analysis must match the detected objects with known 1150 astrometric reference objects. 1151 1152 \item The astrometric reference object coordinates must be adjusted 1153 for proper motion. 1154 1155 \item The reference and detected object coordinates must be fit to 1156 determine astrometric parameters for the individual OTAs. 1157 1158 \item The OTA astrometric parameters must include Chebychev 1159 polynomials of the coordinates up to 3rd order. 1160 1161 \item The fitted number of polynomial orders must be a user-configured 1162 parameter. 1163 1164 \item The Cell astrometric parameters must not be allowed to vary in 1165 the fit. 1166 1167 \item The fit must be robust, rejecting outlier matches (either stars 1168 with poorly determined proper motion or spurious matches). 1169 1170 \item The resulting astrometric solution must be consistent across the 1171 OTA field to within \tbr{300 milli-arcsec}. 1172 \end{enumerate} 1070 1173 1071 1174 \paragraph{Postage Stamps} 1072 1073 The Phase 2 analysis must extract subrasters (`postage stamps') 1074 surrounding a user-specified list of coordinates from the flattened 1075 images. 1076 1077 The postage stamp images must be saved in the IPP Image Server. 1078 1175 \begin{enumerate} 1176 1177 \item The Phase 2 analysis must extract subrasters (`postage stamps') 1178 surrounding a user-specified list of coordinates from the flattened 1179 images. 1180 1181 \item The postage stamp images must be saved in the IPP Image Server. 1182 \end{enumerate} 1183 1079 1184 \subsubsection{Phase 3 : exposure analysis} 1080 1081 The Phase 3 analysis must use the objects detected in Phase 2, matched 1082 with a user-specified reference photometry catalog, to determine the 1083 image photometric zero point and zero-point variations across the 1084 field. 1085 1086 If zero-point variations are significant \tbd{level TBD}, the 1087 zero-point variations must be modeled with a Chebychev polynomial 1088 correction of order 3 or less. 1089 1090 The photometric nature of the FPA image must be categorized 1091 \tbd{numerical scale?} on the basis of the zero-point consistency, the 1092 transparency compared with recent long-term measurements in the 1093 filter, and the external indicators of photometricity. 1094 1095 The Phase 3 analysis must use the objects detected in Phase 2, matched 1096 with an appropriate astrometric reference catalog, to improve the 1097 distortion model used for the image. 1098 1099 The resulting astrometric accuracy must be limited by the astrometric 1100 reference catalog, ie, 250 mas for USNO-B1.0. 1185 \begin{enumerate} 1186 1187 \item The Phase 3 analysis must use the objects detected in Phase 2, 1188 matched with a user-specified reference photometry catalog, to 1189 determine the image photometric zero point and zero-point variations 1190 across the field. 1191 1192 \item If zero-point variations are significant \tbd{level TBD}, the 1193 zero-point variations must be modeled with a Chebychev polynomial 1194 correction of order 3 or less. 1195 1196 \item The photometric nature of the FPA image must be categorized 1197 \tbd{numerical scale?} on the basis of the zero-point consistency, 1198 the transparency compared with recent long-term measurements in the 1199 filter, and the external indicators of photometricity. 1200 1201 \item The Phase 3 analysis must use the objects detected in Phase 2, 1202 matched with an appropriate astrometric reference catalog, to 1203 improve the distortion model used for the image. 1204 1205 \item The resulting astrometric accuracy must be limited by the 1206 astrometric reference catalog, ie, 250 mas for USNO-B1.0. 1207 \end{enumerate} 1101 1208 1102 1209 \subsubsection{Phase 4 : image combination} … … 1108 1215 1109 1216 \paragraph{Extract image pixels} 1110 1111 The Phase 4 analysis must determine the corresponding set of image 1112 pixels for a given sky cell. 1113 1114 The corresponding image pixels must be extracted from the input 1115 images, using the astrometric information for each OTA and Cell to 1116 determine the exact overlaps. 1117 1118 The Phase 4 analysis must not miss any pixels in this match, and it 1119 must read no more than 20\% more pixels than necessary from the input 1120 images. 1121 1122 The Phase 4 analysis must skip any sky cells with fewer than 5\% of 1123 their pixels overlapping the input images. 1217 \begin{enumerate} 1218 1219 \item The Phase 4 analysis must determine the corresponding set of 1220 image pixels for a given sky cell. 1221 1222 \item The corresponding image pixels must be extracted from the input 1223 images, using the astrometric information for each OTA and Cell to 1224 determine the exact overlaps. 1225 1226 \item The Phase 4 analysis must not miss any pixels in this match, and 1227 it must read no more than 20\% more pixels than necessary from the 1228 input images. 1229 1230 \item The Phase 4 analysis must skip any sky cells with fewer than 5\% 1231 of their pixels overlapping the input images. 1232 \end{enumerate} 1124 1233 1125 1234 \paragraph{Transform pixel coordinates} 1126 1127 Pixels which have been extracted from the input images must be mapped 1128 to the corresponding pixels in the sky image. 1129 1130 The transformation must be based on the measured astrometric solution 1131 for the input images relative to the reference catalog used to 1132 generate the static sky image. 1133 1134 This warping must use a locally-linear astrometric solution. 1135 1136 The output image must maintain photometric consistency with the input 1137 image to within 0.2\%. 1235 \begin{enumerate} 1236 1237 \item Pixels which have been extracted from the input images must be 1238 mapped to the corresponding pixels in the sky image. 1239 1240 \item The transformation must be based on the measured astrometric 1241 solution for the input images relative to the reference catalog used 1242 to generate the static sky image. 1243 1244 \item This warping must use a locally-linear astrometric solution. 1245 1246 \item The output image must maintain photometric consistency with the 1247 input image to within 0.2\%. 1248 \end{enumerate} 1138 1249 1139 1250 \tbd{interpolation? does interpolation method choice risk losing flux?} … … 1147 1258 1148 1259 \paragraph{Image outlier pixel rejection} 1149 1150 When multiple images are combined, the group of input pixels which 1151 contribute to an output pixel must be examined and pixels from the 1152 group of images which are inconsistent with the ensemble \tbd{how 1153 much?} must be identified and flagged. 1154 1155 This outlier rejection must be performed optionally. 1260 \begin{enumerate} 1261 1262 \item When multiple images are combined, the group of input pixels 1263 which contribute to an output pixel must be examined and pixels from 1264 the group of images which are inconsistent with the ensemble 1265 \tbd{how much?} must be identified and flagged. 1266 1267 \item This outlier rejection must be performed optionally. 1156 1268 1157 1269 \tbd{for moving objects and images which are not simultaneous, do we … … 1160 1272 \tbd{use the spatial information? fit a 2-D Nth order polynomial to 1161 1273 the collection of pixels and then look for outliers} 1274 \end{enumerate} 1162 1275 1163 1276 \paragraph{Initial cleaned image} … … 1178 1291 1179 1292 \paragraph{Find objects in the image} 1180 1181 The Phase 4 analysis must perform object detection on the difference 1182 images. 1183 1184 All objects in the difference image must be detected and the pixels 1185 belonging to variable sources flagged in the input image. 1186 1187 The object detection must detect all objects above a user-configured 1188 threshold. Both positive and negative objects must be detected; the 1189 specified threshold must define the absolute value of the detection 1190 thresholds. The detection threshold must optionally be a function of 1191 the average background flux or the local noise level. 1192 1193 The object detection must measure the following object parameters: 1194 \begin{enumerate} 1195 \item object centroid and position errors 1196 \item instrumental PSF magnitude and error 1197 \item local background level and error 1198 \item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$ 1199 \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix 1200 \end{enumerate} 1201 1202 Minimal object classification must be performed to distinguish objects 1203 which are consistent with a single PSF, objects which are 1204 inconsistent, and objects which are saturated. 1205 1206 The resulting collection of detected objects must be saved along with 1207 the relevant image metadata (\ie filter, exposure time, etc). 1293 \begin{enumerate} 1294 1295 \item The Phase 4 analysis must perform object detection on the 1296 difference images. 1297 1298 \item All objects in the difference image must be detected and the 1299 pixels belonging to variable sources flagged in the input image. 1300 1301 \item The object detection must detect all objects above a 1302 user-configured threshold. 1303 1304 \item Both positive and negative objects must be detected: the 1305 specified threshold must define the absolute value of the detection 1306 thresholds. 1307 1308 \item The detection threshold must optionally be a function of the 1309 average background flux or the local noise level. 1310 1311 \item The object detection must measure the following object parameters: 1312 \begin{enumerate} 1313 \item object centroid and position errors 1314 \item instrumental PSF magnitude and error 1315 \item local background level and error 1316 \item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$ 1317 \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix 1318 \end{enumerate} 1319 1320 \item Minimal object classification must be performed to distinguish 1321 objects which are consistent with a single PSF, objects which are 1322 inconsistent, and objects which are saturated. 1323 1324 \item The resulting collection of detected objects must be saved along 1325 with the relevant image metadata (\ie filter, exposure time, etc). 1326 \end{enumerate} 1208 1327 1209 1328 \paragraph{Cleaned Input Image} 1210 1211 The pixels flagged as being from the difference image sources must be 1212 masked in the input images. 1213 1214 A new, cleaned image must be constructed from the masked input images. 1329 \begin{enumerate} 1330 1331 \item The pixels flagged as being from the difference image sources 1332 must be masked in the input images. 1333 1334 \item A new, cleaned image must be constructed from the masked input 1335 images. 1336 1337 \end{enumerate} 1215 1338 1216 1339 \tbd{how to handle variable stars?} 1217 1340 1218 1341 \paragraph{Find objects in the image} 1219 1220 The Phase 4 analysis must perform object detection on the cleaned, 1221 summed image. 1222 1223 The object detection must detect all objects above a user-configured 1224 threshold. The threshold must be a positive value; negative values 1225 must invoke an error. The detection threshold optionally must be a 1226 function of the average background flux or the local noise level. 1227 1228 The object detection must measure the following object parameters: 1229 \begin{enumerate} 1230 \item object centroid and position errors 1231 \item an extended object position ($x_g, y_g$) 1232 \item instrumental PSF magnitude and error 1233 \item local background level and error 1234 \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their 1235 covariance matrix 1236 \item the Petrosian radius, magnitude, axis ratio, and angle 1237 \item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$. 1238 \end{enumerate} 1239 1240 Minimal object classification must be performed to distinguish objects 1241 which are consistent with a single PSF, objects which are 1242 inconsistent, and objects which are saturated. 1243 1244 The resulting collection of detected objects must be saved along with 1245 the relevant image metadata (\ie filter, exposure time, etc). 1342 \begin{enumerate} 1343 1344 \item The Phase 4 analysis must perform object detection on the 1345 cleaned, summed image. 1346 1347 \item The object detection must detect all objects above a 1348 user-configured threshold. 1349 1350 \item The threshold must be a positive value; negative values must 1351 invoke an error. 1352 1353 \item The detection threshold optionally must be a function of the 1354 average background flux or the local noise level. 1355 1356 \item The object detection must measure the following object parameters: 1357 \begin{enumerate} 1358 \item object centroid and position errors 1359 \item an extended object position ($x_g, y_g$) 1360 \item instrumental PSF magnitude and error 1361 \item local background level and error 1362 \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their 1363 covariance matrix 1364 \item the Petrosian radius, magnitude, axis ratio, and angle 1365 \item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$. 1366 \end{enumerate} 1367 1368 \item Minimal object classification must be performed to distinguish 1369 objects which are consistent with a single PSF, objects which are 1370 inconsistent, and objects which are saturated. 1371 1372 \item The resulting collection of detected objects must be saved along 1373 with the relevant image metadata (\ie filter, exposure time, etc). 1374 \end{enumerate} 1246 1375 1247 1376 \paragraph{Image Processing Q/A} … … 1281 1410 \label{mkcal} 1282 1411 1283 tbd{Requirements on the speed of processing the calibration images.}1412 \tbd{Requirements on the speed of processing the calibration images.} 1284 1413 1285 1414 The Calibration analysis stages must construct the various types of … … 1288 1417 1289 1418 \paragraph{bias images} 1290 1291 The \code{bias} calibration stage must construct a master bias image 1292 from a collection of raw bias images. 1293 1294 The \code{bias} calibration stage must correct the input images based 1295 on the overscan region. 1296 1297 The \code{bias} calibration stage must combine the input images using 1298 the statistic specified by the user, selected from one of the 1299 following: sample mean, median, and mode, robust mean, median, and 1300 mode, and the clipped mean and median. 1301 1302 The \code{bias} calibration stage must construct residual images, in 1303 which the master bias is applied to the input images. 1304 1305 Outlier residual images, those for which the residual bias and 1306 variance in the bias image are excessive ($> 1DN$), must be excluded 1307 from the input image stack the the bias image reconstructed. 1419 \begin{enumerate} 1420 1421 \item The \code{bias} calibration stage must construct a master bias 1422 image from a collection of raw bias images. 1423 1424 \item The \code{bias} calibration stage must correct the input images 1425 based on the overscan region. 1426 1427 \item The \code{bias} calibration stage must combine the input images 1428 using the statistic specified by the user, selected from one of the 1429 following: sample mean, median, and mode, robust mean, median, and 1430 mode, and the clipped mean and median. 1431 1432 \item The \code{bias} calibration stage must construct residual 1433 images, in which the master bias is applied to the input images. 1434 1435 \item Outlier residual images, those for which the residual bias and 1436 variance in the bias image are excessive ($> 1DN$), must be excluded 1437 from the input image stack the the bias image reconstructed. 1438 \end{enumerate} 1308 1439 1309 1440 \paragraph{dark images} 1310 1311 The \code{dark} calibration stage must construct a master dark image 1312 from a collection of raw dark images. 1313 1314 The \code{dark} calibration stage must raise an error if the input 1315 images have exposure time which deviate by more than \tbr{2\%}. 1316 1317 The \code{dark} calibration stage must correct the input dark images 1318 for the bias. 1319 1320 The \code{dark} calibration stage must combine the input images using 1321 the statistic specified by the user, selected from one of the 1322 following: sample mean, median, and mode, robust mean, median, and 1323 mode, and the clipped mean and median. 1324 1325 The \code{dark} calibration stage must construct residual images, in 1326 which the master dark is applied to the input images. 1327 1328 Outlier residual images, those for which the residual level and 1329 variance are excessive ($> 1DN$), must be excluded from the input 1330 image stack the the dark image reconstructed. 1441 \begin{enumerate} 1442 1443 \item The \code{dark} calibration stage must construct a master dark 1444 image from a collection of raw dark images. 1445 1446 \item The \code{dark} calibration stage must raise an error if the 1447 input images have exposure time which deviate by more than 1448 \tbr{2\%}. 1449 1450 \item The \code{dark} calibration stage must correct the input dark 1451 images for the bias. 1452 1453 \item The \code{dark} calibration stage must combine the input images 1454 using the statistic specified by the user, selected from one of the 1455 following: sample mean, median, and mode, robust mean, median, and 1456 mode, and the clipped mean and median. 1457 1458 \item The \code{dark} calibration stage must construct residual 1459 images, in which the master dark is applied to the input images. 1460 1461 \item Outlier residual images, those for which the residual level and 1462 variance are excessive ($> 1DN$), must be excluded from the input 1463 image stack the the dark image reconstructed. 1464 \end{enumerate} 1331 1465 1332 1466 \paragraph{flat-field images} 1333 1334 The \code{flat-field} calibration stage must construct a master 1335 flat-field image from a collection of raw flat-field images. 1336 1337 The \code{flat-field} calibration stage must accept a group of images 1338 from one of the following flat-field sources: dome, twilight, 1339 night-sky. 1340 1341 The \code{flat-field} calibration stage must raise an error if the 1342 input images in a single stack used more than one of the above 1343 flat-field sources or multiple filters. 1344 1345 The \code{flat-field} calibration stage must correct the input 1346 flat-field images for the bias and dark. 1347 1348 The \code{flat-field} calibration stage must combine the input images 1349 using the statistic specified by the user, selected from one of the 1350 following: sample mean, median, and mode, robust mean, median, and 1351 mode, and the clipped mean and median. 1352 1353 The \code{flat-field} calibration stage must construct residual 1354 images, in which the master flat-field is applied to the input images. 1355 1356 Outlier residual images, those for which the residual level and 1357 variance are excessive ($> 0.1$\%, or 1.02 times the Poisson limit of 1358 the flat-field image), must be excluded from the input image stack the 1359 the flat-field image reconstructed. 1467 \begin{enumerate} 1468 1469 \item The \code{flat-field} calibration stage must construct a master 1470 flat-field image from a collection of raw flat-field images. 1471 1472 \item The \code{flat-field} calibration stage must accept a group of 1473 images from one of the following flat-field sources: dome, twilight, 1474 night-sky. 1475 1476 \item The \code{flat-field} calibration stage must raise an error if 1477 the input images in a single stack used more than one of the above 1478 flat-field sources or multiple filters. 1479 1480 \item The \code{flat-field} calibration stage must correct the input 1481 flat-field images for the bias and dark. 1482 1483 \item The \code{flat-field} calibration stage must combine the input 1484 images using the statistic specified by the user, selected from one 1485 of the following: sample mean, median, and mode, robust mean, 1486 median, and mode, and the clipped mean and median. 1487 1488 \item The \code{flat-field} calibration stage must construct residual 1489 images, in which the master flat-field is applied to the input 1490 images. 1491 1492 \item Outlier residual images, those for which the residual level and 1493 variance are excessive ($> 0.1$\%, or 1.02 times the Poisson limit 1494 of the flat-field image), must be excluded from the input image 1495 stack the the flat-field image reconstructed. 1496 \end{enumerate} 1360 1497 1361 1498 \paragraph{mask images} 1362 1363 The \code{mask} calibration stage must construct a bad-pixel mask from 1364 a stack of raw flat-field images and a master flat-field image. 1365 1366 The \code{mask} calibration stage must mask the pixels which are 1367 inconsistent in the input flats by more than \tbr{1\%}, given 1368 sufficient signal-to-noise in the input flats. 1369 1370 The \code{mask} calibration stage must mask the pixels which are 1371 consistently low or high in the input flats by more than a factor of 1372 \tbr{3} beyond the typical pixel. 1373 1374 The \code{mask} calibration stage must mask the pixels identified in a 1375 table of bad pixels generated externally to the calibration stage. 1376 1377 The \code{mask} calibration stage must use multiple bit values to 1378 identify the different types of masked pixels. 1379 1380 The \code{mask} calibration stage must raise an error if the input 1381 images generate too many bad pixels in the mask. 1499 \begin{enumerate} 1500 1501 \item The \code{mask} calibration stage must construct a bad-pixel 1502 mask from a stack of raw flat-field images and a master flat-field 1503 image. 1504 1505 \item The \code{mask} calibration stage must mask the pixels which are 1506 inconsistent in the input flats by more than \tbr{1\%}, given 1507 sufficient signal-to-noise in the input flats. 1508 1509 \item The \code{mask} calibration stage must mask the pixels which are 1510 consistently low or high in the input flats by more than a factor of 1511 \tbr{3} beyond the typical pixel. 1512 1513 \item The \code{mask} calibration stage must mask the pixels 1514 identified in a table of bad pixels generated externally to the 1515 calibration stage. 1516 1517 \item The \code{mask} calibration stage must use multiple bit values 1518 to identify the different types of masked pixels. 1519 1520 \item The \code{mask} calibration stage must raise an error if the 1521 input images generate too many bad pixels in the mask. 1522 \end{enumerate} 1382 1523 1383 1524 \paragraph{fringe frames} 1384 1385 The \code{fringe} calibration stage must construct a master fringe 1525 \begin{enumerate} 1526 1527 \item The \code{fringe} calibration stage must construct a master fringe 1386 1528 frame from a stack of raw night-time sky images or from a stack of 1387 1529 dome fringe frames. 1388 1530 1389 The \code{fringe} calibration stage must raise an error if the input1531 \item The \code{fringe} calibration stage must raise an error if the input 1390 1532 stack consists is images generated with more than one type of fringe 1391 1533 source or with multiple filters. 1392 1534 1393 The \code{fringe} calibration stage must flatten the input images1535 \item The \code{fringe} calibration stage must flatten the input images 1394 1536 to remove the pixel-to-pixel sensitivity variations of the detector. 1395 1537 1396 The \code{fringe} calibration stage must measure the fringe amplitude1538 \item The \code{fringe} calibration stage must measure the fringe amplitude 1397 1539 on the input fringe images. 1398 1540 1399 The \code{fringe} calibration stage must scale the input fringe images1541 \item The \code{fringe} calibration stage must scale the input fringe images 1400 1542 based on the fringe amplitude. 1401 1543 1402 The \code{fringe} calibration stage must combine the scaled input1544 \item The \code{fringe} calibration stage must combine the scaled input 1403 1545 images using the statistic specified by the user, selected from one of 1404 1546 the following: sample mean, median, and mode, robust mean, median, and 1405 1547 mode, and the clipped mean and median. 1406 1548 1407 The \code{fringe} calibration stage must construct residual images, in1549 \item The \code{fringe} calibration stage must construct residual images, in 1408 1550 which the master fringe image is applied to the input images, along 1409 1551 with all necessary preceding calibration images. 1410 1552 1411 The \code{fringe} calibration stage must measure the residual fringe1553 \item The \code{fringe} calibration stage must measure the residual fringe 1412 1554 amplitude on the residual images. 1555 \end{enumerate} 1413 1556 1414 1557 \paragraph{low-spatial-frequency sky models} … … 1420 1563 1421 1564 \paragraph{Flat-field correction frame} 1422 1423 The \code{flat-field correction} calibration stage must construct a 1565 \begin{enumerate} 1566 1567 \item The \code{flat-field correction} calibration stage must construct a 1424 1568 flat-field correction image from dithered observations of a stellar 1425 1569 field. 1426 1570 1427 The \code{flat-field correction} calibration stage must construct a1571 \item The \code{flat-field correction} calibration stage must construct a 1428 1572 flat-field correction image for each filter and camera independently. 1429 1573 1430 The \code{flat-field correction} calibration stage must construct a1574 \item The \code{flat-field correction} calibration stage must construct a 1431 1575 correction image which makes the photometry of multiple observations 1432 1576 of the same stellar source consistent at different locations in the 1433 1577 focal plane. 1434 1578 1435 The \code{flat-field correction} calibration stage must construct1579 \item The \code{flat-field correction} calibration stage must construct 1436 1580 corrected flat-field images using the measured correction. 1437 1581 1438 The \code{flat-field correction} calibration stage must determine the1582 \item The \code{flat-field correction} calibration stage must determine the 1439 1583 consistency of the corrected flat-field images using the dithered 1440 1584 stellar field observations flattened with the corrected flat-field 1441 image.. 1585 image. 1586 \end{enumerate} 1442 1587 1443 1588 \paragraph{Non-linearity correction frames} 1444 1445 The \code{non-linear correction} calibration stage must construct a 1589 \begin{enumerate} 1590 1591 \item The \code{non-linear correction} calibration stage must construct a 1446 1592 non-linear correction from a collection of images of a constant source 1447 1593 with varying exposure times. 1448 1594 1449 The \code{non-linear correction} calibration stage must construct a1595 \item The \code{non-linear correction} calibration stage must construct a 1450 1596 non-linear correction which linearizes the detector fluxes $<0.5\%$. 1451 1597 1452 The \code{non-linear correction} calibration stage must determine the1598 \item The \code{non-linear correction} calibration stage must determine the 1453 1599 saturation regime, in which the non-linear correction is no longer 1454 1600 consistent to $<0.5\%$. 1601 \end{enumerate} 1455 1602 1456 1603 \subsubsection{Reference Catalog Creation} … … 1460 1607 future Pan-STARRS calibration. The generation of these catalogs is 1461 1608 inherently a research project, and will require human control and 1462 intervention. The IPP is required to provide the data access,1463 manipulation and visualization tools needed to construct these 1464 reference catalogs and to assess their quality. In this section, we 1465 list the requirements ofthe tools needed for this effort.1466 1467 \ subsubsection{Astrometry Reference Creation}1609 intervention. The IPP must provide the data access, manipulation and 1610 visualization tools needed to construct these reference catalogs and 1611 to assess their quality. In this section, we list the requirements of 1612 the tools needed for this effort. 1613 1614 \paragraph{Astrometry Reference Creation} 1468 1615 1469 1616 \begin{table} … … 1487 1634 \end{table} 1488 1635 1489 The IPP must have the capability of generating an astrometric 1490 reference on the basis of the observations obtained by the AP survey. 1491 The IPP must provide the analysis tools needed to generate the master 1492 astrometric reference catalog. Much of the required functionality is 1493 covered by the AP Database. 1494 1495 The Astrometry Reference creation tools must return the match between 1636 The IPP will generate an astrometric reference on the basis of the 1637 observations obtained by the AP survey. The IPP must provide the 1638 analysis tools needed to generate the master astrometric reference 1639 catalog. Much of the required functionality is covered by the AP 1640 Database. The specific requirements for the Astrometric Reference 1641 creation are listed below: 1642 1643 \begin{enumerate} 1644 \item The Astrometry Reference creation tools must return the match between 1496 1645 stars observed with Pan-STARRS and any of several astrometric 1497 1646 reference catalogs listed in Table~\ref{AstroRefs}. 1498 1647 1499 The tools must convert the reference catalog object coordinates to all1648 \item The tools must convert the reference catalog object coordinates to all 1500 1649 of the coordinate frames of relevance in the telescope and camera 1501 1650 system: 1502 \begin{ itemize}1651 \begin{enumerate} 1503 1652 \item Catalog to mean positions 1504 1653 \item Mean to apparent positions … … 1507 1656 \item focal plane to specific detector (OTA) 1508 1657 \item specific detector to detector cell 1509 \end{ itemize}1510 1511 The tools must provide the necessary calibration data: the telescope1658 \end{enumerate} 1659 1660 \item The tools must provide the necessary calibration data: the telescope 1512 1661 and camera optical distortion models and the variation of the image 1513 1662 PSF across the camera field, as a function of color. 1514 1663 1515 The tools must fit the observed stellar coordinates to the astrometric1664 \item The tools must fit the observed stellar coordinates to the astrometric 1516 1665 reference catalog coordinates to determine improved astrometric 1517 1666 solutions for both the stars and the detectors. 1518 1667 1519 The tools must determine improved telescope optical distortion models1668 \item The tools must determine improved telescope optical distortion models 1520 1669 based on the astrometric solutions. 1521 1670 1522 The tools must optionally determine the fit coefficients as a function1671 \item The tools must optionally determine the fit coefficients as a function 1523 1672 of position along, or with combinations of magnitude or color. 1524 1673 1525 The fitting method must include robust outlier rejection.1526 1527 The tools must identify objects which are detected in the catalog, but1674 \item The fitting method must include robust outlier rejection. 1675 1676 \item The tools must identify objects which are detected in the catalog, but 1528 1677 not the science image or vice-versa. 1529 1678 1530 The tools must determine average centroiding errors for each object.1531 1532 The tools must plot the fit residuals against a wide variety of1679 \item The tools must determine average centroiding errors for each object. 1680 1681 \item The tools must plot the fit residuals against a wide variety of 1533 1682 parameters: the object positions, magnitudes, colors, etc. 1534 1683 1535 The tools must allow the fit to exclude subsets of objects from the1684 \item The tools must allow the fit to exclude subsets of objects from the 1536 1685 fits on the basis of these parameters. . 1537 1686 1538 The tools must provide coordinates of the guide stars in the same frame1687 \item The tools must provide coordinates of the guide stars in the same frame 1539 1688 of reference as the normal image data. 1540 1689 1541 The tools must perform the various fitting steps for the guide stars1690 \item The tools must perform the various fitting steps for the guide stars 1542 1691 rather than for the normal image data. 1543 1544 \subsubsection{Photometry Reference Creation} 1692 \end{enumerate} 1693 1694 \paragraph{Photometry Reference Creation} 1545 1695 1546 1696 \begin{table} … … 1561 1711 \end{table} 1562 1712 1563 The IPP must have the capability of generating a photometric reference 1564 on the basis of the observations obtained by the AP survey. The IPP 1565 must provide the analysis tools needed to generate a master 1566 photometric reference catalog. Much of the required functionality is 1567 covered by the AP Database. 1568 1569 The Photometry Reference creation tools must return the match between 1713 The IPP will generate a photometric reference catalog on the basis of 1714 the observations obtained by the AP survey. The IPP must provide the 1715 analysis tools needed to generate a master photometric reference 1716 catalog. Much of the required functionality is covered by the AP 1717 Database. The specific requirements for the Photometric Reference 1718 creation are listed below: 1719 1720 \begin{enumerate} 1721 \item The Photometry Reference creation tools must return the match between 1570 1722 stars observed with Pan-STARRS and any of several photometric 1571 1723 reference catalogs listed in Table~\ref{PhotoRefs}. 1572 1724 1573 The tools must convert between different photometric systems, including:1574 \begin{ itemize}1725 \item The tools must convert between different photometric systems, including: 1726 \begin{enumerate} 1575 1727 \item instrumental to nominal detector magnitude 1576 1728 \item nominal detector magnitude to average filter system 1577 1729 \item average filter system to reference photometry system 1578 \end{ itemize}1579 1580 These transformations must account for color and airmass terms.1581 1582 The tools must measure and apply relative photometry corrections1730 \end{enumerate} 1731 1732 \item These transformations must account for color and airmass terms. 1733 1734 \item The tools must measure and apply relative photometry corrections 1583 1735 between images. 1584 1736 1585 The tools must determine photometric transformation fit coefficients1737 \item The tools must determine photometric transformation fit coefficients 1586 1738 as a function of airmass, magnitude, color and detector coordinates, 1587 1739 or with combinations of the above. 1588 1740 1589 The fitting method must include robust outlier rejection.1590 1591 The tools must reject specific objects from the fit on the basis of1741 \item The fitting method must include robust outlier rejection. 1742 1743 \item The tools must reject specific objects from the fit on the basis of 1592 1744 object locations, instrumental magnitudes, observed and reference 1593 1745 errors, and in particular time of the observations. 1594 1746 1595 The tools must plot the fit residuals against a wide variety of1747 \item The tools must plot the fit residuals against a wide variety of 1596 1748 parameters, including the object positions, magnitudes, colors, etc. 1597 1749 1598 The tools must provide photometry from the guide stars in the same1750 \item The tools must provide photometry from the guide stars in the same 1599 1751 system as observations of stars from the normal imaging data. 1600 1752 1601 The tools must perform the above fitting steps for the guide stars1753 \item The tools must perform the above fitting steps for the guide stars 1602 1754 rather than for the normal image data. 1755 \end{enumerate} 1603 1756 1604 1757 \subsection{Modules} … … 1640 1793 \subsubsection{Image Formats} 1641 1794 1642 Certain IPP programs must be able to read and write standard FITS images. 1643 1644 Certain IPP programs must be able to read and write files in modified 1645 FITS format with Pan-STARRS definitions for non-square pixel arrays. 1795 \begin{enumerate} 1796 \item Certain IPP programs must be able to read and write standard 1797 FITS images. 1798 1799 \item Certain IPP programs must be able to read and write files in 1800 modified FITS format with Pan-STARRS definitions for non-square 1801 pixel arrays. 1802 \end{enumerate} 1646 1803 1647 1804 \subsubsection{Table Formats} … … 1658 1815 externally provided reference catalogs: 1659 1816 1660 \begin{ itemize}1817 \begin{enumerate} 1661 1818 \item Hipparcos 1662 1819 \item Tycho2 … … 1667 1824 \item USNO-Bx 1668 1825 \item YBx 1669 \end{ itemize}1826 \end{enumerate} 1670 1827 1671 1828 \subsubsection{Analysis Reference Data} … … 1673 1830 The IPP must store reference data describing the following entities: 1674 1831 1675 \begin{ itemize}1832 \begin{enumerate} 1676 1833 \item Telescopes 1677 1834 \item Cameras … … 1680 1837 \item software basic parameters 1681 1838 \item computer configuration 1682 \end{ itemize}1839 \end{enumerate} 1683 1840 1684 1841 \subsection{External Interfaces} … … 1709 1866 hardware requirements addressed in this section consist of: 1710 1867 1711 \begin{ itemize}1868 \begin{enumerate} 1712 1869 \item Total Disk Volume 1713 1870 \item Total Processing Power … … 1715 1872 \item Sustained Node Network I/O 1716 1873 \item Sustained Disk I/O 1717 \end{ itemize}1874 \end{enumerate} 1718 1875 1719 1876 The report, `The Pan-STARRS Image Processing Pipeline Computational … … 1735 1892 calibration frames & 1.8 TB \\ 1736 1893 metadata db & 0.2 TB \\ 1737 AP db & 24 TB \\1894 AP db & 24 TB \\ 1738 1895 \hline 1739 1896 total & 461 TB \\ … … 1751 1908 types of data. 1752 1909 1753 The IPP must store all raw images from the first year from the AP and 1754 IVP surveys. This corresponds to 175,000 images, or 175 TB, assuming 1755 1 GB per image and compression. The IPP will require space for 200 TB 1756 of raw imagery to store the data from these two survey components 1757 along with raw calibration, test, and other raw images not in the AP 1758 and IVP surveys. 1759 1760 The IPP must store a single copy of the complete static sky in all 1761 four filters. With the assumed image sampling of 0.2 arcsec per 1762 pixel, this corresponds to 9.7 Tpix per filter, or a total of 235 TB 1763 for the 6 filters, with 2 bytes for the noise map and 2 bytes for the 1764 image map. 1765 1766 The IPP must also store other, smaller collections of data. The other 1767 components contribute only a small fraction of the data storage 1768 requirement. The metadata is a fraction of a terabyte, while the 1769 calibration frames (all master detrend frames) represent at most a few 1770 terabytes. The AP object and detection data make up a total of 24 1771 terabytes (see Table~\ref{APrates}). 1772 1773 The IPP must have storage capacity for a total of 461 TB of data. 1910 \begin{enumerate} 1911 \item The IPP must store all raw images from the first year from the 1912 AP and IVP surveys. This corresponds to 175,000 images, or 175 TB, 1913 assuming 1 GB per image and compression. The IPP will require space 1914 for 200 TB of raw imagery to store the data from these two survey 1915 components along with raw calibration, test, and other raw images 1916 not in the AP and IVP surveys. 1917 1918 \item The IPP must store a single copy of the complete static sky in 1919 all four filters. With the assumed image sampling of 0.2 arcsec per 1920 pixel, this corresponds to 9.7 Tpix per filter, or a total of 235 TB 1921 for the 6 filters, with 2 bytes for the noise map and 2 bytes for 1922 the image map. 1923 1924 \item The IPP must also store other, smaller collections of data. The 1925 other components contribute only a small fraction of the data 1926 storage requirement. The metadata is a fraction of a terabyte, 1927 while the calibration frames (all master detrend frames) represent 1928 at most a few terabytes. The AP object and detection data make up a 1929 total of 24 terabytes (see Table~\ref{APrates}). 1930 1931 \item The IPP must have storage capacity for a total of 461 TB of data. 1932 \end{enumerate} 1774 1933 1775 1934 \subsubsection{CPU Requirements} … … 1788 1947 be addressed independently. 1789 1948 1790 The IPP must perform the Phase 2 analysis within an average time of 45 1949 \begin{enumerate} 1950 \item The IPP must perform the Phase 2 analysis within an average time of 45 1791 1951 seconds per single Gigapixel camera image. The Phase 2 analysis has 1792 1952 been measured to require 3200 GHz-sec on a x86/32 bit machine, … … 1794 1954 are devoted to I/O. 1795 1955 1796 The IPP must perform the Phase 4 analysis on a set of 4 input frames1956 \item The IPP must perform the Phase 4 analysis on a set of 4 input frames 1797 1957 within an average time of 180 seconds. The Phase 4 analysis has been 1798 1958 measured to require a total of 7800 GHz-sec on an x86/32 bit machine 1799 1959 for a major frame of 4 input Gigapixel camera images. 1960 \end{enumerate} 1800 1961 1801 1962 \subsubsection{Network I/O Requirements} … … 1808 1969 and updated static sky images, the total I/O for a 180 second 1809 1970 major-frame period is: 1810 \begin{ itemize}1971 \begin{enumerate} 1811 1972 \item 8 GB from summit to Phase 2 (4 images @ 2 GB each) 1812 1973 \item 18 GB from Phase 2 to Phase 4 (3 bytes per pixel for image + … … 1815 1976 input image pixel, 4 bytes per pixel). 1816 1977 \item 9 GB from Phase 4 to Static Sky 1817 \end{ itemize}1978 \end{enumerate} 1818 1979 for a grand total of 44 GB over 180 seconds, or 244 MB/second, of 1819 1980 which 26 GB are processed by the Phase 2 nodes and 36 are processed by … … 1826 1987 read from and written to disk. For each major frame processed, the 1827 1988 total I/O to and from disk for Phase 2 is: 1828 \begin{ itemize}1989 \begin{enumerate} 1829 1990 \item 8 GB raw image from summit to Phase 2 nodes (4 images @ 2 GB each) 1830 1991 \item 8 GB raw image from Phase 2 disk to memory … … 1833 1994 + 1 byte mask). 1834 1995 \item 18 GB processed image from Phase 2 disk to Phase 4 1835 \end{ itemize}1996 \end{enumerate} 1836 1997 for a grand total of 86 GB I/O for Phase 2. Equivalently, for each 1837 1998 major frame processed, the total I/O to and from disk for Phase 4 is: 1838 \begin{ itemize}1999 \begin{enumerate} 1839 2000 \item 18 GB processed image from Phase 2 disk to Phase 4 1840 2001 \item 9 GB static image from Phase 4 disk to memory 1841 2002 \item 9 GB static image from memory to Phase 4 disk 1842 \end{ itemize}2003 \end{enumerate} 1843 2004 for a total of 36 GB I/O for Phase 4. 1844 2005 … … 1876 2037 1877 2038 \subsection{Software Integrity Tests} 1878 1879 MHPCC must test that the code does not produce memory leaks. 1880 1881 MHPCC must test that the code does not produce segmentation faults. 2039 \begin{enumerate} 2040 2041 \item MHPCC must test that the code does not produce memory leaks. 2042 2043 \item MHPCC must test that the code does not produce segmentation faults. 2044 \end{enumerate} 1882 2045 1883 2046 \subsection{Basic Unit Tests}
Note:
See TracChangeset
for help on using the changeset viewer.
