Changeset 2168
- Timestamp:
- Oct 18, 2004, 12:05:43 PM (22 years ago)
- File:
-
- 1 edited
-
trunk/doc/design/ippSDRS.tex (modified) (70 diffs)
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/design/ippSDRS.tex
r2114 r2168 1 %%% $Id: ippSDRS.tex,v 1. 5 2004-10-14 05:06:31eugene Exp $1 %%% $Id: ippSDRS.tex,v 1.6 2004-10-18 22:05:43 eugene Exp $ 2 2 \documentclass[panstarrs]{panstarrs} 3 3 … … 7 7 \shorttitle{IPP SDRS} 8 8 \author{Eugene Magnier, Paul Price, Josh Hoblitt} 9 \group{ \PS{}Algorithm Group}10 \project{ \PS{}Image Processing Pipeline}9 \group{Pan-STARRS Algorithm Group} 10 \project{Pan-STARRS Image Processing Pipeline} 11 11 \organization{Institute for Astronomy} 12 12 \version{DR} … … 26 26 DR.03 & 2004.03.25 & Section reorganization \\ \hline 27 27 DR.04 & 2004.04.13 & Most sections fleshed out \\ \hline 28 DR.05 & 2004.04.29 & Reorganisation for consistency --- PAP.\\ \hline28 DR.05 & 2004.04.29 & Reorganisation for consistency \\ \hline 29 29 \RevisionsEnd 30 30 31 31 \listoffigures 32 33 \begin{verbatim} 34 TODOs 35 - add hardware org diagram to section 3 36 - clean 3.4 system ifs: describe types of interactions, which are push which are pull? 37 - 3.5: move to 3.1? summary of requirements? 38 - add Image Server figure 39 - discuss AP DB operations: addstar, delstar, relphot, etc 40 - discuss AP DB throughput issues 41 - unify controller discussion 42 - scheduler: distinguish states 43 \end{verbatim} 44 32 45 \pagebreak 33 46 … … 40 53 \subsection{Identification} 41 54 42 This document establishes additional design requirements, beyond those43 specified in the Software Requirement Specification (PSDC-430-005), for 44 the Pan-STARRS Image Processing Pipeline (IPP) as applied to 45 Pan-STARRS 1 (PS-1), the initial demonstration telescope to be 46 constructed on Haleakala by Jan 2006. 55 This document establishes Software Design Requirements for the 56 Panoramic Survey Telescope and Rapid Response System (Pan-STARRS) 57 Image Processing Pipeline (IPP) for the prototype telescope PS-1, and 58 is a System-level controlled specification/design description document 59 in the official Pan-STARRS engineering specification tree. 47 60 48 61 \subsection{System Overview} 49 62 50 \PS{} is a survey telescope system being developed by the University 51 of Hawaii Institute for Astronomy (IfA), the Maui High Performance 52 Computing Center (MHPCC), Science Applications International 53 Corporation (SAIC), and Massachusetts Institute of Technology (MIT) 54 Lincoln Laboratory. The baseline system will consist of four 1.8m 55 telescopes, each with a 1 gigapixel camera capable of sustained image 56 rates of 2 per minute. A single initial test telescope (PS-1) will 57 be constructed on Haleakala and will see first light at the beginning 58 of 2006. The full four-telescope system (PS-4) will follow PS-1 by 59 roughly 2 years. 63 The Institute for Astronomy at the University of Hawaii is developing 64 a large optical synoptic survey telescope system, the Panoramic Survey 65 Telescope and Rapid Response System (Pan-STARRS). The science goals, 66 priorities, top-level concept of operations with associated 67 operational requirements, and system performance drivers with 68 associated system performance requirements are described in the 69 Pan-STARRS Science Goals Statement (SGS). As described in this 70 document, The system conceptual design for Pan-STARRS utilizes an 71 array of four 1.8m telescopes each with a 7 degree$^2$ field of view, 72 giving the system an \'etendue larger than all existing survey 73 instruments combined (defined as the product of the collecting area 74 $A$ multiplied by the field-of-view solid angle $\Omega$). Each 75 telescope will be equipped with a 1 billion pixel CCD camera with low 76 noise and rapid read-out, and the data will be reduced in near real 77 time to produce both cumulative static sky and difference images from 78 which transient, moving, and variable objects can be 79 detected. Pan-STARRS will be able to survey up to $\approx 6,000$ 80 degree$^{2}$ per night to a detection limit of approximately 24$^{th}$ 81 magnitude. This unique combination of sensitivity and sky coverage 82 will open up many new possibilities in time domain astronomy including 83 a major goal of surveying the Potentially Hazardous Object (PHO) 84 population down to a diameter of $\approx 300$ meters. In addition, 85 the Pan-STARRS data will be used to investigate a broad range of 86 astronomical problems of extreme current interest concerning the Solar 87 System, the Galaxy, and the Cosmos at large. A prototype single 88 telescope system, PS-1, is being developed as a preliminary step 89 before construction of the complete four telescope system. 90 91 \begin{tabular}{ll} 92 Project sponsor:& AFRL, United States Air Force \\ 93 Acquirer: & University of Hawaii Institute for Astronomy \\ 94 User: & Astronomical community \\ 95 Developer: & University of Hawaii Institute for Astronomy, participating \\ 96 & institutions, and associated subcontractors 97 \end{tabular} 60 98 61 99 \subsection{Document Overview} 100 101 The Pan-STARRS IPP Software Requirements Specification contains the 102 complete system requirements of the Pan-STARRS PS-1 IPP in order to 103 achieve the top-level performance and operational requirements 104 specified by the SCD. The requirements flow begun in the SGS and 105 continued in the SCD is further developed in this SRS to provide 106 additional derived system and subsystem requirements. 107 108 \subsection{Requirements Definitions} 62 109 63 110 The Pan-STARRS document naming scheme is PSDC-NNN-MMM-VV, where the VV … … 66 113 that series is implied. 67 114 68 Open Issues and TBDs in this document are marked \tbd{in bold, red 69 type with surrounding square brackets}. 115 Open issues (TBDs) in this document are marked \tbd{in bold red}. 116 117 Quantities which should be reviewed (TBRs) are marked \tbr{in bold 118 blue}. 119 120 \subsubsection{``Shall''} When used in this specification, the word 121 ``shall'' refers to an explicit requirement of a system component or 122 the complete system. 123 124 \subsubsection{``Should''} When used in this specification, the word 125 ``should'' refers to a desired characteristic of a system component or 126 the complete system. 127 128 \subsubsection{``Will''} When used in this specification, the word 129 ``will'' provides information about a characteristic of a related 130 system component or a complete related system. 131 132 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 70 133 71 134 \DocumentsInternalSection … … 215 278 \begin{figure} 216 279 \begin{center} 217 \resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}}280 %\resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}} 218 281 \caption{ \label{hardware} IPP Hardware Organization} 219 282 \end{center} … … 251 314 Database) are also shown. 252 315 253 %%% needs some work / move around elsewhere254 \subsection{System Interfaces}255 256 \paragraph{MOPS and other Client Science Pipelines}257 258 The Client Science Programs (CSPs) and the Moving Object Processing259 System (MOPS) are not a part of the IPP, but are external systems. We260 include them here to show the required interfaces.261 262 The CSPs and MOPS may query the Pixel DB, the Metadata DB and the263 Object DB. In addition, they may write certain fields to the object264 DB (e.g., the external identifiers and class of object) as they265 process objects, and they may retrieve pixel data from the Nodes.266 267 Since ``CSPs'' is a vague term, we now give some examples which may268 help to illustrate the functionality.269 270 One example of a CSP is a web front-end to retrieve (published) images271 and objects from the Pixel DB and Object DB.272 273 Another example would be a program interested in searching for274 transiting extrasolar planets. Such a program may periodically poll275 the Metadata DB for new processed observations in its region of276 interest (such as the Galactic Plane), retrieve the object photometry277 of all high signal-to-noise stars in the processed observations, and278 attempt to identify a planetary transit in progress.279 280 Yet another example would be a Stationary Transient Object Pipeline,281 which would periodically poll the Metadata DB for new processed282 observations, and query the Object DB for variable sources which were283 identified twice (so that they are not moving objects).284 285 316 \subsection{System Design Decisions} 286 317 287 Since \PS{}is a survey project, all data from the telescopes will be288 uniformly analysed by the \PS{}Image Processing Pipeline (IPP) and318 Since Pan-STARRS is a survey project, all data from the telescopes will be 319 uniformly analysed by the Pan-STARRS Image Processing Pipeline (IPP) and 289 320 the appropriate resulting data products made available to internal and 290 321 external science analysis systems as they become available. The … … 301 332 object photometry, and reference astrometry and photometry. 302 333 303 The IPP interacts closely with other \PS{}systems responsible for304 other aspects of the \PS{}operation, including the summit systems334 The IPP interacts closely with other Pan-STARRS systems responsible for 335 other aspects of the Pan-STARRS operation, including the summit systems 305 336 (OATS), the science object database, the Moving Object Processing 306 337 System (MOPS), and potentially other client science pipelines. … … 311 342 312 343 \begin{figure} 313 \psfig{file=pics/ImageServer,width=15cm,angle=0}344 % \psfig{file=pics/ImageServer,width=15cm,angle=0} 314 345 \caption{The components of the IPP Image Server.} 315 346 \label{fig:ImageServer} … … 331 362 computer and storage system. In order to achieve the data throughput 332 363 requirements, the IPP Image Server may distribute the images across 333 the processor nodes in an organized fashion, i.e. \associating364 the processor nodes in an organized fashion, i.e., associating 334 365 specific machines with specific detectors. It is not the 335 366 responsibility of the IPP Image Server to determine which computer … … 356 387 Image Server requires a file system which provides files in the local 357 388 file system. This may be done over many machines with a network file 358 system such as NFS or GFS. \tbd{select file system for IPP / test NFS 359 vs GFS vs Mogile, etc}. 389 system such as NFS or GFS. 360 390 361 391 The IPP Image Server provides the storage and access mechanisms, but … … 373 403 \end{itemize} 374 404 375 \ paragraph{IPP Image Server Client APIs}405 \subsubsection{IPP Image Server Client APIs} 376 406 377 407 Clients interact with the IPP Image Server with a small number of C … … 427 457 The IPP Image Server daemon uses a database to store the information 428 458 about the data storage objects, their instances, and the available 429 hardware resources. A \tt{mysql} database engine is used to manage459 hardware resources. A {\tt mysql} database engine is used to manage 430 460 the database. The database tables defined for the Image Server are 431 461 listed in Table~\ref{ImageServerTables}, and their current contents … … 458 488 data volume. 459 489 460 \ paragraph{IPP Image Server Maintenance Tools}490 \subsubsection{IPP Image Server Maintenance Tools} 461 491 462 492 The IPP Image Server provides a collection of administration tools … … 473 503 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 474 504 475 \subs ubsection{Metadata Database}505 \subsection{Metadata Database} 476 506 477 507 The IPP Metadata Database acts as a repository for all non-pixel data … … 489 519 for the Metadata Database may be collected and inserted by a separate, 490 520 dedicated process or analysis pipeline collection of processes. 521 Metadata which is large in volume or poorly structure may also be 522 stored in an appropriate container file (FITS Table, FITS Header, XML 523 File) in the Image Server with the Metadata DB providing pointers to 524 these files. 525 526 The IPP Metadata Database is a simple database system, consisting of a 527 number of simple tables without extensive inter-table links. The 528 \code{mysql} database engine will be used to drive the database. 529 530 \subsubsection{Metadata Tables} 531 \label{Metadata} 532 533 The complete contents of the Metadata Database will not be completely 534 specified until the complete collection of data analysis scripts are 535 available. Even so, we can make a good first pass at the likely 536 collection of long-term tables, and some of the temporary processing 537 tables. Table~\ref{MetadtaDBTables} lists the Metadata tables 538 identified to date for the Metadata Database. The contents of these 539 tables are outlined in Appendix~\ref{MetadataContents}, with examples 540 for the data entries and thier data types in many cases. 541 542 \subsubsection{Metadata Queries} 543 544 The IPP provides simple queries to the Metadata Database tables using 545 autocoded APIs. These queries allow for a single row or a simple 546 collection of rows based on the primary key. The format of the API is 547 identical for all Metadata tables. New tables and APIs can be added 548 to the IPP system by adding to the autocoding table description 549 files. See Appendix~\ref{Autocode} for futher information. 550 551 \begin{table} 552 \begin{center} 553 \caption{Metadata Database Tables\label{MetadataDBTables}} 554 \begin{tabular}{ll} 555 \hline 556 \hline 557 {\bf Table Name} & {\bf Description} \\ 558 \hline 559 Weather & Details on the weather, including internal temperatures. \\ 560 SkyProbe Transparency & Analysis of SkyProbe B \& V data. \\ 561 SkyProbe Absorption & Analysis of SkyProbe A data. \\ 562 SkyProbe Emission & Analysis of SkyProbe E data. \\ 563 DIMM & Summary of DIMM data analysis. \\ 564 NIR & Summary statistics from NIR camera. \\ 565 Dome Status & The time history of the dome status. \\ 566 Telescope Status & The time history of the telescope status. \\ 567 Raw FPAs & Information about the raw FPA exposures. \\ 568 Pending Science Chips & Science images to be processed and status. \\ 569 Processed Science Chips & Science images which have been migrated to the processed state. \\ 570 Observation Group & Details about a group of associated observations. \\ 571 Observation Frame & Major frame information. \\ 572 Science Processing stats & Details on processed cells. \\ 573 Chip / Sky overlaps & List of overlaps between sky cells and detectors. \\ 574 Processed Sky-Cell stats & Details of the sky cell processing. \\ 575 \hline 576 \end{tabular} 577 \end{center} 578 \end{table} 491 579 492 580 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 493 581 494 \paragraph{Metadata Tables} 495 496 Table \tbd{NN} lists the Metadata tables identified for the Metadata 497 Database. 498 582 \subsection{AP Database} 583 584 The AP (Astrometry \& Photometry) Database is a mechanism to store 585 data related to astronomical objects derived from various sources with 586 a variety of associations. The AP Database deals with two related 587 concepts: {\em objects} and {\em detections}. The objects are 588 descriptions of astronomical objects while the detections are the 589 specific measurements of those objects, typically measured from 590 astronomical images. A collection of {\em detections} may be used to 591 derive average quantities which describe a particular {\em object}. A 592 third class of object information which must also be considered are 593 those supplied by external references. These may be treated as {\em 594 detections}, with the caveat that access to the raw measurements and 595 metadata are usually unavailable; the reported measurements and errors 596 must be accepted as they are reported. 597 598 The AP Database stores the collections of detections which were 599 derived from specific images from any of the analysis stages. It 600 provides a mechanism to determine and (in conjunction with the Image 601 Server) locate the image from which a specific detection was derived. 602 The AP Database also makes it possible to extract all detections 603 derived from a specific image and to determine quantities such as the 604 coordinates of the detection in pixel coordinates on the image. 605 606 The AP Database also has the capability to associate multiple 607 detections of a specific object. Several major classes of objects 608 will be present, each of which must be handled correctly. 609 610 First, the most distant stars, compact galaxies, and QSOs will have 611 nearly fixed locations relative to other nearby stars, with only small 612 deviations for individual measurements. The association between 613 multiple detections of such objects is made on the basis of their 614 coincident positions. The AP Database determines the average position 615 of the object and the deviations of the individual detections from 616 that average on the basis of the ensemble of individual detection. 617 618 Second, solar system objects do not have a fixed location. Detections 619 of such objects are linked by their orbits, and depend on both the 620 position and the time of the image. The AP Database does not attempt 621 to make this link, which is the role of the MOPS system. However, it 622 has the ability to accept identifications made externally with 623 specified detections and to return the identifier of the moving object 624 associated with the specific detections. These associations also 625 include descriptive information such as the offset of the detection 626 from the predicted location of the detection based on the orbit. This 627 functionality is required to allow the AP Database to ignore known 628 moving object detections from other types of queries. 629 630 Third, stars in the general vicinity of the solar system fall in 631 between these first two classes of objects. Their proper motion and 632 parallax response is significant enough ($>1$ arcsec in 1 year) that 633 they are not well-described by an average location and a collection of 634 offsets. These objects are described by a distance and a proper 635 motion vector. The AP Database provides the association between the 636 specific detections and an average object which includes finite 637 parallax and proper motion. 638 639 Fourth, many detections, especially in their initial states, will not 640 be associated with a specific astronomical object of any of the above 641 classes and are treated as orphans. Most of these will be spurious 642 (not represent real objects), some will be from solar system objects 643 for which orbits are not yet determined, some will be from faint stars 644 near the detection limits, some will be from short-term transients 645 which have only been detected once. The AP Database maintains these 646 detections until they have been associated with one of the objects 647 above. The AP Database provides mechanisms by which individual 648 detections may be migrated back and forth between the orphan state and 649 association with an astronomical object. 650 651 For every object, and all orphaned detections, the AP Database also 652 provides the capability to determine the images which observed the 653 location of the object but for which no detection was made. The 654 minimum set of information which must be carried for these 655 non-detections is the image and the associated object or orphan. 656 657 The AP Database also stores the relationships between various 658 photometric systems and, in some cases, the evolution of that 659 relationship. It provides mechanisms to convert between the measured 660 instrumental magnitude of a detection with a specific filter, 661 detector, and telescope, and at a particular time and the implied 662 magnitude in the average Pan-STARRS photometry system, given a 663 determined set of calibrations. It also provides the capability to 664 convert magnitudes in one system to the magnitudes in another system; 665 an example of such a conversion is between the average Pan-STARRS 666 filter systems and the various reference systems appropriate for those 667 filters. 668 669 The AP Database provides interfaces to extract lists of objects and 670 detections based on various query parameters. It provides the 671 capability to extract all detections associated with a specific 672 object, all non-detections of that object, all non-detections of an 673 orphan, and summary statistics from these collections. It will also 674 return all objects or detections within specified spatial regions 675 including regions bounded by great circles (RA,DEC; GLAT,GLON; 676 ELAT,ELON) and regions described by a location and a search radius. 677 It will also return the image parameters associated with a specific 678 detection including image coordinates of the detection, exposure time, 679 time and date of the detection, etc. 680 681 The IPP AP Database consists of the following components: 682 683 \begin{itemize} 684 \item AP Database servers 685 \item AP Database client APIs 686 \item AP Database storage hardware 687 \item AP Database database engine 688 \item AP Database database tables 689 \end{itemize} 690 691 \subsubsection{AP Database Tables} 692 693 The AP Database divides the sky into a regions, which are in turn 694 sub-divided into regions, in a hierarchical series. The regions are 695 used to subdivide the tables of images, objects, and detections. 696 These three tables are the three largest in terms of both data volume 697 and number of rows. Since nearly all interactions with the AP 698 Database performed by the IPP are limited in spatial coverage, 699 subdividing the tables allows a specific interaction to search only a 700 small subset of the data. The table of images is the smallest of the 701 three; the table of detections is likely to be the largest. As a 702 result, the images tables will be subdivided at a shallow hierarchical 703 level, while the objects and detections are subdivided on deeper (more 704 finely sampled) levels. The region table defines the sky regions and 705 specifies if the region corresponds to an image table, and object 706 table, and/or a detection table. It also specified which regions in 707 the next level of the hierarchy are contained by the region, and which 708 parent region it belongs to. In addition to improving the spatial 709 access to the image, object, and detection data, the region table 710 allows for the multiple computers to serve the database tables. The 711 region file specifies the machine which stores the specific table. 712 713 The table of Images lists all of the images which provided the data in 714 the AP Database. In general, these images correspond to the Chips. 715 \tbd{how does the AP Database know about the relationship between a 716 collection of chips?}. This table includes sufficient astrometric 717 parameters to represent the coordinates of the detections to a 718 sufficient accuracy: \tbr{3rd order polynomial across the chip?}. 719 \tbr{does the AP Database know about FPA, Chip, Distortion Model, etc? 720 I think it probably needs to if it is going to solve for distortion 721 models. however, this operation may be a combination of AP DB 722 interaction and MD DB interaction.} 723 724 The Images in the image table group are stored in the Image table 725 which contains the (center? 0,0 pixel?) of the chip. A specific 726 coordinate can be specified to a single Image region table. However, 727 it is frequently useful to determine all regions which a specific 728 image overlaps. The Image Overlaps tables contains a list of the 729 image regions which are overlapped by each image. 730 731 The Objects table group (divided by region) stores the average 732 parameters for each astronomical object. Certain details of this 733 table have not yet been specified. In particular, objects with 734 significant parallax and/or proper motion may potentially be stored in 735 a distinct table. Solar system objects, to the extent average 736 properties are maintained, are certainly stored in a separate table. 737 A related table, also divided in the same regions, is the Average 738 Magnitudes table. In this table, there are multiple rows per average 739 object, one for each of the primary filters of interest for which 740 photometric averaging is performed. 741 742 The Detections table stores all of the measurements of astronomical 743 objects on specific images. \tbd{is this table divided into P2, P4S, 744 P4D tables? 3$\sigma$ objects vs 5$\sigma$ objects? We don't want to 745 store all detections in a single table, I think}. 746 747 The Non-detections table stores information about detection failures 748 for each object. If an image is added to the database which overlaps 749 an object but the object is not detected, an entry is made in this 750 table. In fact, this table may store only the most recent 751 non-detection and the total number, or a similar reduced set of 752 non-detection statistics. 753 754 The Filters table identifies all of the physical filters (specific, 755 named pieces of glass) known to the system. A related table, 756 photcodes, defines relationships between specific photometry systems. 757 A system may consist of a detector, telescope, and specific filter, or 758 it may be a derived photometry system. \tbd{distinguish between 759 reference, average, and detection photcodes}. 760 761 \subsubsection{AP Database servers} 762 763 The AP Database functions on a group of computers, with portions of 764 the tables stored on separate machines, as described above. The 765 association between a machine and the corresponding table or part of 766 the sky is defined by the Region table. Each machine has a 767 corresponding AP Database server which runs on that machine to 768 interact with the tables available on that machine. A client chooses 769 one of the machines and sends its query or data to be inserted to that 770 machine. The server then uses the region table to determine which 771 machines contain the relevant portion of the sky. The data to be 772 inserted is divided into corresponding region chunks and sent to the 773 appropriate servers. In the case of queries, the queries are 774 redirected to the appropriate server(s). The original server may 775 collect the results and return them to the original client. 776 777 \subsubsection{AP DB Operations} 778 779 \begin{itemize} 780 \item addstar 781 \item delstar 782 \item relphot 783 \item uniphot 784 \item mosastro 785 \item distortion 786 \end{itemize} 787 788 \begin{table} 789 \begin{center} 790 \caption{AP Detection Classes \& Object Parameters\label{APdetections}} 791 \begin{tabular}{lrrrr} 792 \hline 793 \hline 794 Object Parameter & P2 & P4S & P4D & SS \\ 795 \hline 796 PSF x,y, covar, $\alpha,\delta$ & + & + & + & + \\ 797 PSF mag, $\sigma_{\rm mag}$ & + & + & + & + \\ 798 star/gal sep & + & + & + & + \\ 799 $\sigma_x$, $\sigma_y$, $\theta$ & + & + & + & + \\ 800 local sky data & + & + & + & + \\ 801 Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ 802 S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ 803 W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ 804 exp. spaced aps., Poisson noise, variance & - & - & - & + \\ 805 \hline 806 \end{tabular} 807 \end{center} 808 \end{table} 809 810 \begin{table} 811 \begin{center} 812 \caption{AP Database Tables\label{APDBTables}} 499 813 \begin{tabular}{ll} 500 814 \hline 501 \multicolumn{2}{l}{\bf Metadata Tables} \\ 502 Weather & Details on the weather, including internal temperatures. \\ 503 SkyProbe & Analysis of SkyProbe data. \\ 504 LRProbe & Analysis of LRProbe data. \\ 505 DIMM & Analysis of DIMM data. \\ 506 NIR & Analysis of NIR data. \\ 507 Dome Status & The status of the dome. \\ 508 Telescope Status & The status of the telescope. \\ 509 Raw FPAs & Details on raw FPA exposures. \\ 510 Raw Chips & Details on raw chips. \\ 511 Raw Cells & Details on raw cells. \\ 512 Observation Group & Details on a group of observations to be processed. \\ 513 Chip Guide Stars & Details on guide stars \\ 514 Science Chip stats & Details on processed chips. \\ 515 Science Cell stats & Details on processed cells. \\ 516 Science FPA stats & Details on processed FPAs. \\ 517 Sky-Detector overlaps & List of overlaps between sky cells and detectors. \\ 518 Processed Sky-Cell stats & Details on sky cells. \\ 519 Calibration 1 input stats & Details on input images for Cal 1. \\ 520 Calibration 1 output stats & Details on output detrend images from Cal 1. \\ 521 Calibration 2 input stats & Details on input images for Cal 2. \\ 522 Calibration 2 output stats & Details on output detrend images from Cal 2. \\ 523 Calibration 3 input stats & Details on input images for Cal 3. \\ 524 Calibration 3 output stats & Details on output detrend images from Cal 3. \\ 815 \hline 816 {\bf Table Name} & {\bf Description} \\ 817 \hline 818 Region Table & spatial distribution of tables \\ 819 Images & The images that have objects in the DB. \\ 820 Image Overlaps & Image regions which are touched by specific images. \\ 821 Objects & The objects --- average properties of multiple detections of the same object. \\ 822 Average Magnitudes & Average photometry in multiple filters \\ 823 Detections & Detections of sources in an image. \\ 824 Non-Detections & Non-detections of objects in an image. \\ 825 Filters & Filters understood by the system. \\ 826 Photcodes & Transformations between different photometric systems \\ 827 Database Machines & computers used to store the tables \\ 525 828 \hline 526 829 \end{tabular} 830 \end{center} 831 \end{table} 527 832 528 833 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 529 834 530 \paragraph{Metadata Table Contents} 531 532 Tables \tbd{NN} -- \tbd{NN} list the basic contents of each of the Metadata tables 533 listed above. 534 535 \begin{tabular}{ll} 536 \hline 537 \multicolumn{2}{l}{\bf Weather} \\ 538 Time & The time the weather information was measured. \\ 539 Temperature & The temperature at \tbd{some place. Will likely want temperatures for a range of locations: 540 external, dome, secondary, primary for starters.} \\ 541 Humidity & The relative humidity. \\ 542 Pressure & The (external) atmospheric pressure. \\ 543 \hline 544 \end{tabular} 545 546 \begin{tabular}{ll} 547 \hline 548 \multicolumn{2}{l}{\bf SkyProbe} \\ 549 Time & The time the SkyProbe image was taken. \\ 550 Filter & Filter used for SkyProbe image. \\ 551 Transparency & The derived transparency. \\ 552 Error in transparency & The error in the derived transparency. \\ 553 Number of stars & The number of stars used to measure the transparency. \\ 554 Astrometry & The astrometry used on the SkyProbe image. \\ 555 Exposure time & The exposure time of the SkyProbe image. \\ 556 Sky brightness & The measured sky (surface) brightness, in physical units. \\ 557 \hline 558 \end{tabular} 559 560 \begin{tabular}{ll} 561 \hline 562 \multicolumn{2}{l}{\bf LRProbe} \\ 563 Time & The time the LRProbe observation was taken. \\ 564 A band absorption & The absorption EW of the atmospheric A band. \\ 565 B band absorption & The absorption EW of the atmospheric B band. \\ 566 Absorption component 3 & The absorption EW by some other atmospheric component. \\ 567 Emission 1 & The emission EW of some sky line. \\ 568 emission 2 & The emission EW of another sky line. \\ 569 emission 3 & The emission EW of some other sky line. \\ 570 Number of stars & Number of stars used to measure the absorptions. \\ 571 Astrometry & The astrometry used on the LRProbe image. \\ 572 Exposure time & The exposure time of the LRProbe image. \\ 573 Sky brightness & The measured sky (surface) brightness, in physical units. \\ 574 \hline 575 \end{tabular} 576 577 \begin{tabular}{ll} 578 \hline 579 \multicolumn{2}{l}{\bf DIMM} \\ 580 Time & The time the DIMM observation was taken. \\ 581 $\sigma_x$ & \tbd{The dispersion in $x$}. \\ 582 $\sigma_y$ & \tbd{The dispersion in $y$}. \\ 583 FWHM & The seeing full width at half maximum. \\ 584 Star coordinates & The coordinates of the measured star. \\ 585 Exposure time & The exposure time of the DIMM observation. \\ 586 \hline 587 \end{tabular} 588 589 \begin{tabular}{ll} 590 \hline 591 \multicolumn{2}{l}{\bf NIR} \\ 592 Time & The time the NIR observation was taken. \\ 593 Sky brightness & The sky (surface) brightness in the NIR observation. \\ 594 Sky variance & The variance in the sky (surface) brightness. \\ 595 Astrometry & The astrometry used on the NIR image. \\ 596 \hline 597 \end{tabular} 598 599 \begin{tabular}{ll} 600 \hline 601 \multicolumn{2}{l}{\bf Dome Status} \\ 602 Time & The time for which the dome status is valid. \\ 603 Azimuth & The azimuth of the dome. \\ 604 Open status & Whether the dome is open or not. \\ 605 Lights status & Whether lights are on in the dome or not. \\ 606 \hline 607 \end{tabular} 608 609 \begin{tabular}{ll} 610 \hline 611 \multicolumn{2}{l}{\bf Telescope Status} \\ 612 Time & The time for which the telescope status is valid. \\ 613 Guide status & The status of the guiding. \\ 614 Altitude & The telescope altitude. \\ 615 Azimuth & The telescope azimuth. \\ 616 RA & The telescope Right Ascension (ICRS $\approx$ J2000). \\ 617 Dec & The telescope Declination (ICRS $\approx$ J2000).\\ 618 \hline 619 \end{tabular} 620 621 \begin{tabular}{ll} 622 \hline 623 \multicolumn{2}{l}{\bf Raw FPAs} \\ 624 Coords & Coordinates of the boresight (i.e. telescope pointing). \\ 625 Filter & Filter used for the exposure. \\ 626 Exposure status & Status of the exposure. \\ 627 Exposure time & Exposure time for the image. \\ 628 Airmass & Airmass at which the image was taken. \\ 629 ObsGroup ID & \tbd{The ObsGroup identification number.} \\ 630 Observer & The name of the observer, or the version of the telescope scheduler software. \\ 631 Program & The observing program being executed. \\ 632 Number of chips & The number of chips that comprise the FPA. \\ 633 NX, NY & \tbd{Assuming the chips are laid out rectilinearly,} the number of chips in each dimension. \\ 634 Astrometry & The astrometry used for the FPA. \\ 635 \hline 636 \end{tabular} 637 638 \begin{tabular}{ll} 639 \hline 640 \multicolumn{2}{l}{\bf Raw Chips} \\ 641 i, j & \tbd{Assuming a rectilinear FPA,} the chip number in each dimension. \\ 642 ID & Chip identification number. \\ 643 temps & The chip temperature. \\ 644 Astrometry & The astrometry used for the chip. \\ 645 Number of cells & The number of component cells. \\ 646 NX, NY & \tbd{Assuming the cells are rectilinear,} the number of cells in each dimension. \\ 647 \hline 648 \end{tabular} 649 650 \begin{tabular}{ll} 651 \hline 652 \multicolumn{2}{l}{\bf Raw Cells} \\ 653 Astrometry & The astrometry used for the cell. \\ 654 Validity & Is the cell working? \\ 655 \hline 656 \end{tabular} 657 658 \begin{tabular}{ll} 659 \hline 660 \multicolumn{2}{l}{\bf Observation Group} \\ 661 ID & Identification number for the observation group. \\ 662 Number of images & Number of images in the observation group. \\ 663 Type & Type of observation. \\ 664 Status & Status of the observation group. \\ 665 \tbd{etc} & \\ 666 \hline 667 \end{tabular} 668 669 \begin{tabular}{ll} 670 \hline 671 \multicolumn{2}{l}{\bf Chip guide stars} \\ 672 Chip ID & The identification number for the chip. \\ 673 Guide Star ID & The identification number for the guide star. \\ 674 X, Y & The centroided pixel coordinates of the guide star. \\ 675 RA, DEC & The sky coordinates of the guide star. \\ 676 $\sigma_{x}$, $\sigma_{y}$ & The dispersion in the centroids over the particular exposure.\\ 677 $\Delta X_{\rm max}$, $\Delta Y_{\rm max}$ & The maximum deviation in the centroid over the 678 particular exposure. \\ 679 \hline 680 \end{tabular} 681 682 \begin{tabular}{ll} 683 \hline 684 \multicolumn{2}{l}{\bf Science Chip stats} \\ 685 Chip ID & The chip identification number. \\ 686 State & \tbd{The state of the processing.} \\ 687 Major frame & \tbd{The major frame the chip belongs to.} \\ 688 ObsGroup & The observation group the science exposure belongs to. \\ 689 P1 astrom & The Phase 1 astrometry. \\ 690 P2 astrom & The Phase 2 astrometry. \\ 691 P3 astrom & The Phase 3 astrometry. \\ 692 Number of guide stars & Number of guide stars used for the exposure. \\ 693 Bias correction method & Method used to correct the bias. \\ 694 Bias stats & Summary statistics for bias (mean, number of parameters, deviation of residuals, 695 bias section used). \\ 696 Flat-field image & The flat-field image that was applied. \\ 697 Kernel convolution parameters & A description of the OT kernel. \\ 698 Flat-field stats & Summary statistics for flat-field (sigma of sky). \\ 699 Mask image & The mask image that was applied. \\ 700 Masking algorithm & \tbd{The algorithm used to mask the bad pixels.} \\ 701 Fringe images & The fringe model images that were used. \\ 702 Fringe stats & Summary statistics for fringes (fringe amplitude, sky sigma) \\ 703 Object detection stats & Summary statistics for object detection (number of objects, depth, other 704 input parameters). \\ 705 Updated astrometry & \tbd{Updated astrometry parameters.} \\ 706 Astrometry stats & Summary statistics for astrometry (number of stars, $sigma_x$, $sigma_y$) \\ 707 Reference catalog & The reference catalog that was used for the astrometry. \\ 708 Updated photometry parameters & The parameters used to update the photometry: magnitude zero point 709 and other corrections. \\ 710 Photometry stats & Summary statistics for the photometry (number of stars, $sigma_m$) \\ 711 Reference catalog & The reference catalog that was used for the photometry. \\ 712 PSF stats & Summary statistics of the PSF. \\ 713 Chip state & \tbd{The state of the chip?} \\ 714 Software versions & Versions of each of the modules used in the processing. \\ 715 \hline 716 \end{tabular} 717 718 \begin{tabular}{ll} 719 \hline 720 \multicolumn{2}{l}{\bf Science Cell stats} \\ 721 Bias stats & Summary statistics for the bias (mean, parameters, dispersion of residuals, biassec) \\ 722 P1 astrom & The Phase 1 astrometry. \\ 723 P2 astrom & The Phase 2 astrometry. \\ 724 P3 astrom & The Phase 3 astrometry. \\ 725 \hline 726 \end{tabular} 727 728 \begin{tabular}{ll} 729 \hline 730 \multicolumn{2}{l}{\bf Science FPA stats} \\ 731 FPA ID & The FPA identification number. \\ 732 State & \tbd{The state of the FPA.} \\ 733 P1 astrom & The Phase 1 astrometry. \\ 734 P1 astrom stats & Summary statistics for the Phase 1 astrometry (number of stars, $\sigma_x$, $sigma_y$). \\ 735 P1 reference catalog & The reference catalog that was used for the astrometry. \\ 736 P1 software versions & The versions of each of the modules used in the Phase 1 processing. \\ 737 P1 bright stars & Pointers to the bright stars in the field. \\ 738 P1 ghosts & Pointers to the ghosts in the field. \\ 739 P1 large objects & Pointers to the large astronomical objects in the field. \\ 740 P1 PSF model & Description of the PSF model used in Phase 1. \\ 741 P3 astrom & The Phase 3 astrometry. \\ 742 P3 astrom stats & Summary statistics for the Phase 3 astrometry (number of stars, $sigma_x$, $sigma_y$). \\ 743 P3 reference catalog & The reference catalog that was used for the astrometry. \\ 744 P3 photom & The Phase 3 photometry. \\ 745 P3 photom stats & Summary statistics for the Phase 3 photometry (number of stars, $sigma_m$). \\ 746 P3 reference catalog & The reference catalog that was used for the photometry. \\ 747 P3 PSF model & Description of the PSF model used in Phase 3. \\ 748 P3 software versions & The versions of each of the modules used in the Phase 3 processing. \\ 749 \hline 750 \end{tabular} 751 752 \begin{tabular}{ll} 753 \hline 754 \multicolumn{2}{l}{\bf Sky-Detector overlaps} \\ 755 Chip ID & The identification number of the chip. \\ 756 Sky Cell ID & The identification number of the sky cell. \\ 757 State & \tbd{The state of the processing?} \\ 758 \hline 759 \end{tabular} 760 761 \begin{tabular}{ll} 762 \hline 763 \multicolumn{2}{l}{\bf Processed Sky-Cell stats} \\ 764 Input Chips & Identification numbers of the chips used to produce the sky cell. \\ 765 PSF adjustments & \tbd{Adjustments to the PSF.} \\ 766 CR rejection stats & Statistics from the CR rejection (number of CRs, distribution, limiting flux). \\ 767 Image combination parameters & Parameters used for the image combination. \\ 768 Difference image parameters & Parameters used for the image differencing. \\ 769 Average reference image depth / weight & \tbd{The weight of the reference image?} \\ 770 Difference image object detection stats & Summary statistics of the object detection (number of objects, 771 depth, other input parameters). \\ 772 Summed image object detection stats & Summary statistics of the object detection (number of objects, 773 depth, other input parameters). \\ 774 Software versions & Software versions of modules used in the sky cell processing. \\ 775 Processing stats & Summary statistics of the processing (CPU time, etc). \\ 776 \hline 777 \end{tabular} 778 779 \begin{tabular}{ll} 780 \hline 781 \multicolumn{2}{l}{\bf Calibration 1 input stats} \\ 782 Input ID & The input chip identification number. \\ 783 Output ID & The identification number of the output detrend image. \\ 784 State & \tbd{State of the processing?} \\ 785 Accepted? & Is the detrend image of acceptable quality? \\ 786 Image stats & Assorted image statistics (mean flux, exposure time, airmass) \\ 787 Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ 788 \hline 789 \end{tabular} 790 791 \begin{tabular}{ll} 792 \hline 793 \multicolumn{2}{l}{\bf Calibration 1 output stats} \\ 794 Output ID & The identification number of the output detrend image. \\ 795 Data type & The type of the detrend image (bias | dark | flat) \\ 796 Number accepted & Number of accepted input images that contributed. \\ 797 Number rejected & Number of rejected input images (no contribution). \\ 798 Summary stats & Summary statistics of the combination (deviation, normalisations). \\ 799 Applicability period & The time period the detrend image is applicable for. \\ 800 Software versions & The software versions of the modules used in processing. \\ 801 Processing stats & Summary statistics of the processing (CPU time, etc). \\ 802 \hline 803 \end{tabular} 804 805 \begin{tabular}{ll} 806 \hline 807 \multicolumn{2}{l}{\bf Calibration 2 input stats} \\ 808 Input ID & The input chip identification number. \\ 809 Output ID & The identification number of the output detrend image. \\ 810 State & \tbd{State of the processing?} \\ 811 Accepted? & Is the detrend image of acceptable quality? \\ 812 Image stats & Assorted image statistics (mean flux, exposure time, airmass). \\ 813 Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ 814 Applied reduction & \tbd{Reduction method used?} \\ 815 Applied params & Parameters of reduction. \\ 816 \hline 817 \end{tabular} 818 819 \begin{tabular}{ll} 820 \hline 821 \multicolumn{2}{l}{\bf Calibration 2 output stats } \\ 822 Output ID & The identification number of the output detrend image. \\ 823 Data type & The type of the detrend image (bias | dark | flat) \\ 824 Number accepted & Number of accepted input images that contributed. \\ 825 Number rejected & Number of rejected input images (no contribution). \\ 826 Summary stats & Summary statistics of the combination (deviation, normalisations). \\ 827 Applicability period & The time period the detrend image is applicable for. \\ 828 Software versions & The software versions of the modules used in processing. \\ 829 Processing stats & Summary statistics of the processing (CPU time, etc). \\ 830 \hline 831 \end{tabular} 832 833 \begin{tabular}{ll} 834 \hline 835 \multicolumn{2}{l}{\bf Calibration 3 input stats} \\ 836 Input ID & The input chip identification number. \\ 837 Output ID & The identification number of the output detrend image. \\ 838 State & \tbd{State of the processing?} \\ 839 Accepted? & Is the detrend image of acceptable quality? \\ 840 Image stats & Assorted image statistics (mean flux, exposure time, airmass). \\ 841 Residual stats & Statistics of the residual image (mean, sigma, clipped sigma) \\ 842 Applied reduction & \tbd{Reduction method used?} \\ 843 Applied params & Parameters of reduction. \\ 844 \hline 845 \end{tabular} 846 847 \begin{tabular}{ll} 848 \hline 849 \multicolumn{2}{l}{\bf Calibration 3 output metadata } \\ 850 Input images & Identification numbers of the input chips. \\ 851 Input image stats & Summary statistics of the input chips. \\ 852 Input object summary stats & Summary statistics of the objects on the input chips (number, density, etc) \\ 853 Object rejection criteria & Parameters of the rejection step. \\ 854 Phot stats & Summary statistics of the relative photometry (Mcal, dMcal, Klam, etc, bin size) \\ 855 Residual stats & Summary statistics of the residuals. \\ 856 Output image params & Parameters of the output image (size, etc) \\ 857 \hline 858 \end{tabular} 859 860 \begin{tabular}{ll} 861 \hline 862 \multicolumn{2}{l}{\bf Astrometric Reference Generation output metadata } \\ 863 \hline 864 \end{tabular} 865 866 \begin{tabular}{ll} 867 \hline 868 \multicolumn{1}{l}{\bf Photometric Reference Generation output metadata } \\ 869 \hline 870 \end{tabular} 871 872 \begin{tabular}{ll} 873 \hline 874 \multicolumn{2}{l}{\bf Reference Data} \\ 875 \hline 876 \end{tabular} 877 878 \begin{tabular}{ll} 879 \hline 880 \multicolumn{2}{l}{\bf Configuration Data} \\ 881 \hline 882 \end{tabular} 883 884 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 885 886 \paragraph{Metadata Queries} 887 888 \tbd{How is the Metadata DB queried?} 889 890 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 891 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 892 893 \subsubsection{Object Database} 894 895 The IPP Object Database (IOD) acts as a repository for data on all 896 astronomical objects. This database is required to provide organized 897 access to objects on the sky, including the access to the photometry 898 associated with specific input images, moving objects associated with 899 specific chips. Detailed requirements for the IOD are described in 900 \tbd{the IOD subsystem specification document xxx-xxx-xxxx}. 901 902 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 903 904 \paragraph{Object DB Tables} 905 906 \begin{tabular}{ll} 907 \hline 908 \multicolumn{2}{l}{\bf Object DB Tables} \\ 909 Images & The images that have objects in the DB. \\ 910 Objects & The objects --- average properties of multiple detections of the same object. \\ 911 Detections & Detections of sources in an image. \\ 912 Non-Detections & Non-detections of objects in an image. \\ 913 Filters & Filters understood by the system. \\ 914 Photcodes & \tbd{Transformations between different photometric systems?} \\ 915 Bright Objects & \tbd{Links to postage stamp images of bright objects.} \\ 916 Region Tables & \tbd{???} \\ 917 Average Magnitudes & \tbd{How is this different from an `object'?} \\ 918 USNO Objects & Objects from the USNO database. \\ 919 Reference Objects & The reference catalogs for astrometry and photometry. \\ 920 \hline 921 \end{tabular} 922 923 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 924 925 \paragraph{Object DB Table Contents} 926 927 \tbd{Dunno yet} 928 929 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 930 931 \paragraph{Object DB Queries} 932 933 \tbd{Dunno yet} 934 935 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 936 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 937 938 \subsubsection{Controller} 939 940 \tbd{can a process send a message back to the controller before 941 process is complete? messages via controller?} 942 943 \tbd{does the controller or the image server decide if a machine is 944 offline or both?} 945 946 \tbd{I/O tasks vs CPU tasks?} 947 948 The IPP Controller is responsible for managing the processing stages. 949 The Controller manages the parallel processing of these stages in the 950 IPP computer hardware environment and reports the completion to the 951 Scheduler. The Controller must be able to manage more than a single 952 processing thread to make maximum use of available processor 953 resources. 954 955 The Controller must honour demands that a processing stage run on a 956 particular Node. Requests that a processing stage run on a particular 957 node should be honoured if possible. Where no restriction is placed 958 on the choice of Node choice by the Scheduler, the processing stage 959 may be run on any available Node. 835 \subsection{Controller} 836 837 The IPP uses a group of computers to store and process images and to 838 manipulate collections of detections. These computers perform any of 839 a large number of analysis stages or other processing tasks without 840 significant interprocess communication. It is necessary to have a 841 mechanism which initiates computing tasks on the different computers, 842 which monitors the tasks as they are executed, which handles the 843 output and the errors from these tasks, and which reacts to the 844 failure of any of the computing nodes. The system responsible for the 845 tasks in the IPP is the IPP Controller. 846 847 The IPP Controller interacts with the collection of computers under 848 its management and with other subsystems in the IPP. The IPP 849 Controller receives a variety of inputs from other subsystems, 850 described below, and initiates actions such as adding a new process to 851 its queue. The IPP Controller also provides information to other 852 subsystems on demand about its processing history and current state. 853 Each physical computer may have multiple processors; since the IPP 854 Controller is managing processing tasks, it treats each processor 855 independently. It is up to the system configuration if each computer 856 needs to reserve one of its CPUs to manage background tasks or if the 857 IPP Controller should attempt to send one task per CPU and let the 858 kernel handle the I/O load. 859 860 Computers managed by the IPP Controller are allowed to be in one of 861 several states, and the IPP Controller must interact with it in an 862 appropriate way for each of those states. A computer may be {\tt 863 alive}, {\tt dead} or {\tt off}. If the computer is {\tt alive}, it 864 responds to commands from the IPP Controller and may be used for tasks 865 subject to other constraints. If it is {\tt dead}, the computer is 866 not responsive and must not be used for executing tasks. The IPP 867 Controller must identify computers which have died and occasionally 868 test them to see if they are {\tt alive} again. Computers which are 869 {\tt off} are not available for tests and must not be tested. 870 Computers may be set to the {\tt off} or {\tt dead} states by external 871 subsystems; it is the responsibility of the IPP Controller to return a 872 computer to the {\tt alive} state if possible. An example scenario: a 873 computer crashes. At this point the IPP Controller should detect that 874 the computer is no longer responsive and mark it {\tt dead}. It 875 should occasionally try to re-establish communication with the 876 computer, potentially with longer and longer delays between attempts. 877 A human could be notified if the computer seems to remain {\tt dead} 878 for a very long time. In another circumstance, a person needs to work 879 on a computer. They should have the ability to notify the IPP 880 Controller that the machine is off, perhaps with a prior notification 881 that the machine should be prepared to go off. Only when the person 882 is done working and testing the machine, and tells the IPP Controller 883 that the machine is now {\tt dead} can the IPP Controller attempt to 884 re-start communications and processing on that computer. 885 886 CPUs on computers which are in the {\tt alive} state may be in one of 887 two modes: {\tt busy} and {\tt free}. A CPU which is {\tt busy} 888 currently has a task assigned to it. The IPP Controller may only 889 assign one task to one CPU at a time. A CPU which is in the {\tt 890 free} state may have tasks assigned to it. The IPP Controller must 891 also respect a list of task restrictions which may require specific 892 tasks to run on specific CPUs or exclude specific tasks from specific 893 CPUs. 894 895 The IPP Controller accepts tasks from other IPP subsystems. The task 896 requests include the specific command to be executed and are in the 897 form of a UNIX command which could be performed on any of the 898 computing nodes. Any input or output data structures in the commands 899 must be a valid resource regardless of the node on which the task is 900 executed. Input and output data resources must be unique where 901 necessary to avoid conflicts. The IPP Controller gives each task a 902 unique identifier, which is returned to the requesting entity. The 903 requestor may then use that ID to obtain status information on that 904 task or to send control signals to the specific task. 905 906 Task requests may specify a desired node for the task execution. The 907 IPP Controller attempts to honor the request if the node is {\tt 908 alive}, but will execute it on another node if the requested one is 909 {\tt dead} or {\tt off}. Even if a node is {\tt alive}, the IPP 910 Controller will choose another node if the specified task is not 911 allowed on the requested node. In all other cases, the IPP Controller 912 waits until the currently executing processes, and processes with 913 higher priority, are completed before executing the specified task on 914 the requested node. 915 916 Task requests may specify an urgency level. The IPP Controller 917 determines the priority of the task on the basis of both the priority 918 and the age of the request. An executing task must be completed on a 919 CPU before any new task is started on that CPU, regardless of 920 priority. Tasks may be assigned a priority of 0 in which case they 921 are maintained in the queue and never executed. 922 923 The IPP Controller monitors the output streams from the executing 924 tasks and the exit status of the tasks. Each task is associated with 925 a log file, to which all output is written. The status, including the 926 exit status, of each task is maintained by the IPP Controller so that 927 other subsystems may determine if specific tasks have started or 928 completed. 929 930 The IPP Controller must accept commands from other IPP subsystems. 931 These commands include those which govern the processing of specified 932 tasks, those which govern the behavior of specific computing nodes, 933 and those which request information from the IPP Controller. The IPP 934 Controller must be able to halt the execution of a specified task, 935 delete an unexecuted task from the task list, change the priority of 936 tasks, and change the requested nodes for tasks. The IPP Controller 937 must also be able to stop the current execution of a task and push it 938 to the end of the queue and also change its priority. 939 940 The IPP Controller must honor requests (normally from the users) to 941 change the mode of any computing node on demand between {\tt off} and 942 {\tt dead}. This would normally be done after a computer has been 943 rebooted and is release to the IPP Controller for its use. It must 944 also be able to change the list of allowed tasks as requested by 945 external commands. 946 947 The IPP Controller must respond to informational requests regarding the 948 collection of machines and their states as well as the collection of 949 tasks and their states. The IPP Controller must monitor the execution 950 times of the different tasks and provide summary statistics. Finally, 951 the IPP Controller must respond to three top-level commands: {\tt finish}, 952 {\tt stop} and {\tt abort}. When {\tt finish} is requested, no more 953 new tasks are accepted on the stack of task, and when all tasks in the 954 stack have completed, the IPP Controller must exit. When {\tt stop} is 955 requested, the currently executing tasks must be completed at which 956 point the IPP Controller must exit, but tasks remaining in the stack which 957 have not been started are flushed. When {\tt abort} is issued, the 958 IPP Controller immediately kills all executing tasks and exits. 959 960 The IPP Controller and the IPP Image Server have related needs for 961 information from the combined storage-and-processing nodes regarding 962 which nodes are available. It is not yet clear if this information is 963 best stored in a single location (either IPP Controller or IPP Image 964 Server), which provides the information to other systems on demand, or 965 if both systems should maintain the information. Also, it may be 966 necessary to distinguish nodes which are available for processing from 967 those that are available to serve data as part of the IPP Image 968 Server. 969 970 It may be useful for the Controller to distinguish between tasks 971 dominated by I/O and tasks dominated by data processing. It is 972 possible that one of each of these types of tasks may be sent to the 973 same node without significantly impacting the system performance. 974 Alternatively, it may be necessary to limit a single machine with 2 975 CPUs to only one of each of these types of tasks (i.e., one processor 976 will be working on I/O while the other is working on processing). 977 Such details will be studied by the IfA IPP Team. 960 978 961 979 The Controller maintains a table of processing nodes available to it … … 973 991 clients and sends them new pending stages when they become free. 974 992 975 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%976 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%977 978 993 \subsubsection{Node Agents} 979 994 980 A Node Agent runs on each of the individual nodes to perform the 981 processing stages as directed by the Controller. The Node Agents 982 communicate with the Controller via a socket connection. 983 984 A processing stage is executed in the UNIX user space, and is run as a fork by the 985 Node Agent. The Node Agent must monitor the standard error and 986 standard output of the processing stage and save them in separate buffers. If the 987 process dies, the Node Agent must detect the crash. The Node Agent 988 must respond to various commands from the Controller. 989 990 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 995 A Node Agent runs on each of the individual nodes to perform the tasks 996 as directed by the Controller. The Node Agents communicate with the 997 Controller via a socket connection. 998 999 A processing stage is executed in the UNIX user space, and is run as a 1000 fork by the Node Agent. The Node Agent must monitor the standard 1001 error and standard output of the processing stage and save them in 1002 separate buffers. If the process dies, the Node Agent must detect the 1003 crash. The Node Agent must respond to various commands from the 1004 Controller, as follows: 991 1005 992 1006 \paragraph{Report status} … … 1012 1026 indication that there is no current processing stage (`none'). 1013 1027 1014 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%1015 1016 1028 \paragraph{Report stdout} 1017 1029 … … 1023 1035 accept all of the buffer output. 1024 1036 1025 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%1026 1027 1037 \paragraph{Report stderr} 1028 1038 1029 1039 Identical to `report stdout', but for stderr. 1030 1031 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%1032 1040 1033 1041 \paragraph{Kill processing stage} … … 1038 1046 `done'. 1039 1047 1040 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%1041 1042 1048 \paragraph{Clear processing stage} 1043 1049 … … 1045 1051 and the Node state to `idle'. If a processing stage is currently 1046 1052 running, it should be killed before the processing stage is cleared. 1047 1048 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%1049 1053 1050 1054 \paragraph{Start processing stage} … … 1055 1059 of security, for example, by employing SSL authentication. 1056 1060 1061 \subsubsection{Controller User Interface} 1062 1063 The IPP Controller provides a mechanism for users (either other 1064 programs or humans) to interact with it. The user interface provides 1065 commands to check the current processing job queues, the tables of 1066 successful and failed jobs, to stop or delete jobs, etc. 1067 1068 \subsubsection{Notes} 1069 1070 can a process send a message back to the controller before process is 1071 complete? messages via controller? 1072 1073 does the controller or the image server decide if a machine is offline 1074 or both? 1075 1076 I/O tasks vs CPU tasks? 1077 1057 1078 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1058 1079 1059 \paragraph{Matrix} 1060 1061 \tbd{The Node Agent does not wear a suit, nor does it know kung fu.} 1080 \subsection{Scheduler} 1081 1082 The IPP is responsible for a variety of analysis tasks: processing of 1083 the science images through several stages; routine assessment of the 1084 detrend (instrumental calibration) images used in processing the 1085 science images; construction of replacement detrend images when 1086 needed; generation of astrometric and photometric reference catalogs 1087 based on the collected dataset; and the performance of test analysis 1088 programs. At any point, decisions need to be made about which of 1089 these tasks should be performed, based on an analysis of the contents 1090 of the metadata database, the requirements of the people monitoring 1091 the IPP, and the near-term observing plans. The IPP Scheduler is the 1092 mechanism that assesses these various inputs to guide the decisions 1093 and initiate the actions. 1094 1095 The IPP Scheduler acts as an intermediary between several components 1096 of the IPP and also between the IPP and external agents such as OTIS 1097 and the users who must monitor the behavior of the IPP. 1098 1099 The IPP Scheduler sends commands to the IPP Controller for execution. 1100 While the IPP Scheduler chooses the tasks to be performed, it is the 1101 IPP Controller's responsibility to manage the specific tasks executing 1102 on a given processing node. Examples of these tasks include the 1103 process of copying or moving data from the Summit data systems to the 1104 IPP Image Server; image processing analysis stages performed on the 1105 science images by the appropriate processing nodes; and the analysis 1106 of the data in the AP Database. This division of responsibilites 1107 allows us to isolate and encapsulate the functionality of the IPP 1108 Scheduler and the IPP Controller. With this separation, the IPP 1109 Controller does not need to have any information about the details of 1110 the tasks which it executes, while the IPP Scheduler does not need to 1111 have detailed information about the available computer hardware. 1112 1113 Communication between the IPP Scheduler and the IPP Controller is 1114 bi-directional; the IPP Scheduler sends tasks to the IPP Controller, 1115 while the IPP Controller informs the IPP Scheduler of the outcome of 1116 those tasks. It is not specified whether the IPP Scheduler and IPP 1117 Controller are components of a single software system or interacting 1118 but distinct software components. 1119 1120 The IPP Scheduler takes as input the current list of pending images, 1121 both science and calibration, and a description of the current 1122 observing plan or strategy on some time-scale. The IPP Scheduler also 1123 takes input from humans who manage the IPP. 1124 1125 The IPP Scheduler must choose between several types of analysis tasks 1126 based on the contents of those lists and on the requirements of the 1127 users. The list of tasks which the IPP Scheduler must decide between 1128 includes: 1129 \begin{itemize} 1130 \item moving data from the Summit pixel server ($\sim 30$ second timescales) 1131 \item running the science analysis stages ($\sim 30$ second timescales) 1132 \item testing the validity of the current detrend images ($\sim$ 1133 nightly) 1134 \item constructing new detrend images ($\sim$ weekly) 1135 \item updating and improving the photometric and astrometric reference 1136 catalogs ($\sim$ yearly). 1137 \end{itemize} 1138 1139 The IPP Scheduler chooses between tasks which are relevant on several 1140 different time-scales. The time-scales range from 2 times per minute 1141 to once or twice a year, as noted in the list above. The IPP 1142 Scheduler must also make use of user input in managing such choices. 1143 Users have the option to specify that a particular task or set of 1144 tasks is of higher or lower priority than the norm. 1145 1146 The scheduler may be viewed as a complex state machine. Our goal is 1147 to design the rules independently from the engine which parses the 1148 rules to detemine which specific jobs to send to the controller. 1149 1150 \subsubsection{Scheduler User Interface} 1151 1152 The IPP Scheduler provides a user interface which allows a human 1153 operator, or other processes, to monitor the current state of the 1154 Scheduler. 1155 1156 The IPP Scheduler defines the operating state of the IPP. When the 1157 IPP is in the {\em automatic state}, the IPP Scheduler performs the 1158 most appropriate of all possible tasks at a particular time. When the 1159 IPP is in the {\em interactive state}, the IPP Scheduler performs only 1160 the requested action regardless of the outcome of the decision trees. 1161 In addition, in the interactive state, the IPP Scheduler must only 1162 perform the requested actions and not attempt to perform the other 1163 normally-required actions. The only exception to this exclusion is 1164 that, in the interactive state, data is still copied from the summit 1165 system. An additional IPP state is the {\em paused state}, intended 1166 for tests or maintenance, in which case the IPP Scheduler does not 1167 perform even the data copy tasks. Every task is performed on demand 1168 by the user. The user command sets the IPP Scheduler in one of these 1169 three states, {\em automatic}, {\em interactive}, and {\em paused}. 1062 1170 1063 1171 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1172 1173 \section{System Design : Science Analysis Tasks and Stages} 1174 1175 In this section, we discuss the design of the science analysis stages 1176 which perform the fundamental image analysis steps of the IPP. The 1177 IPP science image processing stages perform analyses on the night-sky 1178 science images to extract the science data from these images. These 1179 consist of: Phase 1, the image processing preparation stage; Phase 2, 1180 the image reduction stage; Phase 3, the exposure analysis stage; and 1181 Phase 4, the image combination stage. These analysis tasks must 1182 process the images in a timely manner so that the incoming data stream 1183 will not overload the IPP Image Server. The decision to execute a 1184 specific pipeline for a specific dataset is made by the Scheduler, 1185 which sends the infomation to the Controller. The Controller executes 1186 the pipeline for the data on an appropriate machine and monitors the 1187 success or failure of the processing stage. 1188 1189 The analysis stages are written as UNIX commands, which may be 1190 executed by the IPP Controller, or may be executed individually by 1191 hand. This aspect makes testing of the complete analysis system much 1192 easier because the individual analysis stages may be tested 1193 independently of each other and the IPP infrastructure. 1194 1195 In keeping with this design model, the analysis stages have several 1196 methods for accepting and returning the input and output data. All of 1197 the analysis stages load an analysis recipe file, which defines the 1198 details of the analysis. This includes the location of the data 1199 sources (from the metadata, from the image headers, from other 1200 external files, or supplied directly), and which steps to employ. For 1201 example, in the discussion of the Phase 2 analysis below, the recipe 1202 file may specify {\em if} a bias subtraction should be applied, {\em 1203 where} to find the overscan region and {\em which} bias image, if any, 1204 to apply. 1205 1206 \tbd{further discussion of the recipe / configuration files?} 1207 1208 1064 1209 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1065 1210 1066 \subsubsection{Scheduler} 1067 1068 The IPP Scheduler is responsible for initiating the various processing 1069 stages (which are executed by the IPP Controller), based on the state 1070 of the survey as reflected by the IPP Metadata Database (IMD). 1071 1072 The Scheduler shall maintain a list of processing stages, as well as 1073 the required input and dependencies for each of the processing stagesFor example, the 1074 dependencies for copying pixel data from OATS may be: 1075 \begin{itemize} 1076 \item OATS has new pixel data available; 1077 \item The new pixel data has not been copied. 1078 \end{itemize} 1079 Similarly, the dependencies for executing Phase 2 processing on a chip 1080 may be: 1081 \begin{itemize} 1082 \item The chip pixel data has been copied. 1083 \item Phase 1 has run successfully on the metadata for the FPA to which 1084 the chip belongs. 1085 \item A reduced image (i.e., output from Phase 2) does not already 1086 exist. 1087 \end{itemize} 1088 1089 When the dependencies are satisfied, the Scheduler shall prepare for 1090 execution the particular processing stage on the appropriate data. 1091 The Scheduler must query the Metdata DB for the most appropriate 1092 calibration data, if required. The processing stage should be 1093 filtered through the IPSDLO in order to assign the processing stage to 1094 a particular Node (if desired) and to determine the URIs for the 1095 required inputs. The processing stage is then passed to the 1096 Controller. 1097 1098 The Scheduler must also be able to send requests for new calibration 1099 data to OATS, including required flat-fields, flat-field correction 1100 observations, or other specialized observations needed to improve the 1101 calibrations. The Scheduler must balance the need for improved 1102 calibrations with the need to process the science images in a timely 1103 manner given the capabilities of the science pipelines. 1104 1105 \paragraph{Pollster} 1106 1107 The Pollster is a program that polls OATS at regular intervals for the 1108 existence of observations not contained in the Metadata DB. New 1109 weather and image metadata are written to the Metadata DB. 1110 1111 There is no reason why this architectural component cannot be 1112 contained within another (such as the Scheduler), but it is shown here 1113 as separate for simplicity. 1114 1115 A polling model is adopted so that OATS' interface may be kept as 1116 simple as possible --- OATS should not be concerned with whether the 1117 IPP has received notifications. Under this polling model, it is 1118 specifically the responsibility of the IPP to retrieve from OATS the 1119 metadata that is not not already in the Metadata DB. 1120 1121 \subsubsection{Pollster} 1122 1123 The Pollster simply polls OATS on a regular basis for metadata 1124 (including telescope exposures) which is not known by the IPP (i.e., 1125 already written in the Metadata DB). On the discovery of such metadata, 1126 it is written to the Metadata DB. 1211 \subsection{Phase 1: image processing preparation} 1212 1213 The Phase 1 analysis stage is performed on each science exposure (each 1214 complete FPA image) to calculate basic astrometric data needed by the 1215 later stages. Phase 1 uses the static (pre-determined) telescope 1216 distortion model and a table of nominal OTA positions and rotations, 1217 combined with the guide star pixel and celestial coordinates, to 1218 determine the correct telescope bore-sight, field rotation and 1219 magnification. The guide star coordinates are loaded from the 1220 Metadata database. These calculations are performed by comparing the 1221 observed guide star detector coodinates with the known astrometic 1222 positions of these same stars as reported by an external astrometric 1223 reference. The accuracy of the resulting astrometric solution is 1224 expected to be $\sim 1$ arcsec across the field, sufficient in later 1225 stages to match the vast majority of astrometric reference stars with 1226 their detections with minimal effort. 1227 1228 In some circumstances, science images may have no guide stars. This 1229 may occur in the Pan-STARRS system if the detectors are not run in OTA 1230 mode, for example for short snapshot images. This may also be the 1231 case if the IPP is being run on non-Pan-STARRS data. In such a 1232 circumstance, the Phase 1 stage uses the provided boresight 1233 coordinates, exposure time, and camera zero-point to predict the pixel 1234 coordinates of known bright stars expected to be found on the 1235 detectors. It then extracts a large box ($\sim$ 30 $\times$ 1236 30\arcsec) around these locations and performs extremely basic object 1237 detection to determine the detector coordinates of those bright stars 1238 which are not saturated but which are significantly above the 1239 background level. By targetting known locations in the image files, 1240 only a small amount of data will have to be read. 1241 1242 If the image has invalid coordinates or no detectable bright stars, 1243 Phase 1 fails and reports a descriptive error. 1244 1245 Given the above astrometric solution, the Phase 1 analysis stage 1246 constructs a table of the overlaps between the science image to be 1247 processed and the static sky images that must be constructed. This 1248 table will be used to guide the processing of the static sky in Phase 1249 4. The overlaps should be generously calculated so that small errors 1250 in astrometry at Phase 1 will not cause any valid static sky / science 1251 image pairs to be missed because of the astrometric error at this 1252 phase. It is acceptable for a small number of invalid overlaps to be 1253 identified as these will be excluded in Phase 4. Static Sky cells 1254 which do not have sufficient science image overlap \tbr{$< 5\%$} need 1255 not be processed because the few new measured pixels do not add 1256 significantly to the Static Sky. 1257 1258 \subsubsection{Notes} 1259 1260 \begin{verbatim} 1261 possible command forms: 1262 1263 P1 filename.fits [FPA is single fits file] 1264 P1 filename.list [FPA is collection of files] 1265 P1 FPA IA [FPA info from metadata db] 1266 1267 sources for the input data: 1268 1269 distortion model: 1270 metadata table 1271 XML file 1272 FITS table 1273 metadata -> image server 1274 user provided on command line 1275 recipe provided 1276 1277 camera layout: 1278 metadata table 1279 XML file 1280 FITS table 1281 metadata -> image server 1282 user provided on command line 1283 recipe provided 1284 1285 boresite coordinates guess: 1286 image header (keywords from recipe) 1287 metadata table 1288 1289 guide stars 1290 collection of video streams 1291 collection of centroid time histories 1292 list of centroids, coordinates 1293 \end{verbatim} 1127 1294 1128 1295 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1129 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1130 1131 \subsubsection{System UI} 1132 1133 A user interface allows a human operator to monitor the Controller and 1134 Scheduler through some user interface (UI). The System UI may 1135 interact with the Controller and Scheduler via a socket connection 1136 using a defined set of commands. 1137 1138 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1139 1140 \paragraph{Execute processing stage} 1141 1142 A new processing stages is sent to the Scheduler. The Scheduler may 1143 filter the processing stages through the IPSDLO, or it may be 1144 prevented from doing so by the user. The Scheduler then passes the 1145 processing stages to the Controller for execution. 1146 1147 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1148 1149 \paragraph{Kill processing stage} 1150 1151 The user may kill an existing processing stage. The Controller is 1152 commanded to kill the particular processing stage. 1153 1154 \tbd{Should we allow a System UI to kill processing stages sent by 1155 other System UIs?} 1156 1157 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1158 1159 \paragraph{Get status} 1160 1161 The System UI may request the current status of the Controller, 1162 including the list of pending, active, and completed processing stages 1163 and the status of the individual processing stages. 1164 1165 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1166 1167 \paragraph{Available Nodes} 1168 1169 The System UI may view and configure the list of Nodes available to 1170 the Controller (e.g., to remove a Node temporarily for maintenance). 1171 1172 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1173 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1174 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1175 1176 \subsection{Analysis Tasks and Stages} 1177 1178 In this section, we review the processing stages which are executed on 1179 the Nodes. 1180 1181 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1182 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1296 1297 \subsection{Phase 2 : image reduction} 1183 1298 1184 1299 \subsubsection{Overview} 1185 1300 1186 The processing stages are the software that process data. These 1187 processing stages are divided into five categories which are 1188 summarised in \S\ref{sec:processingStages}. Each of the processing 1189 stages are described below. 1190 1191 The processing stages are initiated by the Scheduler, parallized and 1192 managed by the Controller, and executed through the Node Agents on the 1193 nodes. Processing stages are purely serial, and so they may be run on 1194 a single node at once without the need for interprocess communication. 1195 1196 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1197 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1198 1199 \subsubsection{Retrieval} 1200 1201 The retrieval stages simply retrieve pixel data from an external 1202 source (ordinarily OATS at the Summit, but it could conceivably be 1203 some other external source) and store it on the nodes. 1204 1205 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1206 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1207 1208 \subsubsection{Science Image Processing} 1209 1210 The IPP science image processing stages perform analyses on the 1211 night-sky science images to extract the science data from these 1212 images. These consist of: Phase 1, the image processing preparation 1213 stage; Phase 2, the image reduction stage; Phase 3, the exposure 1214 analysis stage; and Phase 4, the image combination stage. These 1215 pipelines must process the images in a timely manner so that the 1216 incoming data stream will not overload the IPS. The decision to 1217 execute a specific pipeline for a specific dataset is made by the 1218 Scheduler, which sends the infomation to the Controller. The 1219 Controller executes the pipeline for the data on an appropriate 1220 machine and monitors the success or failure of the processing stage. 1221 1222 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1223 1224 \paragraph{Phase 1: image processing preparation} 1225 1226 The Phase 1 system operates on data from each FPA to calculate basic 1227 astrometric information needed by other stages of the analysis. The 1228 analysis includes: 1229 1230 \begin{itemize} 1231 \item preliminary astrometry based on the guide-star centroids 1232 \item sky-cell / detector-cell overlaps 1233 \end{itemize} 1234 1235 The input to this analysis is the list of guide-star pixel centroids 1236 and their celestial coordinates as saved in the metadata database, as 1237 well as the FPA and chip organization and geometry, and the basic 1238 optical distortion for the camera. For the sky-cell / detector-cell 1239 overlaps, the sky tiling scheme is required. 1240 1241 The output consists of calculated astrometric parameters (linear 1242 transformation + static distortion) for each of the FPA chips. On the 1243 basis of this astrometry, the overlap between the detectors and the 1244 sky-cells is calculated. The output of this calculation is a list of 1245 sky-cell / chip links in a database table. This list of links can be 1246 used by the later stages to initiate the analyses. 1247 1248 The phase 1 analysis is performed on an FPA basis to ensure that 1249 enough reference stars are available for the astrometry calculation. 1250 Phase 1 cannot be usefully calculated on the basis of a major frame 1251 since the telescope positions are independent; no additional 1252 information is available by combining stars from different FPAs. This 1253 analysis does not restrict the definition of a major frame in any way. 1254 1255 \tbd{Phase 1 command: P1 (exposure)} 1256 1257 \tbd{Megacam: P1 654321o} 1258 1259 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1260 1261 \paragraph{Phase 2 : image reduction : new version} 1262 1263 \tbd{how long are processed images kept?} 1264 1265 \tbd{what subsystem deletes processed images?} 1266 1267 \tbd{does 'remove' mean 'mask' or 'replace'} 1268 1269 \tbd{what is the absolute astrometry accuracy at phase 2? 0.1 arcsec 1270 == 0.33 pix?} 1271 1272 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1273 1274 \subparagraph{Concept} 1275 1276 Phase~2 processing within the \PS{} image processing pipeline is 1277 the de-trend stage, where the images from the detector are processed 1278 to remove instrumental signatures. 1279 1280 \begin{figure} 1281 \begin{center} 1282 \resizebox{8cm}{!}{\includegraphics{pics/phase2}} 1283 \caption{ \label{phase2} Phase 2 dataflow} 1284 \end{center} 1285 \end{figure} 1286 1287 Prior to Phase~2, the Phase~1 process operates on an entire telescope 1288 Focal Plane Array to set the boresight astrometric solution using 1289 the guide stars and initial masking of ghost reflections. 1290 1291 Phase~2 consists of the following modules: 1292 \begin{enumerate} 1293 \item Form OT kernel; 1294 \item Convolve de-trend images with the OT kernel; 1301 Phase 2 processing within the Pan-STARRS image processing pipeline is 1302 the detrend stage, where the images from the detector are processed to 1303 remove instrumental signatures. This analysis is performed on 1304 individual chips, which can be identified as the data entity which has 1305 a single, continuous astrometric solution. 1306 1307 Phase 2 consists of the following operations, some of which as noted 1308 may be skipped by the recipe: 1309 \begin{itemize} 1310 \item Load science image 1311 \item Identify appropriate detrend images 1312 \item Load detrend images 1313 \item Form OT kernel 1314 \item Convolve detrend images with the OT kernel 1295 1315 \item Mask bad pixels 1296 \item Mask diffraction spikes and optical ghosts; 1297 \item Bias/dark/overscan subtraction; 1298 \item Trim overscan; 1299 \item Non-linearity correction; 1300 \item Flat-field; 1301 \item Subtract sky; 1302 \item Identify CRs by morphology; 1303 \item Determine PSF model; 1304 \item Find and photometer objects in the image; 1305 \item Improved astrometry; and 1306 \item Bright object postage stamps. 1307 \end{enumerate} 1308 These modules are each explained below. 1309 1310 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1311 1312 \subparagraph{Form OT Kernel} 1313 1314 The first module for Phase~2 is to form the OT kernel from the image 1315 metadata of pixel shifts made during the exposure. This involves 1316 decoding the metadata and converting it to a data type that can be 1317 used to convolve by. The output is the OT convolution kernel. 1318 1319 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1320 1321 \subparagraph{Convolve de-trend images} 1322 1323 \tbd{Must this be a formal convolution with the analytical OT kernel, 1324 or can it be a convolution with a decomposed kernel?} 1325 1326 \tbd{what is the source of the OT kernel? pixel server?} 1327 1328 This module convolves the de-trend images with the OT convolution kernel 1329 so that they can be used to de-trend the object image. The inputs 1316 \item Mask diffraction spikes and optical ghosts 1317 \item Bias/dark/overscan subtraction 1318 \item Trim overscan 1319 \item Non-linearity correction 1320 \item Flat-field 1321 \item Subtract sky 1322 \item Identify CRs by morphology 1323 \item Determine PSF model 1324 \item Find and photometer objects in the image 1325 \item Improved astrometry 1326 \item Extract Bright object postage stamps 1327 \end{itemize} 1328 1329 Several of the steps are explained in detail below. 1330 1331 \subsubsection{Form OT Kernel} 1332 1333 Certain detrend images are convolved by the OT kernel, so that they 1334 accurately represent the detrend images appropriate for the object 1335 images, which have been shifted using OT. The detrend images which 1336 must be convolved include: the flat-field and the 1337 high-spatial-frequency fringe images. The appropriate kernel for each 1338 cell of an OTA must be determined from the guide star history, 1339 extracted from the IPP Metadata Database\footnote{or image header}. 1340 If the OT kernel is not available, but the image metadata notes that 1341 it should be, the convolution is skipped, with a warning. 1342 1343 The first module for Phase 2 forms the OT kernel from the list of 1344 pixel shifts made during the exposure. This involves decoding the 1345 metadata and converting it to a data type that can be used to convolve 1346 by. The output is the OT convolution kernel. 1347 1348 \subsubsection{Convolve detrend images} 1349 1350 This module convolves the detrend images with the OT convolution kernel 1351 so that they can be used to detrend the object image. The inputs 1330 1352 are: 1331 \begin{ enumerate}1353 \begin{itemize} 1332 1354 \item The OT convolution kernel --- from the previous module; 1333 1355 \item The appropriate dark frame --- from the IPP Pixel Server; … … 1335 1357 \item The appropriate fringe frame(s) --- from the IPP Pixel Server; and 1336 1358 \item The appropriate static bad pixel mask --- from the IPP Pixel Server. 1337 \end{ enumerate}1359 \end{itemize} 1338 1360 1339 1361 The module convolves each of the dark frame, flat-field, and the fringe … … 1341 1363 bad pixel mask are grown by the outline of the OT convolution kernel 1342 1364 (see Section \ref{ap:masks}). The output results are: 1343 \begin{ enumerate}1365 \begin{itemize} 1344 1366 \item The convolved flat-field; 1345 1367 \item The convolved fringe frame(s); and 1346 1368 \item The updated pixel mask. 1347 \end{enumerate} 1348 Each of these will be used for a later module. 1349 1350 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1351 1352 \subparagraph{Overscan Subtraction} 1369 \end{itemize} 1370 Each of these will be used for a later module. The convolution method 1371 depends on the size and structure of the OT kernel. If the kernel is 1372 small ($< 5x5$ pixels), direct convolution may be employed. If the 1373 kernel is large, but may be decomposed using Gaussians, then it may be 1374 convolved using a decomposition method. 1375 1376 \subsubsection{Bias Correction / Overscan Subtraction} 1353 1377 1354 1378 This module corrects the object exposures for the electronic pedestal 1355 1379 introduced by the readout electronics. The inputs are: 1356 \begin{ enumerate}1380 \begin{itemize} 1357 1381 \item The object image --- from the IPP Pixel Server; 1358 1382 \item The pixel mask --- from the previous module; … … 1361 1385 \item Detector characteristics (gain, read noise) --- from the 1362 1386 Metadata. 1363 \end{ enumerate}1387 \end{itemize} 1364 1388 1365 1389 The overscan is averaged (either in bulk, or individually by rows) or … … 1370 1394 regions grown by an additional pixel to counter CCD ``blooming''. The 1371 1395 output is: 1372 \begin{ enumerate}1396 \begin{itemize} 1373 1397 \item The overscan-subtracted object image; and 1374 1398 \item The updated pixel mask. 1375 \end{ enumerate}1399 \end{itemize} 1376 1400 These will be used for a subsequent module. 1377 1401 1378 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1379 1380 \subparagraph{Trim} 1402 \subsubsection{Trim} 1381 1403 1382 1404 This module trims the object image and each of the calibration frames to 1383 1405 remove the outer edge which was affected by the OT during the 1384 1406 exposure. The inputs, each from previous modules, are: 1385 \begin{ enumerate}1407 \begin{itemize} 1386 1408 \item The overscan-subtracted object image; 1387 1409 \item The corresponding pixel mask; … … 1389 1411 \item The convolved fringe frame(s); and 1390 1412 \item The dimension of the OT convolution kernel in each direction. 1391 \end{ enumerate}1413 \end{itemize} 1392 1414 1393 1415 Each of the input frames (object image, flat-field, fringe frame(s) … … 1397 1419 modules. 1398 1420 1399 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1400 1401 \subparagraph{Non-Linearity Correction} 1421 \subsubsection{Non-Linearity Correction} 1402 1422 1403 1423 This module corrects images for non-linearity in the detector. The 1404 1424 inputs are: 1405 \begin{ enumerate}1425 \begin{itemize} 1406 1426 \item The trimmed object image --- from a previous module; and 1407 1427 \item The detector non-linearity correction coefficient(s) --- from 1408 1428 the Metadata. 1409 \end{ enumerate}1429 \end{itemize} 1410 1430 1411 1431 The module corrects the flux in each pixel for non-linearity by applying … … 1413 1433 is the corrected object image, which is used for a later module. 1414 1434 1415 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1416 1417 \subparagraph{Flat field} 1435 \subsubsection{Flat field} 1418 1436 1419 1437 This module corrects the object image for variations in sensitivity over 1420 1438 the image. The inputs are: 1421 \begin{ enumerate}1439 \begin{itemize} 1422 1440 \item The object image corrected for non-linearity; 1423 1441 \item The corresponding pixel mask; and 1424 1442 \item The convolved, trimmed flat-field. 1425 \end{ enumerate}1443 \end{itemize} 1426 1444 Each of these comes from a previous module. 1427 1445 1428 1446 The module divides the object image by the flat-field, masking pixels 1429 1447 that are non-positive in the flat-field. The outputs are: 1430 \begin{ enumerate}1448 \begin{itemize} 1431 1449 \item The flattened object image; and 1432 1450 \item The updated pixel mask. 1433 \end{ enumerate}1451 \end{itemize} 1434 1452 Both of these will be used in later modules. 1435 1453 1436 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1437 1438 \subparagraph{Subtract sky} 1454 \subsubsection{Subtract sky} 1439 1455 1440 1456 This module subtracts the sky background from the object image. The 1441 1457 inputs are: 1442 \begin{ enumerate}1458 \begin{itemize} 1443 1459 \item The object image --- from the previous module; 1444 1460 \item The list of objects on the image --- from the object database; and 1445 1461 \item The convolved, trimmed fringe frame(s) --- from a previous module. 1446 \end{ enumerate}1462 \end{itemize} 1447 1463 1448 1464 The module masks (though {\em not} in the ``official'' pixel mask) all … … 1456 1472 which is used for the next module. 1457 1473 1458 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1459 1460 \subparagraph{Identify CRs by morphology} 1474 \subsubsection{Identify CRs by morphology} 1461 1475 1462 1476 This module identifies cosmic rays (or other hot pixels missed in the 1463 1477 static bad pixel mask) on the basis of their morphology. The inputs 1464 1478 are: 1465 \begin{ enumerate}1479 \begin{itemize} 1466 1480 \item The object image; and 1467 1481 \item The corresponding pixel mask. 1468 \end{ enumerate}1482 \end{itemize} 1469 1483 Both of these come from a previous module. 1470 1484 … … 1473 1487 in each direction. Masked pixels are interpolated over. The outputs 1474 1488 are the updated pixel mask, which is sent to the IPP pixel server for 1475 use in Phase ~3, and is also used for the next module; and the object image,1489 use in Phase 3, and is also used for the next module; and the object image, 1476 1490 which is sent to the IPP Pixel Server. 1477 1491 1478 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1479 1480 \subparagraph{Find objects} 1492 \subsubsection{Detect and Measure objects} 1481 1493 1482 1494 This module finds objects on the object image. The inputs are: 1483 \begin{ enumerate}1495 \begin{itemize} 1484 1496 \item The sky-subtracted object image; and 1485 1497 \item The corresponding pixel mask. 1486 \end{ enumerate}1498 \end{itemize} 1487 1499 Both of these come from a previous module. 1488 1500 … … 1493 1505 object image. 1494 1506 1495 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1496 1497 \subparagraph{Bright object postage stamps} 1507 Object catalogs from Phase 2 shall consist of at least the 1508 following elements for each object: 1509 \begin{itemize} 1510 \item Object centre, with corresponding errors; 1511 \item Object magnitude, with corresponding error; 1512 \item Object isophotal magnitude, with corresponding error; 1513 \item Object FWHM; 1514 \item Object elliptical axis lengths; and 1515 \item Object position angle for ellipse. 1516 \end{itemize} 1517 1518 Though further details may be required for catalogs in Phase 4, 1519 the above details are minimum requirements for Phase 2 catalogs. 1520 1521 \subsubsection{Bright object postage stamps} 1498 1522 1499 1523 This module saves postage stamps of bright objects, so that extra care 1500 1524 with regard to astrometry and photometry can be taken with them at a 1501 1525 later stage. The inputs, each from a previous module, are: 1502 \begin{ enumerate}1526 \begin{itemize} 1503 1527 \item The sky-subtracted object image; 1504 1528 \item The corresponding pixel mask; and 1505 1529 \item The catalog of objects. 1506 \end{ enumerate}1530 \end{itemize} 1507 1531 1508 1532 The module makes postage stamps of all objects brighter than a given … … 1511 1535 the IPP Pixel Server. 1512 1536 1513 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1514 1515 \subparagraph{Metadata Required} 1537 \subsubsection{Pixel Masks} 1538 \label{ap:masks} 1539 1540 This section describes the requirements on Bad Pixel Masks (BPMs). 1541 These will consist of bit masks for each pixel. For Phase 2, flags 1542 are required for at least each of the following pixel attributes: 1543 \begin{itemize} 1544 \item The pixel is a charge trap; 1545 \item The pixel is a bad column; 1546 \item The pixel is saturated in the A/D converter; 1547 \item The pixel is non-positive in the flat-field; 1548 \item The pixel is part of a row that has excess noise; and 1549 \item The pixel is determined to be a cosmic ray, based on its 1550 morphology. 1551 \end{itemize} 1552 1553 Of these, only masks for the charge traps need to be grown by the 1554 extent of the OT convolution kernel. For other pixel types, 1555 orthogonal transfer of the flux in this pixel will not (necessarily) 1556 affect the flux in neighbouring pixels 1557 1558 \subsubsection{Phase 2 Metadata} 1516 1559 1517 1560 The following metadata associated with the images are required for 1518 Phase ~2 operation:1561 Phase 2 operation: 1519 1562 \begin{itemize} 1520 1563 \item The orthogonal transfer (OT) image shifts made during the … … 1526 1569 detrend images; 1527 1570 \item Exposure time --- for the photometric calibration; 1528 \item Detector gain --- for calculating photometric errors; and 1571 \item Detector gain --- for calculating photometric errors and 1572 determining the quality of the overscan; 1529 1573 \item Detector read noise --- for calculating photometric errors and 1530 1574 determining the quality of the overscan; 1531 1575 \end{itemize} 1532 1576 1533 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1534 1535 \subparagraph{Pixel Masks} 1536 \label{ap:masks} 1537 1538 This section describes the requirements on Bad Pixel Masks (BPMs). 1539 These will consist of bit masks for each pixel. For Phase 2, flags 1540 are required for at least each of the following pixel attributes: 1541 \begin{enumerate} 1542 \item The pixel is a charge trap; 1543 \item The pixel is a bad column; 1544 \item The pixel is saturated in the A/D converter; 1545 \item The pixel is non-positive in the flat-field; 1546 \item The pixel is part of a row that has excess noise; and 1547 \item The pixel is determined to be a cosmic ray, based on its 1548 morphology. 1549 \end{enumerate} 1550 1551 Of these, only masks for the charge traps need to be grown by the 1552 extent of the OT convolution kernel. For other pixel types, 1553 orthogonal transfer of the flux in this pixel will not (necessarily) 1554 affect the flux in neighbouring pixels 1555 1556 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1557 1558 \subparagraph{Object Catalogs} 1559 \label{ap:catalogs} 1560 1561 Object catalogs from Phase 2 shall consist of at least the 1562 following elements for each object: 1563 \begin{enumerate} 1564 \item Object centre, with corresponding errors; 1565 \item Object magnitude, with corresponding error; 1566 \item Object isophotal magnitude, with corresponding error; 1567 \item Object FWHM; 1568 \item Object elliptical axis lengths; and 1569 \item Object position angle for ellipse. 1570 \end{enumerate} 1571 1572 Though further details may be required for catalogs in Phase~4, 1573 the above details are minimum requirements for Phase~2 catalogs. 1574 1575 \tbd{Phase 2 command: P2 (exposure.ota.fits)} 1576 \tbd{Megacam: P2 654321o.fits[ccd00] - what are output names?} 1577 \tbd{PS FPA is saved as a collection of MEF files. Megacam FPA is 1578 saved as a single MEF file. how to handle this difference?} 1577 \subsubsection{Notes} 1578 1579 \tbd{how long are processed images kept?} 1580 1581 \tbd{what subsystem deletes processed images?} 1582 1583 \begin{figure} 1584 \begin{center} 1585 \resizebox{8cm}{!}{\includegraphics{pics/phase2}} 1586 \caption{ \label{phase2} Phase 2 dataflow} 1587 \end{center} 1588 \end{figure} 1579 1589 1580 1590 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1581 1591 1582 \ paragraph{Phase 3 : exposure analysis}1592 \subsection{Phase 3 : exposure analysis} 1583 1593 1584 1594 The Phase 3 system operates on the combined Phase 2 results from an 1585 1595 FPA to determine improved solutions for the image calibrations and to 1586 1596 provide the parameters needed by Phase 4. The Phase 3 output is saved 1587 by the IMD, and consists largely of improved values of the1588 calibrations already determined by Phase 2. The analysis performed by 1589 this pipeline consists of:1597 by the Metadata Database, and consists largely of improved values of 1598 the calibrations already determined by Phase 2. The analysis 1599 performed by this pipeline consists of: 1590 1600 1591 1601 \begin{itemize} … … 1598 1608 \end{itemize} 1599 1609 1610 In the Phase 2 analysis, the astrometric solutions were determined 1611 independently for each chip. These solutions are limited by the 1612 assumption of a static distortion and by the accuracy of the 1613 astrometric reference. In the phase 3 analysis, the astrometric 1614 solutions of the $N$ FPA images are improved by... 1615 1616 For image combination in phase 4, should we use relative astrometry to 1617 map N-1 images to 1, or are we sufficiently accurate to use absolute 1618 astrometry to map N images to the sky-cells? 1619 1620 In the Phase 2 analysis, the background is determined based only on 1621 the available sky in a single chip. However, the background 1622 structures are normally correlated on the scale of the FPA, so an 1623 improved background solution can be determined by combining the 1624 information from many chips. \tbd{is the background correlated 1625 between FPAs?} 1626 1627 Phase 3 photometric improvement 1628 1629 In the Phase 4 analysis, the $N$ FPA images are optimally combined to 1630 create a single image of the sky with bad-pixel and cosmic-ray 1631 rejection. This combination requires the calculation of a set of PSF 1632 kernels to convert each of the input images to a single, common PSF. 1633 These PSF kernels are determined from the per-chip PSFs measured in 1634 Phase 2. 1635 1600 1636 \begin{figure} 1601 1637 \begin{center} … … 1605 1641 \end{figure} 1606 1642 1607 In the Phase 2 analysis, the astrometric solutions were determined1608 independently for each chip. These solutions are limited by the1609 assumption of a static distortion and \tbd{by the accuracy of the1610 astrometric reference}. In the phase 3 analysis, the astrometric1611 solutions of the $N$ FPA images are improved by \tbd{???}.1612 1613 \tbd{what is the expected accuracy of the relative astrometric1614 solution compared to the absolute astrometric solution?}1615 1616 \tbd{for image combination in phase 4, should we use relative1617 astrometry to map N-1 images to 1, or are we sufficiently accurate1618 to use absolute astrometry to map N images to the sky-cells?}1619 1620 In the Phase 2 analysis, the background is determined based only on1621 the available sky in a single chip. However, the background1622 structures are normally correlated on the scale of the FPA, so an1623 improved background solution can be determined by combining the1624 information from many chips. \tbd{is the background correlated1625 between FPAs?}1626 1627 \tbd{Phase 3 photometric improvement??} \tbd{Phase 3 determined1628 accurate relative photometry between the N images which are to be1629 combined in the Phase 4 analysis. Is this more accurate than the1630 absolute photometry solution? (probably)}1631 1632 In the Phase 4 analysis, the $N$ FPA images are optimally combined to1633 create a single image of the sky with bad-pixel and cosmic-ray1634 rejection. This combination requires the calculation of a set of PSF1635 kernels to convert each of the input images to a single, common PSF.1636 These PSF kernels are determined from the per-chip PSFs measured in1637 Phase 2.1638 1639 1643 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1640 1644 1641 \paragraph{Phase 4 : image combination} 1642 1643 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1644 1645 \subparagraph{Phase 4 Concept} 1646 1647 Phase 4 processing within the \PS{} image processing pipeline is 1648 the final stage of processing for a science image. It operates on 1649 each sky cell that has overlapping imaging data from the exposure(s) 1650 being processed, and produces the main output image data products of 1651 the pipeline --- the difference images and a deep static sky image --- 1652 along with the associated catalogs of static and variable sources. 1653 1654 \begin{figure} 1655 \begin{center} 1656 \resizebox{8cm}{!}{\includegraphics{pics/phase4}} 1657 \caption{ \label{phase4} Phase 4 dataflow} 1658 \end{center} 1659 \end{figure} 1645 \subsection{Phase 4 : image combination} 1646 1647 \subsubsection{Overview} 1648 1649 Phase 4 processing within the Pan-STARRS image processing pipeline is 1650 the image combination stage of processing for a science image. It 1651 operates on each sky cell that has overlapping imaging data from the 1652 exposure(s) being processed, and produces a set of clean, combined 1653 images of the sky. It also subtracts the current static sky image to 1654 generate a difference image, which it uses to identify transient 1655 objects. These are then excised from the summed image, which is in 1656 turn then added to the static sky image. 1660 1657 1661 1658 Prior to Phase 4, the Phase 3 process produces the following products: … … 1665 1662 \item astrometric calibration with mapping to sky cells; and 1666 1663 \end{itemize} 1664 1667 1665 These will each be used by the Phase 4 modules: 1668 \begin{ enumerate}1666 \begin{itemize} 1669 1667 \item Combine Images; 1670 1668 \item Identify Sources; 1671 1669 \item Transient Identification; and 1672 1670 \item Add to Static Sky. 1673 \end{ enumerate}1671 \end{itemize} 1674 1672 These modules are each explained below. 1675 1673 1676 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1677 1678 \subparagraph{Combine Images} 1674 \subsubsection{Combine Images} 1679 1675 1680 1676 \tbd{for moving objects and images which are not simultaneous, do we … … 1687 1683 telescope, rejecting artifacts such as cosmic rays and low altitude 1688 1684 streaks. The inputs to this module are: 1689 \begin{ enumerate}1685 \begin{itemize} 1690 1686 \item the sky-subtracted images that overlap the sky cell (or portions 1691 1687 thereof) --- from the IPP Pixel Server (or directly from Phase 3); … … 1697 1693 signal-to-noise (i.e.\ sky noise divided by the square of the seeing) 1698 1694 --- from metadata associated with the images. 1699 \end{ enumerate}1695 \end{itemize} 1700 1696 1701 1697 The module maps the detector images to the sky cell using the specified … … 1714 1710 1715 1711 The outputs from this module are: 1716 \begin{ enumerate}1712 \begin{itemize} 1717 1713 \item The combined sky cell image --- sent to the IPP Pixel Server 1718 1714 and/or the next module; … … 1722 1718 \item Catalog of sources on the combined sky cell image --- sent to 1723 1719 the IPP Object Database. 1724 \end{enumerate} 1725 1726 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1727 1728 \subparagraph{Identify Sources} 1720 \end{itemize} 1721 1722 \subsubsection{Identify Sources} 1729 1723 1730 1724 This module identifies sources in the combined sky cell image. The … … 1736 1730 is the catalog of sources on the combined sky cell image, which is to 1737 1731 the IPP Object Database. 1738 1739 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1740 1741 \subparagraph{Transient Identification} 1732 1733 \subsubsection{Transient Identification} 1742 1734 1743 1735 \tbd{what about different stellar colors?} 1744 1736 1745 1737 This module identifies variable/moving sources. The inputs are: 1746 \begin{ enumerate}1738 \begin{itemize} 1747 1739 \item The combined sky cell image --- from the previous module or the 1748 1740 IPP Pixel Server; and 1749 1741 \item The current static sky image --- from the Sky Image Server. 1750 \end{ enumerate}1742 \end{itemize} 1751 1743 1752 1744 The module subtracts the current static sky image from the combined sky … … 1779 1771 1780 1772 The module outputs: 1781 \begin{ enumerate}1773 \begin{itemize} 1782 1774 \item Combined sky cell image, with all variable sources masked --- 1783 1775 used for the next module; … … 1786 1778 \item Catalog of variable sources --- sent to the IPP Object 1787 1779 Database. 1788 \end{enumerate} 1789 1790 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1791 1792 \subparagraph{Add to Static Sky} 1780 \end{itemize} 1781 1782 \subsubsection{Add to Static Sky} 1793 1783 1794 1784 \tbd{how to handle variable stars?} … … 1798 1788 performed if the new data is of sufficient quality that it will not 1799 1789 degrade the static sky image. The inputs are: 1800 \begin{ enumerate}1790 \begin{itemize} 1801 1791 \item The combined sky cell image with variable sources masked --- 1802 1792 from a previous module; … … 1806 1796 each of the images --- estimate made from metadata associated with 1807 1797 each image. 1808 \end{ enumerate}1798 \end{itemize} 1809 1799 1810 1800 The sky cell image is added to the static sky. The sky cell image … … 1816 1806 1817 1807 The output is: 1818 \begin{ enumerate}1808 \begin{itemize} 1819 1809 \item The new static sky image --- sent to the Sky Image Server; 1820 1810 \item The Catalog of sources on the new static sky image --- sent to the IPP Object Database; and 1821 1811 \item The estimated limiting magnitude for the new static sky --- 1822 1812 metadata associated with the the new static sky image. 1823 \end{enumerate} 1824 1825 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1826 1827 \subparagraph{Notes} 1813 \end{itemize} 1814 1815 \subsubsection{Notes} 1828 1816 1829 1817 \begin{itemize} … … 1838 1826 \end{itemize} 1839 1827 1828 \begin{figure} 1829 \begin{center} 1830 \resizebox{8cm}{!}{\includegraphics{pics/phase4}} 1831 \caption{ \label{phase4} Phase 4 dataflow} 1832 \end{center} 1833 \end{figure} 1834 1840 1835 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1836 1837 \section{System Design : Calibration Image Processing} 1838 1839 The Calibration Analysis Stages construct calibrations from the 1840 relevant input data. Some of these combine multiple raw input images 1841 together, after processing, to create a high-quality high-signal 1842 master calibration image. Some of the calibrations are used to 1843 correct other calibrations. Each of the calibration stages must also 1844 provide the tools to test the quality of the input data against 1845 existing master calibration data and to test the consistency of 1846 multiple measurements of the calibration. 1847 1848 The Calibration analysis stages may be performed on whatever 1849 timescales are appropriate and necessary to maintain the quality and 1850 relevance of the calibration images. Below, we list the specific 1851 calibration data which must be constructed in the calibration analysis 1852 stages. 1853 1854 The IPP must generate basic calibration images using the raw bias, 1855 dark, and flat-field (dome or twilight) images obtained by the 1856 telescope as the input. The analysis of these images requires 1857 relatively simple stacking of the input set of images. Outlier 1858 rejection, both of complete input images as well as pixels within the 1859 input stack, must be performed. In addition, each type of image 1860 requires an appropriate normalization which may depend on the data 1861 levels in other detectors in the input set. Each of these calibration 1862 stages must be able to determine from the input stack if the relevant 1863 calibration image needs to be updated and perform an initial test to 1864 see which input images are consistent and valid. 1865 1866 \subsection{Bias Images} 1867 1868 Bias images may be needed to correct for structure in the bias. The 1869 IPP must have the capability of constructing a master bias image from 1870 a stack of raw bias frames. The input bias images, representing 1871 offsets from the overscan level, are processed by subtracting the 1872 overscan, including 1D structure if needed. 1873 1874 The master bias frame construction uses outlier image and outlier 1875 pixel rejection to construct a single high-quality bias frame. The 1876 statistic used to determine pixel values from the input stack can be 1877 set by the user to be one of the following: the sample mean, median, 1878 and mode, robust mean, median, and mode, and the clipped mean and 1879 median. Testing of the input images consists of constructing residual 1880 images, in which the master bias is applied to the input images. 1881 These images may be included or excluded from an additional iteration 1882 of the stack on the basis of their pixel-to-pixel statistics. 1883 1884 \subsection{Dark Images} 1885 1886 Dark images may be needed to correct for structure in the dark 1887 current. The IPP must have the capability of constructing a master 1888 dark image from a stack of raw dark frames. The input dark images are 1889 first corrected for the bias using whatever method is appropriate for 1890 the science images. Master dark frames depend on their exposure time. 1891 As such, the input dark frames must have a limited range of exposure 1892 times, and the output dark frame includes the exposure time as part of 1893 its associated metadata. 1894 1895 The master dark frame construction uses outlier image and outlier 1896 pixel rejection to construct a single high-quality dark frame. The 1897 statistic used to determine pixel values from the input stack can be 1898 set by the user to be one of the following: the sample mean, median, 1899 and mode, robust mean, median, and mode, and the clipped mean and 1900 median. Testing of the input images consists of constructing residual 1901 images, in which the master dark image is applied to the input images. 1902 These images may be included or excluded from an additional iteration 1903 of the stack on the basis of their pixel-to-pixel statistics. A 1904 collection of master dark frames with a range of exposure times are 1905 used to determine the scaling of the dark frame as a function of 1906 exposure time. 1907 1908 \subsection{On-Off Dark Images for Light Leaks} 1909 1910 A type of image which may be necessary for calibrations will be pairs 1911 of images taken at night with the shutter closed with and without the 1912 dome shutter closed. Such a pair of images can be used to determine 1913 any light-leak in the camera which may contribute additional flux 1914 across the mosaic. 1915 1916 \subsection{Flat-Field Images} 1917 1918 Master flat-field images must be constructed from a collection of 1919 input flat-field images. The input flat-field images may be obtained 1920 from any of the standard sources: the dome, the twilight sky, and the 1921 night-time sky. The choice of flat-field input image must be 1922 determined experimentally from observations during the commissioning 1923 phase of the telescope. The IPP flat-field construction system must 1924 be capable of handling any of these sources. 1925 1926 An appropriate set of input images is selected on the basis of their 1927 flux levels, time of observations, and the observing conditions. The 1928 input flat-field images are processed (bias and dark corrected if 1929 needed) and the resulting images are stacked. The master flat-field 1930 construction uses image and pixel outlier rejection to construct a 1931 single high-quality master flat-field frame. The statistic used to 1932 determine pixel values from the input stack can be set by the user to 1933 be one of the following: the sample mean, median, and mode, robust 1934 mean, median, and mode, and the clipped mean and median. Testing of 1935 the input images consists of constructing residual images, in which 1936 the master flat-field image is applied to the input images. These 1937 images may be included or excluded from an additional iteration of the 1938 stack on the basis of their pixel-to-pixel statistics. 1939 1940 \subsection{Mask Images} 1941 1942 Preliminary bad-pixel mask images are generated on the basis of 1943 comparison between raw flat-field images and a cleaned, stacked 1944 master. The mask creation system accepts a collection of flat-field 1945 images and identifies pixels which are consistently poorly flattened. 1946 Pixels which are under-responsive are also identified as pixels to be 1947 masked. 1948 1949 \subsection{Sky \& Fringe Frames} 1950 1951 Fringe-correction frames must be generated to remove the fringe 1952 pattern caused by thin-film interference in the top layers of CCDs, 1953 particularly in the redder passbands. Fringe correction frames may be 1954 constructed on the basis of observations of the night-sky in the 1955 appropriate filters or on the basis of dome fringe lamp observations. 1956 The choice of the appropriate source will be determined experimentally 1957 on the basis of data obtained during the commissioning phase. The IPP 1958 must be capable of handing either source. The images are first 1959 flattened to remove the pixel-to-pixel sensitivity variations of the 1960 detector. The combination of multiple input fringe frames may not be 1961 simply stacked since the amplitude of the fringe pattern varies 1962 independently of other variations in the image. The amplitude of the 1963 fringe pattern in the input frames is measured and the images scaled 1964 to normalize the fringe amplitude to a consistent range (-1 to +1) for 1965 all input images before they are combined with one of the standard 1966 combination statistics (mean, median, mode, etc). The quality of the 1967 input frames is tested by flattening the input image and applying the 1968 master fringe-frame. The resulting residual image statistics are used 1969 to select or exclude specific input images. 1970 1971 \subsection{Shutter Correction Map} 1972 1973 Shutter correction map images may be generated based on the timing 1974 measurements of the shutter itself, or on the basis of dome-flat 1975 images of decreasing exposure times down to the shortest available 1976 exposures. 1977 1978 \subsection{Low-k Sky Models} 1979 1980 Large-scale background structure in images which is not caused by 1981 thin-film interference must also be detected and corrected. Models of 1982 this background structure may be a necessary input to the correction 1983 proceedure. The IPP must have the capability of generating image 1984 models of the large-scale structure patterns observed with the 1985 telescope 1986 1987 \subsection{Flat-Field Correction Frame} 1988 1989 Flat-field images, whether constructed from the dome, twilight, or 1990 night-sky images, do not perfectly correct the detector response in a 1991 consistent fashion across the full field of the camera. The IPP must 1992 have the capability of generating flat-field photometric correction 1993 frames on the basis of the measured photometry of objects which are 1994 moved to a variety of locations on the detector in a sequence of 1995 images. The flat-field correction frames analysis stage makes use of 1996 targetted observations following a specified dither pattern, and 1997 extracts the photometered objects from the AP Database to determine 1998 the necessary photometric corrections. The resulting image is applied 1999 to the master flat-field image. Testing of the correction is 2000 performed by applying the correction to the basic master flat-field 2001 image, applying that flat-field image to the dithered photometry 2002 observations, and performing the object detections. Comparion of the 2003 photometry of individual stars at different locations on the mosaic 2004 will demonstrate the consistency of the flat-field image. 2005 2006 \subsection{Non-Linearity Correction} 2007 2008 The IPP must have the capability of constructing a correction for 2009 non-linearity in the detectors. These frames are constructed from 2010 exposures of a uniform source with a range of exposure times. The 2011 non-linearity correction frames provide polynomial correction 2012 coefficients or a lookup table describing the correction. There is 2013 likely to be a single non-linear correction for each OTA detector, or 2014 potentially for each Cell. The IPP must handle these two cases. 2015 1841 2016 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1842 2017 1843 \paragraph{Calibration Image Processing} 1844 1845 The IPP Calibration Image Pipelines perform the tasks needed to 1846 generate high-quality calibration images from the input image 1847 dataset. These operations may be performed on whatever timescales are 1848 appropriate and necessary to maintain the quality and relevance of the 1849 calibration images. There are four distinct types of calibration 1850 image pipelines: the basic detrend creation pipeline, the photometric 1851 correction image creation pipeline, the fringe pattern generation 1852 pipeline, and the sky foreground pattern generation pipeline. 1853 1854 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1855 1856 \subparagraph{Cal 1: Basic detrend image creation} 1857 1858 The basic detrend image creation pipeline collects the appropriate 1859 input detrend images (bias, dark, dome flat, etc) and generates a 1860 master image by combining the input images in some optimal way 1861 \tbd{median/sigma-clipping/etc}. The master image is used to 1862 determine input image residuals so that poor input images can be 1863 iteratively rejected. 1864 1865 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1866 1867 \subparagraph{Cal 2: Fringe pattern and sky foreground model creation} 1868 1869 The fringe model creation and sky foreground model creation pipelines 1870 use night-sky images with sufficient flux to measure the fringe or sky 1871 models. The input images are processed and optimally combined to yield 1872 a set of correction fringe patterns. The fringe pattern creation and 1873 the sky foreground pattern creation have a similar processing 1874 structure: both require processing of the input images, both determine 1875 a set of principal components as a function of specific input 1876 parameters. 1877 1878 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1879 1880 \subparagraph{Cal 3: Photometric flat correction image creation} 1881 1882 The photometric flat-field correction uses images which have been 1883 dithered with a large range of spatial scales, combined with the 1884 uncorrected flat-field images, to generate a correction to the 1885 flat-field image. This correction compenstates for non-uniform 1886 illumination of the detector during the initial flat-field generation 1887 stage. 1888 1889 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1890 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1891 1892 \paragraph{Calibration Test Processing} 1893 1894 The calibration test processing tests observations to determine if the 1895 calibrations need updating. 1896 1897 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1898 1899 \subparagraph{CalTest 1: Detrend frame testing} 1900 1901 A newly-acquired master detrend frame, having been combined (using Cal 1902 1 or Cal 2) are simply differenced from the old detrend frames. If 1903 there exist significant residuals, the newly-acquired detrend frame 1904 is adopted as the detrend frame of choice. 1905 1906 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1907 1908 \subparagraph{CalTest 2: Photometric flat correction testing} 1909 1910 Newly-acquired photometry of many objects (initially, this may be 1911 standard star fields, but once the PS1 catalog is available, it should 1912 be possible to use all photometry acquired over a given time period) 1913 are compared with previously-acquired photometry. If there exist 1914 significant residuals, a new photometric flat correction should be 1915 produced from the newly-acquired photometry. 1916 1917 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1918 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1919 1920 \paragraph{Reference Catalog Processing} 2018 \section{System Design : Reference Catalog Processing} 1921 2019 1922 2020 The IPP reference catalog pipelines use the data in the IPP Metadata … … 1924 2022 astrometric and photometric calibration references. 1925 2023 1926 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1927 1928 \subparagraph{AstroRef: Astrometric Reference Catalog creation} 2024 \subsection{AstroRef: Astrometric Reference Catalog creation} 1929 2025 1930 2026 This processing stage shall use many observations over a given time … … 1933 2029 published. 1934 2030 1935 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1936 1937 \subparagraph{PhotoRef: Photometric Reference Catalog creation} 2031 \subsection{PhotoRef: Photometric Reference Catalog creation} 1938 2032 1939 2033 This processing stage shall use many observations over a given time … … 1943 2037 1944 2038 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2039 2040 \section{System Design : Miscellaneous Tasks} 2041 2042 In this section, we discuss the design of the science analysis stages 2043 which perform the fundamental image analysis steps of the IPP. 2044 2045 \subsection{Retrieval} 2046 2047 The retrieval stages simply retrieve pixel data from an external 2048 source (ordinarily OATS at the Summit, but it could conceivably be 2049 some other external source) and store it on the nodes. 2050 1945 2051 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1946 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1947 1948 \subsection{Reference Catalogs} 1949 1950 The IPP will employ reference catalogs in order to calibrate the 1951 photometry and astrometry. 1952 1953 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1954 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1955 1956 \subsubsection{Astrometric Reference Catalog} 1957 1958 For PS1, this shall be UCAC. 1959 1960 For PS4, this shall be the PS1 catalog. 1961 1962 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1963 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1964 1965 \subsubsection{Photometric Reference Catalog} 1966 1967 For PS1, absolute photometry will not be available until the master 1968 fit which will be performed when all data is taken. For purposes of 1969 relative photometric extinction, the guide star brightnesses should be 1970 sufficient. 1971 1972 For PS4, the PS1 catalog shall be used. 1973 1974 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1975 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1976 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1977 1978 \subsection{Software Hierarchy} 2052 2053 \section{Software Hierarchy} 1979 2054 1980 2055 In order to facilitate testing and development, and to encourage 1981 2056 flexibility, the IPP will be built in a layered fashion. The lowest 1982 2057 level functions will be written in C and collected together into a 1983 \PS{}library. These library functions will be used to write more2058 Pan-STARRS library. These library functions will be used to write more 1984 2059 complex modules. The modules will be written in C but will make use 1985 2060 of the SWIG tool to make their functionality available within other … … 1997 2072 stringent. 1998 2073 1999 \subs ubsection{External Libraries}2000 2001 \PS{}will employ several external libraries to save duplicating2074 \subsection{External Libraries} 2075 2076 Pan-STARRS will employ several external libraries to save duplicating 2002 2077 functionality that is already available. These external libraries 2003 will be wrapped by the \PS{}Library, insulating the project from the2078 will be wrapped by the Pan-STARRS Library, insulating the project from the 2004 2079 implementation details of the external libraries. Examples of the 2005 2080 external libraries are FFTW and SLALib. 2006 2081 2007 \subs ubsection{\PS{}Library}2008 2009 The \PS{}Library will consist of C structures describing the basic2082 \subsection{Pan-STARRS Library} 2083 2084 The Pan-STARRS Library will consist of C structures describing the basic 2010 2085 data types needed by the IPP and C functions which perform the basic 2011 2086 data manipulation operations. Note that a subset of the library 2012 2087 functions will be provided with SWIG interfaces as well to allow for 2013 2088 their use in the creation of the processing stages. Examples of the 2014 \PS{}Library are fourier transforms and transforming between pixel2089 Pan-STARRS Library are fourier transforms and transforming between pixel 2015 2090 and celestial coordinates. 2016 2091 2017 \subs ubsection{Modules}2092 \subsection{Modules} 2018 2093 2019 2094 The IPP analysis stages are broken down into modules which represent 2020 2095 specific functional operations. The modules will be written in C 2021 using the \PS{} Library functions and will be grouped into a \PS{}2096 using the Pan-STARRS Library functions and will be grouped into a Pan-STARRS 2022 2097 Module Library. The modules will be provided with SWIG interfaces to 2023 2098 all public APIs for their use in processing stages. Examples of … … 2025 2100 (e.g.\ find objects on an image) will be used by multiple stages. 2026 2101 2027 \subs ubsection{Stages}2102 \subsection{Stages} 2028 2103 2029 2104 The major IPP processing tasks are organized into stages, which … … 2036 2111 images from multiple telescopes and search for transients). 2037 2112 2038 \subsection{Modules}2039 2040 \tbd{What goes here? There will be modules?}2041 2042 2113 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2043 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2044 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2045 2046 \subsection{\PS{} Library} 2047 2048 See PSDC-430-007 for the design of the \PS{} Library, PSLib. 2049 2050 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2051 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2052 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2114 2115 \section{Interfaces} 2053 2116 2054 2117 \subsection{Internal Interfaces} … … 2076 2139 C:DB interactions 2077 2140 2078 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2079 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2080 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2081 2082 2141 \subsection{External Interfaces} 2083 2142 … … 2085 2144 2086 2145 This subsection describes the interfaces between the IPP and other 2087 \PS{}systems and the external clients. The interfaces are2146 Pan-STARRS systems and the external clients. The interfaces are 2088 2147 illustrated in Figure~\ref{fig:functionalities}. Incoming data is 2089 2148 received by either the IPS (pixels), the IMD (metadata), or the IOD … … 2092 2151 generated by the IPP Scheduler or the science processing pipelines. 2093 2152 2094 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2095 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2096 2097 \subsubsection{OATS} 2153 \subsubsection{Camera} 2154 2155 \subsubsection{OTIS} 2098 2156 2099 2157 The Summit Pixel Server (SPS) sends raw image data, image metadata, … … 2103 2161 to the IPS while the metadata is sent to the IMD. 2104 2162 2105 The \PS{}Telescope Scheduler (PTS) sends information about the2163 The Pan-STARRS Telescope Scheduler (PTS) sends information about the 2106 2164 telescope schedule to the IPP: observing plan for the night, or longer 2107 2165 time scales. The IPP scheduler sends telescope schedule requests to 2108 2166 the PTS (i.e.\ calibration needs). 2109 2167 2110 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2111 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2112 2113 \subsubsection{Published Static Sky Server} 2168 \subsubsection{PSPS} 2114 2169 2115 2170 The Static Image Server provides segments of the current static sky … … 2118 2173 provides updated static sky images to the SIS when available. 2119 2174 2120 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2121 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2122 2123 \subsubsection{Object Database} 2124 2125 The Master Science Object Database receives new object photometry from 2126 the IPP. The IPP IOD acts as a cache for object photometry data; 2127 \tbd{an IPP subsystem will send photometry data in batches on some 2128 timescale. Is this a function of the IOD?} 2129 2130 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2131 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2132 2133 \subsubsection{Moving Object Processing System} 2175 \subsubsection{MOPS} 2134 2176 2135 2177 The Moving Object Processing System interfaces with the IPP to receive … … 2137 2179 The MOPS may interface with the IMD as needed. 2138 2180 2139 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2140 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2141 2142 \subsubsection{Other Client Science Pipelines} 2181 \subsubsection{Other Preferred Client Science Pipelines} 2143 2182 2144 2183 The client science pipelines may interface with the IPP via requests … … 2147 2186 2148 2187 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2188 2189 \section{Computer Hardware} 2190 2191 \subsection{PS-1 Cluster requirements} 2192 2193 \begin{itemize} 2194 \item CPU requirements 2195 \item per-node I/O requirements 2196 \item switch throughput requirements 2197 \item storage profile 2198 \end{itemize} 2199 2200 \subsection{PS-1 Cluster Hardware Plan} 2201 2202 \begin{itemize} 2203 \item COTS equipment 2204 \item number of processors needed 2205 \item number of I/O ports needed 2206 \item number of disk slots needed 2207 \item switch choice 2208 \item design choice for computer nodes 2209 \item total rack space 2210 \end{itemize} 2211 2212 \subsection{PS-1 Cluster Expected Reliability} 2213 2214 \subsection{PS-1 Cluster Support} 2215 2149 2216 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2150 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2151 2152 \subsection{Computer Hardware} 2153 2154 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2155 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2156 2157 \subsubsection{Overview} 2158 2159 This document discusses the likely range of the \PS{} Image 2160 Processing Pipeline (IPP) hardware requirements. The hardware 2161 requirements addressed in this document consist of: 2162 2163 \begin{itemize} 2164 \item Total Disk Volume 2165 \item Total Processing Power 2166 \item Sustained Switch Bandwidth 2167 \item Sustained Node Network I/O 2168 \item Sustained Disk I/O 2169 \end{itemize} 2170 2171 Even without the complete IPP design, it is possible to identify the 2172 major drivers on the hardware requirements. The total disk volume 2173 requirements are dominated by the need to store raw images for a 2174 certain period, the need to store calibration images for a longer 2175 period, and the need to store the static sky images. Of the various 2176 analysis pipelines, and depending on the data organization as 2177 discussed below, Phase 2 and Phase 4 present the most significant 2178 demands in terms of data I/O throughput on the network. Phase 2 and 2179 Phase 4 also present the most significant CPU demands. In this 2180 discusion, Phase 2 refers to the per-chip pre-processing in which the 2181 instrumental signature is removed and a first pass object detection is 2182 performed. Phase 4 refers to the multiple chip combination in which 2183 the pre-processed images are merged and combined, in both addition and 2184 subtraction, with the static sky image, and up to three object 2185 detection passes are performed. 2186 2187 This document does not address the hardware requirements implied by 2188 the Phase 0, 1, or 3 stages, nor the load required by the calibration 2189 image creation stages. In the first instance, the operations are only 2190 performed on the metadata and are extremely minimal both in terms of 2191 data I/O and computation requirements. In the second case, the 2192 processing is less time critical than the per-image processing and is 2193 performed only infrequently (once per night to once per week or 2194 month). This document also does not address any hardware requirements 2195 introduced by the metadata manipulation. The software implementation 2196 for metadata storage (RDBMS, FITS tables, etc) will have a very large 2197 impact and will be evaluated along with the needed hardware at a later 2198 date. 2199 2200 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2201 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2202 2203 \subsubsection{Scenarios} 2204 2205 We will address the various hardware requirements by referring to a 2206 set of data processing and data organization scenarios. The actual 2207 hardware requirements will depend on design decisions which are not 2208 yet available. It is possible to define the data organization in ways 2209 which will minimize the hardware requirements, but which will increase 2210 the software development effort. We will discuss both the worst-case 2211 data organization scenario, which does not require significant 2212 intelligence in the software systems, and the optimal data 2213 organization scenario, which will require the software to track the 2214 location of data products more carefully. In addition, this document 2215 will address the data requirements of the complete \PS{} pipeline 2216 with 4 telescopes as well as the single-telescope \PS{}-1 scenario 2217 based on the Design Reference Mission [REF]. 2218 2219 The IPP hardware system must provide both data storage and 2220 computational resources. The IPP requires relativley large amounts of 2221 data storage space, primarily for the image data. Image data is 2222 organized in two categories. First, there is the per-chip data -- 2223 data associated with specific chips, including the raw images, the 2224 calibration images, and temporary processed images at various stages. 2225 Second, there is the data associated with the static sky imagery, 2226 which is in turn organized into smaller sky-cell units. The first 2227 assumption we make is that the hardware is organized into nodes which 2228 provide both data storage and computational resources. The second 2229 assumption we make is that the data storage nodes are divided into two 2230 classes: those which deal with the per-chip data and those that 2231 provide the static sky storage. In addition, we assume that the 2232 computational tasks related to Phase 2 take place on the per-chip 2233 storage nodes and the Phase 4 computation takes place on the static 2234 sky storage nodes. 2235 2236 Figure~\ref{hardware} shows our basic concept for the hardware 2237 organization for the IPP. This diagram shows the two types of compute 2238 nodes: chip-level processing and storage nodes (dominated by Phase 2) 2239 and static sky processing and storage nodes (mostly Phase 4). Also 2240 shown are two switches used in this configuration; although it is 2241 currently possible to buy a single switch which would have a 2242 sufficient number of GigE ports for both sections of the PS-1 system, 2243 such a two-switch organization may be needed for the full \PS{} 2244 system. In such a case, the interswitch communication must also meet 2245 the required throughput needs. We discuss the hardware requirements 2246 in the assumption that such an organization will be necessary. 2247 2248 The way in which the images are distributed among the storage and 2249 compute nodes will largely determine the I/O bandwidth requirements. 2250 For data bandwidth requirements calculations, it is necessary to make 2251 some assumptions about the data organization. For the purposes of 2252 this document, we explore two extreme-case options: 2253 \begin{itemize} 2254 \item Random Data Distribution --- Detector \& Sky data is randomly 2255 distributed within the compute node of a given type (ie, chip data 2256 is randomly distributed among the detector compute nodes). 2257 \item Optimal Data Distribution --- Detector \& Sky data is optimally 2258 distributed to compute Detector/Sky nodes (chip processing is always 2259 on a machine with local chip data). 2260 \end{itemize} 2261 A second factor which will have a significant impact on the I/O 2262 requirements is the image storage format for the processed and 2263 calibration images. We have two basic choices: 32 bit floating point 2264 format or 16 bit integer format with appropriate scaling. In the 2265 former case, additional dynamic range is retained, while in the latter 2266 case, we reduce the data volume by a factor of 2. While some may 2267 argue that the higher dynamic range is necessary, arguments can be 2268 made that the 16 bit range is sufficient. (In particular, the 16 bit 2269 data provides a dynamic range far above the expected 1/1000 fractional 2270 accuracy of the flat-field images). A related question is the number 2271 of calibration images needed by the processing system. Since the 2272 complete analysis is not yet defined, this number is difficult to 2273 ascertain. However, we can make a range of assumptions which are 2274 reasonable. We therefore adopt two data volume scenarios to explore 2275 these possibilites: 2276 \begin{itemize} 2277 \item Standard Data Volume - 32 bit data for processed and calibration 2278 images, average of 7 calibration frames per image. 2279 \item Minimal Data Volume - 16 bit data for processed and calibration 2280 images, average of 4 calibration frames per image. 2281 \end{itemize} 2282 In the discussion that follows, we explore the hardware requirements 2283 implied by the collection of four combinations of these two sets of 2284 scenario options. 2217 2218 \clearpage 2219 2220 \section{Appendices} 2221 2222 \subsection{Image Server Database Table Contents} 2285 2223 2286 2224 \begin{table} 2287 2225 \begin{center} 2288 \caption{ Hardware Throughput Tests \label{existing-hardware}}2289 \begin{tabular}{l rrrr}2290 \hline 2291 \hline 2292 Test & where \& when & model & result\\2293 \hline 2294 node I/O & CFHT 11/2002 & Intel 1000 Gigabit & 35 - 40 MB/s sustained\\2295 node I/O & CFHT 2/2004 & Intel 1000 Gigabit & 65 - 70 MB/s sustained\\2296 RAID write & CFHT 2/2004 & 3ware RAID cntl + IDE & 110 MB/s sustained\\2297 Switch Load & VeriTest & Cisco & 3 GB/s (for 32 ports)\\2226 \caption{Storage Object Table Contents\label{ImageServerTables:SO}} 2227 \begin{tabular}{lll} 2228 \hline 2229 \hline 2230 {\bf Column Name} & {\bf Datatype} & {\bf Description} \\ 2231 \hline 2232 \code{so_id} & integer & internal storage object identifier \\ 2233 \code{ext_id} & string & external storage object identifier (file ID) \\ 2234 \code{comment} & string & user description of object \\ 2235 \code{epoch} & date/time & last date of access \\ 2298 2236 \hline 2299 2237 \end{tabular} … … 2301 2239 \end{table} 2302 2240 2303 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2304 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2305 2306 \subsubsection{Existing Hardware Throughput} 2307 2308 We have collected a few representative tests of various pieces of 2309 modern hardware to give a reference for the throughput capabilities. 2310 A number of hardware configurations have been tested at CFHT for the 2311 Elixir project, and we include here their recent reported hardware 2312 RAID-5 I/O speeds and GigE card speeds. We also have included data 2313 from VeriTest studies of Cisco switch throughput, commissioned by 2314 Cisco for a 32 port GigE switch. These tests are summarized in 2315 Table~\ref{existing-hardware}. 2316 2317 \begin{table}[b] 2241 \begin{table} 2318 2242 \begin{center} 2319 \caption{Data Storage Requirements \label{storage}} 2320 \begin{tabular}{lrrrr} 2321 \hline 2322 \hline 2323 & Standard / PS-4 2324 & Standard / PS-1 2325 & Minimal / PS-4 2326 & Minimal / PS-1 \\ 2327 \hline 2328 Raw data & 300 TB & 75 TB & 300 TB & 75 TB \\ 2329 static sky & 512 TB & 64 TB & 256 TB & 32 TB \\ 2330 calibration frames & 175 TB & 18 TB & 17 TB & 5 TB \\ 2331 metadata db & 2 TB & 2 TB & 0.2 TB & 0.2 TB \\ 2332 object db & 60 TB & 4 TB & 60 TB & 4 TB \\ 2333 \hline 2334 totals & 1050 TB & 163 TB & 633 TB & 116 TB \\ 2243 \caption{Instance Table Contents\label{ImageServerTables:INT}} 2244 \begin{tabular}{lll} 2245 \hline 2246 \hline 2247 {\bf Column Name} & {\bf Datatype} & {\bf Description} \\ 2248 \hline 2249 \code{ins_id} & integer & internal instance identifier \\ 2250 \code{so_id} & integer & key to storage object table \\ 2251 \code{uri} & string & location in hardware collection \\ 2252 \code{sha1sum} & string & checksum information \\ 2253 \code{assigned_location} & boolean & is location user-specified? \\ 2254 \code{epoch} & date/time & last date of access \\ 2255 \code{atime} & date/time & last date of access \\ 2335 2256 \hline 2336 2257 \end{tabular} … … 2338 2259 \end{table} 2339 2260 2340 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2341 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2342 2343 \subsubsection{Data Storage Requirements}2344 2345 The \PS{} IPP data storage requirements may be divided into five2346 principal areas: raw image data, static sky image data, master2347 calibration images, the metadata database, and the object database.2348 We discuss each of these data items and their impact on the data2349 storage requirements for the IPP, and identify the impact of the2350 minimal vs standard data storage requirements as well as the2351 requirements specifically for PS-1. Table~\ref{storage} summarizes2352 the data storage requirements in the different scenarios.2353 2354 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2355 2356 \paragraph{Raw Data Storage}2357 2358 There are two basic image types which will be acquired: night-time2359 science images and calibration images. The night-time science images2360 consist of 1Gpix per image, or 2GB in raw format. At nominal cadence,2361 the 4 telescopes can obtain images at a sustained rate of 1 image per2362 30 seconds per telescope for the entire night of 10 hours (360002363 minutes). A total of 100 calibration images per night would be a2364 substantial overestimate of the typical expectation. Combining these2365 numbers, we can expect to receive a total of 1300 image per telescope2366 per night, 5200 image total, or 10.4 TB of data per night. The total2367 data storage requirements for the raw data are governed by the number2368 of nights' worth of data we are required to keep online. A reasonable2369 number is one month to allow a full moon's cycle. Thus, for raw image2370 storage, we require a total of 300 TB data storage. For PS-1, this2371 number is simply scaled down by a factor of 4. The choice of the2372 minimal data volume does not affect these numbers because the raw data2373 is already stored with 16 bit pixels.2374 2375 \tbd{The PS-1 design reference may now require storage of the entire2376 first year of data, calculated to be 200 TB.}2377 2378 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2379 2380 \paragraph{Static Sky Data Storage}2381 2382 The static sky is represented by images with 0.2 arcsec per pixel.2383 There will be one summed image and one weight image for each of the 62384 filters, each stored in floating point format. At this resolution,2385 there are 324 Mpix per square degree, and we will observe a potential2386 total area of 30,000 square degrees. Allowing for 10\% overage for2387 overlapping tiling, we require a total of 10.7 Gpix to cover the sky2388 once, or a total of $\sim 512$ TB for the static sky images. In the2389 minimal data volume scenario, this value is reduced by a factor of 2,2390 while in PS-1, the reduction is a factor of roughly 8 because we only2391 intend to store the static sky for the ecliptic plane survey and the2392 small IPP verification program.2393 2394 \tbd{This last point is no longer valid - the PS-1 static sky may2395 require the entire 3pi.}2396 2397 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2398 2399 \paragraph{Calibration Frame Storage}2400 2401 The possible required calibration frames consist of the bias, dark,2402 and mask images, along with one flat, one flat-correction, and2403 multiple sky/fringe library frames per filter. In fact, not all types2404 are needed at all stages. For the standard data volume, we assume an2405 average of 7 calibration frames per image and filter. This results in2406 a total of 42 master calibration image per telescope. If we intend to2407 keep all master calibration frames for the project lifetime, and2408 generate a new master on a weekly basis (a reasonable time-scale),2409 then we can expect to require a total of 175 TB of calibration image2410 by the end of the 5 year lifetime of the project. For the case of2411 PS-1, the time period is only 2 years, and there is only 1 telescope,2412 resulting in a factor of 10 reduction in the volume. For the minimal2413 data case, we reduce the volume by another factor of 3.5. We also note2414 that this is likely to be a drastic overestimate as we are unlikely to2415 need to regenerate all master calibration frames on a weekly2416 time-scale.2417 2418 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2419 2420 \paragraph{Metadata Database Storage}2421 2422 The metadata data storage requirements are driven by the need to store2423 the data for the project lifetime. There are two types of metadata2424 generated at the summit: data associated with images and environmental2425 data. The environmental data consists of measurements on a regular2426 cadence, roughly 1 per minute, of a variety of parameters. We suggest2427 an expected of 1kB per entry, for a total of 2.6 GB over the lifetime2428 of the project. PS-1 will represent a smaller amount of data per2429 minute, and also a factor of 2.5 fewer minutes. We suggest PS-1 may2430 have a total environmental metadata set smaller by a factor of 5. The2431 additional systems, such as the DIMM, SkyProbe, NIR Sky Camera, and2432 the LRProbe will have higher data requirements, but should be2433 considered as separate, self-contained systems. Their data products2434 are distilled to a limited number of parameters per minute which are2435 included in the 1kB given above. Furthermore, items such as2436 guide-star history, if saved, will be saved with the image data and2437 represents only a small fraction of the total image data volume. Some2438 subset of the telescope diagnosic information may be a high volume2439 data product as well, but only retained by the telescope control2440 system for the purpose of diagnostic studies. Such data will be2441 excluded from this analysis.2442 2443 The image metadata consists of values associated with the FPA (4), the2444 chips (240), and the Cells (15360). Aside from the guide star2445 history, the total data requirements for each of these entries will be2446 scaled by the number of bytes required for the metadata from each data2447 level. Clearly, if the Cell entry is allowed to be large, it will2448 dominate the total Metadata data volume. If we suggest an expected2449 number of 64~bytes per Cell, 256~B per chips, and 1~kB per FPA, we find a2450 total metadata volume per exposure of roughly 1~MB, completely2451 dominated by the Cell metadata. With the exposure rates above, we2452 find a total of metadata volume of 1.8~TB over the lifetime of the2453 project. For PS-1, the total volume is reduced by a factor of 2.52454 (for the shorter lifetime) and another factor of 4 (for the lone2455 telescope). Neither data quantity is affected by the minimal vs2456 standard data volume choice.2457 2458 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2459 2460 \paragraph{Object Database Storage}2461 2462 The hardware requirements for the IPP object database are rather2463 flexible: the total volume depends critically on the depth to which2464 the object detection analyses are performed (and thus the total number2465 of object detections) and the number of object parameters which are2466 measured. We can make very rough estimates that the total number of2467 detections over the 5 year lifetime of the project may be in the2468 vicinity of $5\times10^{11}$. We can conservatively estimate the2469 number of bytes needed to represent each detection as 128 B, resulting2470 in a total data storage for the object detections of 60 TB. However,2471 this number depends strongly on the timescale for which the IPP is2472 required to maintain all object detections, and may potentially be2473 significantly reduced. For the case of PS-1, the total number of2474 detections is likely to be reduced by a factor of 4 for the number of2475 telescopes, and potentially another significant factor ($\sim 4?$) by2476 limiting the depth of object detections. Again, the minimal data2477 volume scenario is irrelevant to the object database volume.2478 2479 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2480 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2481 2482 \subsubsection{CPU Requirements}2483 2484 Phase 2 and Phase 4 dominate the processing requirement, primarily2485 because they must keep up with the image delivery rate of 1 per 302486 seconds. We have performed benchmarks of a demonstration version for2487 both the Phase 2 and Phase 4 analyses.2488 2489 For the Phase 2, a substantial fraction of the processing time is2490 consumed by the need to perform FFTs on the images in order to2491 convolve them with the guide-star kernel, and in the smoothing used2492 for the object detection process. Additional processing time is2493 needed by the object detection, deblending, and analysis. Experiments2494 with the FFTW package show that FFTs may be performed on Intel2495 processors at rates of approximately 0.25~GHz-sec / Mpix for data sets2496 of order 1 Megapixel. The FFTs required for the Phase 2 analysis are2497 performed on the 512$^2$ pixel cells, so these numbers may roughly be2498 scaled linearly to determine the total time required for chip2499 processing. A single FFT on a full chip, with 64 cells, therefore2500 requires roughly 4~GHz-sec. For the full Phase 2 analysis, there are2501 roughly 4 single direction FFTs required excluding those associated2502 with object detection; thus the total processing time for these FFTs2503 is approximately 16~GHz-sec. The addtional analysis steps, excluding2504 object detection and characterization, account for a small fraction of2505 this compute time, which we estimate at 10\%. The object detection2506 stage depends somewhat on the depth to which the analysis is2507 performed, and the number of measurements made per object. Typical2508 analysis performed by the Sextractor routine, which performs a2509 substantial number of per-object analyses, requires 27~GHz-sec for a2510 full chip, including the FFTs used for smoothing. We can therefore2511 assume a total of 50~GHz-sec per chip for the Phase 2 processing.2512 This converts to a total of 12,000~GHz-sec for a complete major frame.2513 2514 For Phase 4, the main computational tasks are combining the multiple2515 images, with cosmic-ray rejection, and performing the object detection2516 tasks. Nick Kaiser has done tests of the Phase 4 image combine and2517 rejection stages, and finds a total processing time of roughly2518 96~GHz-sec for a full stack of 4 chips. If we add in an additional2519 34~GHz-sec for detailed object detection and image differencing, we2520 find a conservative estimage of 130~GHz-sec for a 4-image chip stack,2521 equivalent to 7800~GHz-sec for a major frame.2522 2523 For PS-1, the data processing will clearly require a smaller amount of2524 computational resources because of the lower image rate. However, the2525 total number of GHz-sec required for the complete analysis of 4 input2526 images and the combination with the static sky will remain2527 more-or-less the same. Some reduction in the load may be gained by2528 reducing the complexity and depth of analysis for PS-1. Depending on2529 the details and depth of the analysis, we may reduce the computational2530 load by a factor of 2.2531 2532 2261 \begin{table} 2533 2262 \begin{center} 2534 \caption{Data Scenarios (MB per Chip or Sky-cell) \label{scenarios}} 2535 \begin{tabular}{lrrrr} 2536 \hline 2537 \hline 2538 & Random / Standard & Random / Minimal & Optimal / Standard & Optimal / Minimal \\ 2539 \hline 2540 {\em Phase 2 input} & & & & \\ 2541 from summit & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB \\ 2542 input image & 32 MB & 32 MB & {\bf 32 MB} & {\bf 32 MB} \\ 2543 calibration & $7 \times 64$ MB & $4 \times 32$ MB & {\bf 7 $\times$ 64 MB} & {\bf 4 $\times$ 32 MB} \\ 2544 mask image & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ 2545 \hline 2546 network I/O: & 560 MB & 232 MB & 64 MB & 64 MB \\ 2547 disk I/O: & (560 MB) & (232 MB) & 496 MB & 168 MB \\ 2548 & & & & \\ 2549 {\em Phase 2 output} & & & & \\ 2550 output image & 64 MB & 32 MB & {\bf 64 MB} & {\bf 32 MB} \\ 2551 output mask & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ 2552 image to P4 & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB \\ 2553 mask to P4 & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB \\ 2554 \hline 2555 network I/O: & 200 MB & 100 MB & 120 MB & 60 MB \\ 2556 disk I/O: & (80 MB) & (40 MB) & 80 MB & 40 MB \\ 2557 & & & & \\ 2558 {\em Phase 4} & & & & \\ 2559 input images & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & & \\ 2560 input masks & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & & \\ 2561 static sky & 64 MB & 64 MB & & \\ 2562 static weight & 64 MB & 32 MB & & \\ 2563 \hline 2564 input: & 608 MB & 336 MB & & \\ 2565 output: & 192 MB & 128 MB & & \\ 2566 \hline 2567 \multicolumn{5}{l}{\em Bold-faced entries are access to local-disk} \\ 2568 \multicolumn{5}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\ 2263 \caption{Volume Table Contents\label{ImageServerTables:VOL}} 2264 \begin{tabular}{lll} 2265 \hline 2266 \hline 2267 {\bf Column Name} & {\bf Datatype} & {\bf Description} \\ 2268 \hline 2269 \code{vol_id} & integer & internal volume identifier \\ 2270 \code{uri} & string & node name? \\ 2271 \hline 2569 2272 \end{tabular} 2570 2273 \end{center} 2571 2274 \end{table} 2572 2573 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2574 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2575 2576 \subsubsection{Per-Node I/O Requirements} 2577 2578 Data I/O per node is defined as the number of bytes per second passed 2579 through the node's network adapter. The data throughput for each node 2580 depends strongly on the scenarios identified above. In this section, 2581 we identify the data which is passed between nodes for each of the 2582 different scenarios. Table~\ref{scenarios} lists the per-node data 2583 I/O for the four scenarios. 2584 2585 For PS-4, there are only 30 seconds of compute time allowed for each 2586 of the Phase 2 and Phase 4 analyses. We use the data I/O volumes and 2587 some assumptions about expected network and disk bandwidth to estimate 2588 the I/O and processing timeline for the four scenarios. From this 2589 analysis, we can judge the total CPU requirements in terms of GHz, not 2590 just GHz-sec. We have assumed that GigE network adapters are capable 2591 of delivering data at 50MB/sec sustained and that a disk RAID can 2592 deliver sustained 100 MB/sec reads and writes. These numbers are 2593 conservative estimates based on recent tests discussed above. Using 2594 these assumptions, Table~\ref{throughput} lists the time allocations 2595 for the complete set of scenarios for the case of PS-4. 2596 2597 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2598 2599 \paragraph{Random / Standard Data Scenario} 2600 2601 In the Random Data Distribution scenario, there is a single CPU 2602 allocated to each chip in the detector farm and a single CPU for each Sky 2603 cell process. The chip data are stored across random machines in the 2604 detector farm, with the result that every Phase 2 processing requires 2605 network access to the data. For each science chip which is 2606 observed, each detector node will read from the network a total of 560 MB 2607 (the 2 raw images for data storage and the 7 calibration frames, along 2608 with one mask and one raw input image) and write a total of 200 MB 2609 (one processed image and the mask along with the 1.5 processed images 2610 and masks for the Phase 4 analysis). Given the assumption of 50 MB/s 2611 from the network adapter, the total data volume implies an I/O period 2612 of 15.2 seconds. Note that the disk I/O is parallel with the network 2613 I/O and substantially underfills the disk bandwidth. 2614 2615 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2616 2617 \paragraph{Random / Minimal Data Scenario} 2618 2619 In the Random-Minimal, there is a single CPU allocated to each chip in 2620 the detector farm and a single CPU for each Sky cell process, and the 2621 chip data are stored across random machines in the detector farm. 2622 However, the calibration and the processed science images are stored 2623 at 2 bytes per pixel, the mask is set at 4 bits per pixel, and only 4 2624 calibration images are assumed. For each science chip which is 2625 observed, each detector node will read from the network a total of 232 MB 2626 (the 2 raw images for data storage and the 4 calibration frames, along 2627 with one mask and one raw input image) and write a total of 100 MB 2628 (one processed image and the mask along with the 1.5 processed images 2629 for the Phase 4 analysis). Given the assumption of 50 MB/s from the 2630 network adapter, the total data volume implies an I/O period of 6.6 2631 seconds. Again, note that the disk I/O is parallel with the network 2632 I/O and substantially underfills the disk bandwidth. 2633 2634 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2635 2636 \paragraph{Optimal / Standard Data Scenario} 2637 2638 In the Optimal Data Distribution scenario, there is a single CPU 2639 allocated to each chip in the detector farm and a single CPU for each 2640 Sky cell process. In addition, all data for the specified chip are 2641 stored on local disks attached to the same computer as the CPU, with 2642 the result that all Phase 2 I/O is made to a local disk. For each 2643 science chip which is observed, each detector node will read from the 2644 network a total of 2 raw images (one for the original image, one for 2645 the backup copy) and write an average of roughly 1.5 processed images 2646 and masks to the Phase 4 machines for a total of 184 MB of network 2647 I/O. During the processing stage, the detector node will read from 2648 disk a total of 496 MB (7 calibration frames at 64 MB each, one 16 MB 2649 mask, and one raw science image at 32 MB) and write a total of 80 MB 2650 (one processed image at 64 MB and one mask at 8 MB). Given the 2651 assumptions for the network and disk bandwidths (50 MB/s and 100 MB/s 2652 respectively), the data volumes imply a total I/O period of 9.5 2653 seconds. In this instance, the network I/O is presumed to be 2654 sequential with the disk I/O. 2655 2656 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2657 2658 \paragraph{Optimal / Minimal Data Scenario} 2659 2660 In the Optimal / Minimal Scenario, the minimal data sizes are used 2661 with the optimal data distribution scheme. In this case, we reduce 2662 the disk I/O volume to 168 read and 40 MB write, and the network 2663 traffic to 124 MB. Given the assumptions for the network and disk 2664 bandwidths, the data volumes imply a total I/O period of 4.6 seconds. 2665 Again, the network I/O is presumed to be sequential with the disk I/O. 2666 2667 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2668 2669 \paragraph{Phase 4 Node I/O Requirements / Standard Data Volume} 2670 2671 Although it is easy to arrange the detector data in such a way that 2672 the majority of I/O is performed locally, it is not as easy to arrange 2673 this for the Static Sky data used by the Phase 4 analysis. We 2674 therefore make the assumption that the Phase 4 analysis will require 2675 all input detector data to be loaded across the network, as well as 2676 all Static Sky data. This is somewhat of an overestimate as some of 2677 the Static Sky data will be processed by machines with the data stored 2678 locally, and clever Static-Sky data organization schemes can enhance 2679 this chance. 2680 2681 In the Phase 4 analysis, the images from the 4 separate telescopes are 2682 combined into a single image, confronted with the appropriate segment 2683 of the static sky, with output difference image and updated static sky 2684 image. If we restrict input access to the individual chip cells, the 2685 maximum read overhead is 50\% (need to read a 10x10 set of cells for 2686 an 8x8 input image). If the processing is performed on Static Sky 2687 segments equivalent in size to the chips, the input data is 608 MB (384 2688 MB of processed science image, 96 MB of mask images, 64 MB of static 2689 sky image and 64 MB of static sky weight map) while the output data is 2690 192 MB (static sky, weight map, and difference image, each 64 MB). 2691 Thus, we require a total of 800 MB network I/O. Given the network 2692 bandwidth, this implies an I/O period of 16 seconds for Phase 4. 2693 2694 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2695 2696 \paragraph{Phase 4 Node I/O Requirements / Minimal Data Volume} 2697 2698 In the minimal data volume scenario, the Phase 4 analysis volume is 2699 significantly reduced. The total volume of input data is 336 MB (192 2700 MB of processed science image, 48 MB of input mask, 64 MB of static 2701 sky image and 32 MB of static sky weight map) while the output data is 2702 128 MB (64 MB static sky, 32 MB weight map, and 32 MB difference 2703 image). Thus, we require a total of 464 MB network I/O, which implies 2704 an I/O period of 9.3 seconds. 2275 \clearpage 2276 2277 \subsection{Metadata Database Table Contents} 2278 2279 Tables \tbd{NN} -- \tbd{NN} list the basic contents of each of the 2280 Metadata Database tables listed in Section~\ref{Metadata}. 2705 2281 2706 2282 \begin{table} 2707 2283 \begin{center} 2708 \caption{Data Throughput for 4 Scenarios \label{throughput}} 2709 \begin{tabular}{lrrrr} 2710 \hline 2711 \hline 2712 & 2713 \multicolumn{1}{c}{Random / Standard} & 2714 \multicolumn{1}{c}{Random / Minimal} & 2715 \multicolumn{1}{c}{Optimal / Standard} & 2716 \multicolumn{1}{c}{Optimal / Minimal} \\ 2717 \hline 2718 Phase 2 per-node network I/O & 15.2 s & 6.6 s & 3.7 s & 2.5 s \\ 2719 Phase 2 per-node disk I/O (read) & (5.6 s) & (2.3 s) & 5.0 s & 1.7 s \\ 2720 Phase 2 per-node disk I/O (write) & (0.8 s) & (0.4 s) & 0.8 s & 0.4 s \\ 2721 Phase 2 CPU total & 14 s : 860 GHz & 23 s : 520 GHz & 20 s : 600 GHz & 25 s : 480 GHz \\ 2722 Phase 4 per-node I/O & 16 s & 9.3 s & & \\ 2723 Phase 4 CPU total & 14 s : 490 GHz & 20 s : 390 GHz & & \\ 2724 Phase 2 switch load & 6.1 GB/s & 2.7 GB/s & 1.5 GB/s & 1.0 GB/s \\ 2725 Phase 4 switch load & 0.8 GB/s & 0.5 GB/s & 0.8 GB/s & 0.5 GB/s \\ 2726 Phase 2 to Phase 4 switch load & 1.1 GB/s & 0.6 GB/s & 1.1 GB/s & 0.6 GB/s \\ 2727 Summit to Phase 2 switch load & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s \\ 2284 \caption{Weather Table: some sample weather points\label{WeatherTable}} 2285 \begin{tabular}{lll} 2286 \hline 2287 \hline 2288 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2289 \hline 2290 Time & date/time & The time the weather information was measured. \\ 2291 Temperature 01 & float & The external temperature \\ 2292 Temperature 02 & float & The temperature at top of the dome \\ 2293 Temperature 03 & float & The temperature on the primary mirror \\ 2294 Humidity & float & The relative humidity. \\ 2295 Pressure & float & The (external) atmospheric pressure. \\ 2728 2296 \hline 2729 2297 \end{tabular} … … 2731 2299 \end{table} 2732 2300 2733 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2734 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2735 2736 \subsubsection{Switch I/O Requirements} 2737 2738 The switch I/O requirements are defined by the total number of bytes 2739 per second serviced by the two switches in the system. For the 2740 analysis of the Switch I/O requirements, the choice of data 2741 distribution again has a major impact. We again test the four 2742 scenarios discussed above: Random Data Distribution, Random / Minimal, 2743 Optimal Data Distribution, and Optimal / Minimal. 2744 2745 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2746 2747 \paragraph{Random / Standard Data Scenario} 2748 2749 In the Random Data Distribution scenario, each detector node needs to 2750 read a total of 560 MB from the network and write a total of 200 MB 2751 every 30 seconds. With 240 detector nodes, this corresponds to a 2752 total bandwidth of 6080 MB/sec, or 49 Gb/sec. Note that this includes 2753 the bandwidth needed to copy data from the summit and make two copies 2754 on the detector machines, as well as the bandwidth to send the processed 2755 image portions to the Phase 4 machines. The Phase 4 processing adds 2756 an additional 320 MB of network I/O per Sky-Cell group, and there are 2757 roughly 60-70 Sky-cells per exposure set. Thus the Phase 4 processing 2758 adds an additional 750 MB/sec network bandwidth. In the architecture 2759 defined in Figure \tbd{NN}, the Sky nodes and the detector nodes are each 2760 attached to separate switches. An additional bandwidth requirement is 2761 derived by the need to exchange data between these switches in for 2762 Phase 4. The total amount of data exchanged between these switches is 2763 480 MB per Sky-cell, for a total bandwidth of 1120 MB/sec. In 2764 addition, the connection to the summit is a single, separate line 2765 which needs to support the bandwidth requirement of copying all intial 2766 raw images. In our simple model, each raw image is copied twice, 2767 accounting for a total of 15360 MB every 30 seconds, or a bandwidth 2768 load of 512 MB/sec. (Note that this last is double the actual 2769 bandwidth requirement to the summit: a dedicated local circular buffer 2770 would reduce the need for the second copy to come directly from the 2771 summit.) 2772 2773 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2774 2775 \paragraph{Random / Minimal Data Scenario} 2776 2777 In the Random / Minimal Scenario, the data volumes are significantly 2778 reduced. The total Phase 2 bandwidth contribution is 332 MB over 30 2779 seconds for 240 nodes (2656 MB/sec) and the residual Phase 4 bandwidth 2780 load is 224 MB per Sky cell over 30 seconds (522 MB/sec). The 2781 inter-switch communication is now 240 MB per sky cell over 30 seconds, 2782 or 560 MB/sec. 2783 2784 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2785 2786 \paragraph{Optimal / Standard Data Scenario} 2787 2788 In the Optimal Data Distribution, the Phase 2 network bandwidth is 2789 reduced significantly to 184 MB per detector node, for a total of 2790 1.5GB/sec, while the Phase 4 network bandwidth remains unchanged at 2791 750 MB/sec. The inter-switch communication also remains the same at 2792 1.12 GB/sec. 2793 2794 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2795 2796 \paragraph{Optimal / Minimal Data Scenario} 2797 2798 In the Optimal / Minimal Scenario, the total Phase 2 network bandwidth 2799 drops to 124 MB per detector node, for a total of 1.0GB/sec, while the 2800 Phase 4 network bandwidth is 552 MB/sec. The inter-switch 2801 communication remains the same as the Random/Minimal Scenario at 560 2802 MB/sec. 2803 2804 \begin{table}[t] 2301 \begin{table} 2805 2302 \begin{center} 2806 \caption{\label{NP2} Phase 2 load per major frame (12000 GHz-sec)} 2807 \begin{tabular}{lrrrr} 2808 \hline 2809 \hline 2810 & Random/Standard 2811 & Random/Minimal 2812 & Optimal/Standard 2813 & Optimal/Minimal \\ 2814 \hline 2815 network I/O (GB) & 182 & 80 & 44 & 30 \\ 2816 PS-1 & & & & \\ 2817 I/O (cpu-sec) & 3640 & 1600 & 880 & 600 \\ 2818 CPU (cpu-sec) & 4000 & 4000 & 4000 & 4000 \\ 2819 \# cpus & 64 & 47 & 41 & 38 \\ 2820 PS-4 & & & & \\ 2821 I/O (cpu-sec) & 1820 & 800 & 440 & 300 \\ 2822 CPU (cpu-sec) & 2000 & 2000 & 2000 & 2000 \\ 2823 \# cpus & 127 & 93 & 81 & 77 \\ 2303 \caption{SkyProbe Transparency Table (sample entries)\label{SkyprobeBVTable}} 2304 \begin{tabular}{lll} 2305 \hline 2306 \hline 2307 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2308 \hline 2309 Time & date/time & The time the SkyProbe image was taken. \\ 2310 Filter & string & Filter used for SkyProbe image. \\ 2311 Transparency & float & The derived transparency. \\ 2312 Number of stars & int & The number of stars used to measure the transparency. \\ 2313 Astrometry & coords & The astrometry used on the SkyProbe image. \\ 2314 Exposure time & float & The exposure time of the SkyProbe image. \\ 2315 Sky brightness & float & The measured sky (surface) brightness, counts / second \\ 2824 2316 \hline 2825 2317 \end{tabular} … … 2827 2319 \end{table} 2828 2320 2829 \begin{table} [b]2321 \begin{table} 2830 2322 \begin{center} 2831 \caption{ \label{NP4} Phase 4 load per major frame (7800 GHz-sec)}2832 \begin{tabular}{l rr}2833 \hline 2834 \hline 2835 & Standard 2836 & Minimal \\ 2837 \hline 2838 network I/O (GB) & 48 & 28\\2839 PS-1 & &\\2840 I/O (cpu-sec) & 960 & 557\\2841 CPU (cpu-sec) & 2600 & 2600\\2842 \# cpus & 30 & 26\\2843 PS-4 & &\\2844 I/O (cpu-sec) & 480 & 278\\2845 CPU (cpu-sec) & 1300 & 1300\\2846 \# cpus & 59 & 53\\2323 \caption{Skyprobe Line Absorption Table (sample entries)\label{SkyprobeATable}} 2324 \begin{tabular}{lll} 2325 \hline 2326 \hline 2327 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2328 \hline 2329 Time & date/time & The time the LRProbe observation was taken. \\ 2330 Disperser ID & string & ID of the dispersing element \\ 2331 Atm Component 1 & float & The strength of the 1st atmospheric component. \\ 2332 Atm Component 2 & float & The strength of the 2nd atmospheric component. \\ 2333 Atm Component 3 & float & The strength of the 3rd atmospheric component. \\ 2334 Disperser ID & string & ID of the dispersing element \\ 2335 Number of stars & int & Number of stars used to measure the absorptions. \\ 2336 Astrometry & coords & The astrometry used on the LRProbe image. \\ 2337 Exposure time & float & The exposure time of the LRProbe image. \\ 2338 Sky brightness & float & The measured sky (surface) brightness, in physical units. \\ 2847 2339 \hline 2848 2340 \end{tabular} … … 2850 2342 \end{table} 2851 2343 2344 \begin{table} 2345 \begin{center} 2346 \caption{Skyprobe Line Emission Table (sample entries)\label{SkyprobeETable}} 2347 \begin{tabular}{lll} 2348 \hline 2349 \hline 2350 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2351 \hline 2352 Time & date/time & The time the LRProbe observation was taken. \\ 2353 Disperser ID & string & ID of the dispersing element \\ 2354 Atm Component 1 & float & The strength of the 1st atmospheric component. \\ 2355 Atm Component 2 & float & The strength of the 2nd atmospheric component. \\ 2356 Atm Component 3 & float & The strength of the 3rd atmospheric component. \\ 2357 Continuum & float & The strength of the continuum emission. \\ 2358 Disperser ID & string & ID of the dispersing element \\ 2359 Exposure time & float & The exposure time of the LRProbe image. \\ 2360 \hline 2361 \end{tabular} 2362 \end{center} 2363 \end{table} 2364 2365 \begin{table} 2366 \begin{center} 2367 \caption{DIMM Measurements Table\label{DimmTable}} 2368 \begin{tabular}{lll} 2369 \hline 2370 \hline 2371 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2372 \hline 2373 Time & date/time & The time the DIMM observation was taken. \\ 2374 $\sigma_x$ & float & Raw dispersion in $x$. \\ 2375 $\sigma_y$ & float & Raw dispersion in $y$. \\ 2376 FWHM & float & Dervied seeing full width at half maximum. \\ 2377 RA & float & The coordinates of the measured star. \\ 2378 DEC & float & The coordinates of the measured star. \\ 2379 Exposure time & float & The exposure time of the DIMM observation. \\ 2380 Telescope ID & string & source of the DIMM data \\ 2381 \hline 2382 \end{tabular} 2383 \end{center} 2384 \end{table} 2385 2386 \begin{table} 2387 \begin{center} 2388 \caption{Near IR Wide-field Camera Results Table\label{NIR-Table}} 2389 \begin{tabular}{lll} 2390 \hline 2391 \hline 2392 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2393 \hline 2394 Time & date/time & The time the NIR observation was taken. \\ 2395 Sky brightness & float & The sky (surface) brightness in the NIR observation. \\ 2396 Sky variance & float & The variance in the sky (surface) brightness. \\ 2397 Astrometry & coords & The astrometry used on the NIR image. \\ 2398 FOV X & float & field width \\ 2399 FOV Y & float & field height \\ 2400 \hline 2401 \end{tabular} 2402 \end{center} 2403 \end{table} 2404 2405 \begin{table} 2406 \begin{center} 2407 \caption{Dome Status Table\label{DomeStatusTable}} 2408 \begin{tabular}{lll} 2409 \hline 2410 \hline 2411 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2412 \hline 2413 Time & date/time & The time for which the dome status is valid. \\ 2414 Azimuth & float & The azimuth of the dome. \\ 2415 Open status & boolean & Whether the dome is open or not. \\ 2416 Lights status & boolean & Whether lights are on in the dome or not. \\ 2417 Track status & boolean & Whether dome is tracking telescope or not. \\ 2418 \hline 2419 \end{tabular} 2420 \end{center} 2421 \end{table} 2422 2423 \begin{table} 2424 \begin{center} 2425 \caption{Telescope Status\label{TelescopeStatusTable}} 2426 \begin{tabular}{lll} 2427 \hline 2428 \hline 2429 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2430 \hline 2431 Time & date/time & The time for which the telescope status is valid. \\ 2432 Guide status & enum & The status of the guiding. \\ 2433 Altitude & float & The telescope altitude. \\ 2434 Azimuth & float & The telescope azimuth. \\ 2435 RA & float & The telescope Right Ascension (ICRS $\approx$ J2000). \\ 2436 Dec & float & The telescope Declination (ICRS $\approx$ J2000).\\ 2437 \hline 2438 \end{tabular} 2439 \end{center} 2440 \end{table} 2441 2442 \begin{table} 2443 \begin{center} 2444 \caption{Raw FPA Images\label{RawFPAs}} 2445 \begin{tabular}{lll} 2446 \hline 2447 \hline 2448 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2449 \hline 2450 ID & string & FPA image ID \\ 2451 RA & float & Coordinates of the boresight (i.e. telescope pointing). \\ 2452 DEC & float & Coordinates of the boresight (i.e. telescope pointing). \\ 2453 Filter & string & Filter used for the exposure. \\ 2454 Image Type & enum & image exposure type \\ 2455 Exposure time & float & Exposure time for the image. \\ 2456 Airmass & float & Airmass at which the image was taken. \\ 2457 ObsFrame ID & int & Observation frame identification number, ties FPAs into major frame \\ 2458 ObsGroup ID & int & Observation group identification number, ties FPAs into observing group \\ 2459 Observer & string & The name of the observer, or the version of the telescope scheduler software. \\ 2460 Program & string & The observing program being executed. \\ 2461 Nchips readout & int & Number of detector chips read out \\ 2462 Camera & string & Identification of camera source \\ 2463 Telescope & string & Telescope used for observation \\ 2464 Astrometry & coords & The astrometry used for the FPA. \\ 2465 Chip Metadata & string & metadata resource file \\ 2466 Cell Metadata & string & metadata resource file \\ 2467 \hline 2468 \end{tabular} 2469 \end{center} 2470 \end{table} 2471 2472 \begin{table} 2473 \begin{center} 2474 \caption{Pending Science Chips\label{PendingChips}} 2475 \begin{tabular}{lll} 2476 \hline 2477 \hline 2478 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2479 \hline 2480 FPA ID & string & FPA image ID \\ 2481 Chip ID & string & Chip identification number. \\ 2482 Proc Status & enum & Current Processing Status. \\ 2483 \hline 2484 \end{tabular} 2485 \end{center} 2486 \end{table} 2487 2488 \begin{table} 2489 \begin{center} 2490 \caption{Processed Science Chips\label{ProcessedChips}} 2491 \begin{tabular}{lll} 2492 \hline 2493 \hline 2494 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2495 \hline 2496 FPA ID & string & FPA Image ID \\ 2497 Chip ID & string & Chip identification number. \\ 2498 Status & enum & Current Processing Status. \\ 2499 Residual Stats & float & quality statistics. \\ 2500 \hline 2501 \end{tabular} 2502 \end{center} 2503 \end{table} 2504 2505 \begin{table} 2506 \begin{center} 2507 \caption{Observation Group Information\label{OBS}} 2508 \begin{tabular}{lll} 2509 \hline 2510 \hline 2511 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2512 \hline 2513 ObsGroup ID & string & Identification number for the observation group. \\ 2514 Number of images & string & Number of images in the observation group. \\ 2515 Type & string & Type of observation. \\ 2516 Status & string & Status of the observation group. \\ 2517 \tbd{etc} & \\ 2518 \hline 2519 \end{tabular} 2520 \end{center} 2521 \end{table} 2522 2523 \begin{table} 2524 \begin{center} 2525 \caption{Observation Frame Information\label{OBS}} 2526 \begin{tabular}{lll} 2527 \hline 2528 \hline 2529 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2530 \hline 2531 ObsFrame ID & string & Identification number for the observation frame. \\ 2532 Number of images & string & Number of images in the observation group. \\ 2533 Type & string & Type of observation. \\ 2534 Status & string & Status of the observation group. \\ 2535 \tbd{etc} & \\ 2536 \hline 2537 \end{tabular} 2538 \end{center} 2539 \end{table} 2540 2541 \begin{table} 2542 \begin{center} 2543 \caption{Science Processing Stats\label{PSStats}} 2544 \begin{tabular}{lll} 2545 \hline 2546 \hline 2547 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2548 \hline 2549 Chip ID & string & The chip identification number. \\ 2550 State & string & The state of the processing. \\ 2551 ObsFrame ID & string & The major frame the chip belongs to. \\ 2552 ObsGroup ID & string & The observation group the chip belongs to. \\ 2553 P1 astrom & string & The Phase 1 astrometry results file. \\ 2554 P2 astrom & string & The Phase 2 astrometry results file. \\ 2555 P3 astrom & string & The Phase 3 astrometry results file. \\ 2556 N guide stars & string & Number of guide stars used for the exposure. \\ 2557 Astrometry stats & string & Summary statistics for astrometry (number of stars, $sigma_x$, $sigma_y$) \\ 2558 Astrom catalog & string & The reference catalog that was used for the astrometry. \\ 2559 Bias method & string & Method used to correct the bias. \\ 2560 Bias stats & string & Summary statistics for bias \\ 2561 Flat-field image & string & The flat-field image that was applied. \\ 2562 Kernel data & & A description of the OT kernel. \\ 2563 Flat-field stats & & Summary statistics for flat-field (sigma of sky). \\ 2564 Mask image & string & The mask image that was applied. \\ 2565 Mask method & string & The algorithm used to mask the bad pixels. \\ 2566 Fringe images & string & The fringe model images that were used. \\ 2567 Fringe stats & & Summary statistics for fringes (fringe amplitude, sky sigma) \\ 2568 Object stats & & Summary statistics for object detection (number of objects, depth, other input parameters). \\ 2569 Photometry data & & photometry information: magnitude zero point and other corrections. \\ 2570 Photometry stats & & Summary statistics for the photometry (number of stars, $sigma_m$) \\ 2571 Photom catalog & string & The reference catalog that was used for the photometry. \\ 2572 PSF stats & & Summary statistics of the PSF. \\ 2573 Software ver & string & Versions of each of the modules used in the processing. \\ 2574 \hline 2575 \end{tabular} 2576 \end{center} 2577 \end{table} 2578 2579 \begin{table} 2580 \begin{center} 2581 \caption{Chip / Sky overlaps\label{overlaps}} 2582 \begin{tabular}{lll} 2583 \hline 2584 \hline 2585 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2586 \hline 2587 Chip ID & string & The identification number of the chip. \\ 2588 Sky Cell ID & string & The identification number of the sky cell. \\ 2589 State & string & Processing state of overlap \\ 2590 \hline 2591 \end{tabular} 2592 \end{center} 2593 \end{table} 2594 2595 \begin{table} 2596 \begin{center} 2597 \caption{Processed Sky-Cell stats\label{}} 2598 \begin{tabular}{lll} 2599 \hline 2600 \hline 2601 {\bf Column Name} & {\bf Datatype } & {\bf Description} \\ 2602 \hline 2603 Input Chips & string & Identification numbers of the chips used to produce the sky cell. \\ 2604 PSF adjustments & string & \tbd{Adjustments to the PSF.} \\ 2605 CR rejection stats & string & Statistics from the CR rejection (number of CRs, distribution, limiting flux). \\ 2606 Image comb params & string & Parameters used for the image combination. \\ 2607 Diff image params & string & Parameters used for the image differencing. \\ 2608 Average weight & string & The weight of the reference image \\ 2609 P4D object stats & string & Summary statistics of the object detection (number of objects, depth, other input parameters). \\ 2610 P4S object stats & string & Summary statistics of the object detection (number of objects, depth, other input parameters). \\ 2611 Software versions & string & Software versions of modules used in the sky cell processing. \\ 2612 Processing stats & string & Summary statistics of the processing (CPU time, etc). \\ 2613 \hline 2614 \end{tabular} 2615 \end{center} 2616 \end{table} 2617 \clearpage 2852 2618 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2853 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2854 2855 \subsubsection{Conclusions} 2856 2857 Table~\ref{throughput} presents one way of analysing the hardware 2858 requirements, making a specific set of assumptions about the number of 2859 nodes for the two phases and the expected network and disk 2860 bandwidths. The important conclusion in this analysis is the implied 2861 number of GHz per processor, given the assumptions laid out. 2862 Phase 2 is specified to have 240 detector nodes, while Phase 4 is specified 2863 to have roughly 60 static sky nodes. The range of Phase 2 CPU 2864 requirements implies that each CPU needs to have speeds in the range 2865 of 2.0 - 3.6 GHz, which sound very plausible for the year 2007, since 2866 these apply to PS-4. 2867 2868 Another way to represent this information is to use the total number 2869 of MB I/O and the total number of GHz-sec required for the two stages, 2870 confront these with an assumption for the bandwidth per network 2871 adapter and an assumption for the CPU speed and use those numbers to 2872 calculate the minimum number of nodes (CPUs) needed to sustain the 2873 timing requirements. There are quite a few parameters and options to 2874 choose from. We have assumed that for PS-1, the time between major 2875 frames (4 images combined in Phase 4) is 120 seconds, and 30 seconds 2876 for PS-4. We have also assumed that each CPU has one network adapter 2877 associated with it, and use the numbers of 50 MB/sec for PS-1 era 2878 network adapters and 100 MB/sec for the PS-4 network adapters (since 2879 there has been some steady improvement in GigE hardware over the past 2880 year). We have also assumed each PS-1 CPU is rated at 3 GHz and those 2881 for PS-4 are rated at 6 GHz (somewhat conservative since 3 GHz 2882 machines are already available). Tables~\ref{NP2} and \ref{NP4} show 2883 the load and resulting number of nodes for both Phase 2 and Phase 4 2884 for both the PS-1 and PS-4 assumptions, using the I/O numbers for all 2885 of the scenarios above. Note that in these discussions, we make the 2886 idealized assumption that the computational and I/O portions of each 2887 process are completely serial. As a result, the CPU is completely 2888 used to perform the I/O during the I/O phase, avoiding any concern 2889 about I/O load on the processor during analysis. 2890 2891 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2892 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2893 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2894 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 2619 2620 \subsection{Software Runtime Configuration Issues} 2621 2622 The IPP Software requires extensive runtime configuration information. 2623 This includes default parameters for analysis to be performed, 2624 descriptions of how a particular analysis is performed, locations of 2625 data sources, and so forth. The IPP may store this information in the 2626 Metadata Database or in configuration files available to the user. 2627 Both methods are implemented in the current design. In either method, 2628 the necessary parameters are identical. In this section, we discuss 2629 the contents of specific portions of the runtime configuration. 2630 2631 \subsubsection{Camera Definition Information} 2632 2633 Every camera which may be analysed by the IPP has differences in how 2634 the data is represented. The IPP is built with the flexibility to 2635 handle data from many different cameras, not just the Pan-STARRS 2636 Gigapix cameras. This is partly to allow testing of the analysis 2637 system on data from other telescopes, such as MegaPrime on CFHT and 2638 Suprime on Subaru, but also to allow us to adapt to changes in the 2639 design of the Gigapix cameras themselves. It also means the IPP 2640 software may be used by astronomers for other analysis projects beyond 2641 the IPP. 2642 2643 Most cameras provide extensive descriptive information in the FITS 2644 image headers when the images are read out. Typically, the location 2645 and orientations of the individual detectors are defined by keywords 2646 such as DATASEC and DETSEC. Other variations on these words are used 2647 for cameras which place the pixels from multiple amplifiers in the 2648 same FITS data segment. Other parameters, such as astrometric 2649 information or exposure times, are stored in headers as well. It is 2650 possible to use these header keywords to guide the analysis software, 2651 but there are two difficulties. 2652 2653 First, it is very common for different keywords to be used by 2654 different cameras, sometimes even the same camera may use different 2655 keywords for the same information at different times (major readout 2656 software upgrades, for example, can be accompanied by keyword 2657 revisions). In addition, within Pan-STARRS and the IPP, we would like 2658 the capability to refer to the Metadata database as the authoratative 2659 sources of some of these entries rather than the image headers. Given 2660 this circumstance, it is at least necessary to define the appropriate 2661 source for a given data concept appropriate to data from a specific 2662 camera. 2663 2664 The second problem arises when actually performing an analysis. In 2665 many circumstances, the software needs to know what data to expect 2666 even when an appropriate camera image is not available. This is 2667 particularly true for a camera which is composed of multiple chips and 2668 multiple amplifiers. It is a frequent circumstance than some subset 2669 of the chips or amplifiers will either be unavailable or are invalid 2670 for one reason or another. It is important for the software to have a 2671 guide for what data should be available from a perfect readout of the 2672 given camera so decisions can be made how to handle data which is not 2673 complete. This is also important to validate that a particular 2674 dataset, which appears to be from a known camera, actually corresponds 2675 to that camera and has all of the necessary information where 2676 expected. 2677 2678 In order to facilitate the operation of the IPP with a variety of 2679 cameras, and to allow the software the flexibility to change the 2680 camera defintion dynamically, we define a collection of software 2681 runtime configuration information which defines a given camera. This 2682 information is represented below in the form of the PSLib Metadata 2683 Config file, but may be stored in the Metadata Database or in an 2684 alternate format as appropriate. 2685 2686 We start by noting that the a single camera is represented as a Focal 2687 Plane Array (FPA), divided into Chips, divided into Cells. For a 2688 single FPA, all imaging data is stored in a FITS file or a collection 2689 of FITS files. Software needs to know where in a given file or set of 2690 files to find a particular Cell, what Cells to expect, what chips to 2691 expect, and the relationships between those entities, etc. 2692 2693 A single camera configuration file (or dataset) represents the 2694 description of a complete FPA. In the configuration file, any 2695 parameters which are specific to the complete FPA are placed on their 2696 own lines. These include the definition of the keywords or database 2697 locations. An incomplete example is given below. 2698 2699 \begin{verbatim} 2700 NCELL S32 NN 2701 NCHIP S32 NN 2702 EXPTIME-SRC STR HD:EXPTIME # need to specify PHU vs EXTNAME? 2703 EXPTIME-KEY STR EXPTIME 2704 DATE-KEY STR DATE-OBS 2705 DATE-FMT STR YYYY/MM/DD 2706 2707 TYPE CELL FILENAME EXTNAME CHIP DATASEC BIASSEC 2708 CELL.nn CELL @ROOT@CELL AMP00 CHIP.00 CF:[0,0:0,0] HD:BIASSEC 2709 CELL.01 CELL @ID/@ID@CELL.fits AMP01 CHIP.00 DB:??? 2710 \end{verbatim} 2711 2712 \subsubsection{Analysis Recipe Information} 2713 2714 In order to maintain flexibility in the analysis details, the IPP uses 2715 recipes to define how a particular analysis is implemented. Each 2716 major analysis script (eg, Phase 2) has its own recipe configuration 2717 information, which may be stored in the Metadata Database or in the 2718 form of the PSLib Metadata Config file. This configuration 2719 information includes all of the user configurable parameters. Many of 2720 these may specify a specific value, or they may specify lookup methods 2721 (database locations, or header locations). The specifies of each 2722 depends on the context. Below, we provide an example recipe file for 2723 the bias subtraction portion of Phase 2, giving several alternative 2724 options for certain entries. Note that, for example, the overscan 2725 subtraction may be specified as using a particular region given in the 2726 recipe file, or on the basis of a particular header keyword. 2727 2728 \begin{verbatim} 2729 # BIAS: 2730 BIAS.IMAGE STR NONE 2731 BIAS.IMAGE STR FILE:bias.fits 2732 BIAS.IMAGE STR DB:BEST 2733 BIAS.IMAGE STR DB:CLOSE 2734 2735 BIAS.OVERSCAN STR HD:BIASSEC 2736 BIAS.OVERSCAN STR CF:[0,16:0,2048] 2737 BIAS.OVERSCAN STR NONE 2738 2739 BIAS.OVERSCAN.STATS STR MEDIAN 2740 BIAS.OVERSCAN.STATS STR MEAN 2741 2742 BIAS.OVERSCAN.FIT STR SPLINE 2743 BIAS.OVERSCAN.FIT.NPTS S32 5 2744 2745 BIAS.OVERSCAN.FIT STR POLYNOMIAL 2746 BIAS.OVERSCAN.FIT.ORDER S32 3 2747 BIAS.OVERSCAN.FIT.NBIN S32 5 2748 \end{verbatim} 2749 2750 \subsection{I/O Code Autogeneration} 2751 2752 Within IPP, we have a number of data collections which have multiple 2753 representations. We define a tool to automatically generate code to 2754 provide I/O APIs to read and write these data and data structures to 2755 carry them within program. Within the IPP, we will use database 2756 tables (ie, in the Metadata Database), FITS Tables (to exchange bulk 2757 data), and XML (to exchange more complete datasets). 2758 2759 I/O API Autocode template (example.def): 2760 \begin{verbatim} 2761 Name Example 2762 Table EXAMPLE 2763 EXTNAME EXAMPLE 2764 2765 KEY XVALUE 2766 2767 # name format unit comment 2768 XVALUE F32 pixels "x coordinate" 2769 BINNING S32 fraction "binning factor" 2770 NAME STR[32] string "description of entry" 2771 \end{verbatim} 2772 2773 Running autocode on such a file would generate an output header and C 2774 files \code{example.h, example.c} with the following structure and APIs: 2775 2776 \begin{verbatim} 2777 typedef struct { 2778 psF32 XVALUE; // x coordinate 2779 psS32 BINNING; // binning factor 2780 char NAME[32]; // description of entry 2781 } Example; 2782 2783 psMetadata *psFITSTableInitExample (); 2784 psExample *psFITSTableLoadExample (char *filename, int *Nrows); 2785 bool psFITSTableSaveExample (char *filename); 2786 2787 psMetadata *psDatabaseTableInitExample (); 2788 psExample *psDatabaseTableLoadExample (char *filename, int *Nrows); 2789 bool psDatabaseTableSaveExample (char *filename); 2790 psExample *psDatabaseTableLoadExampleRow (char *filename, psF32 XVALUE); 2791 \end{verbatim} 2792 2793 \bibliographystyle{plain} 2794 \bibliography{panstarrs} 2795 2796 \end{document} 2797 2798 2895 2799 2896 2800 \section{Notes} 2897 2898 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2899 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2900 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2901 2801 2902 2802 \subsection{Cell vs Chip vs FPA vs Major Frame} … … 2913 2813 possibilities: 2914 2814 2915 \begin{ enumerate}2815 \begin{itemize} 2916 2816 \item exposures in a major frame are always synchronized; the 2917 2817 telescopes are required to take exposures in a coordinated fashion and … … 2936 2836 coincident) than a major frame in which the offsets are larger in 2937 2837 either dimension. 2938 \end{ enumerate}2838 \end{itemize} 2939 2839 2940 2840 A decisions between these possibilities will drive some requirements 2941 2841 either on the IPP side or on the PTS/TCS side. 2942 2943 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2944 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2945 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2946 2842 2947 2843 \subsection{Identifying ghosts, spikes, etc} … … 2957 2853 addition of data. 2958 2854 2959 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2960 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2961 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2962 2963 2855 \subsection{Pending Sky-cell / Detector table} 2964 2856 … … 2967 2859 initiate phase 4. 2968 2860 2969 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2970 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2971 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2972 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%2973 2974 \section{Appendices}2975 2976 \subsection{Image Server Database Tables}2977 2978 \begin{table}2979 \begin{center}2980 \caption{Storage Object Table Contents\label{ImageServerTables:SO}}2981 \begin{tabular}{ll}2982 \hline2983 \hline2984 {\bf Column Name} & {\bf Datatype} & {\bf Description} \\2985 \hline2986 \code{so_id} & integer & internal storage object identifier \\2987 \code{ext_id} & string & external storage object identifier (file ID) \\2988 \code{comment} & string & user description of object \\2989 \code{epoch} & time/date & last date of access \\2990 \hline2991 \end{tabular}2992 \end{center}2993 \end{table}2994 2995 \begin{table}2996 \begin{center}2997 \caption{Instance Table Contents\label{ImageServerTables:INT}}2998 \begin{tabular}{ll}2999 \hline3000 \hline3001 {\bf Column Name} & {\bf Datatype} & {\bf Description} \\3002 \hline3003 \code{ins_id} & integer & internal instance identifier \\3004 \code{so_id} & integer & key to storage object table \\3005 \code{uri} & string & location in hardware collection \\3006 \code{sha1sum} & string & checksum information \\3007 \code{assigned_location} & boolean & is location user-specified? \\3008 \code{epoch} & time/date & last date of access \\3009 \code{atime} & time/date & last date of access \\3010 \hline3011 \end{tabular}3012 \end{center}3013 \end{table}3014 3015 \begin{table}3016 \begin{center}3017 \caption{Volume Table Contents\label{ImageServerTables:VOL}}3018 \begin{tabular}{ll}3019 \hline3020 \hline3021 {\bf Column Name} & {\bf Datatype} & {\bf Description} \\3022 \hline3023 \code{vol_id} & integer & internal volume identifier \\3024 \code{uri} & string & node name? \\3025 \hline3026 \end{tabular}3027 \end{center}3028 \end{table}3029 3030 \bibliographystyle{plain}3031 \bibliography{panstarrs}3032 \end{document}3033 3034 %%%%%% Phase 0 has been dropped: identifying the moving objects is not needed3035 3036 \paragraph{Phase 0 : night preparation}3037 3038 Phase 0 is the night preparation phase of the IPP analysis system.3039 There may be potentially many pieces of information which apply to the3040 processing for an entire night and which take substantial time to3041 calculate. these are pre-calculated by the phase 0 stage and stored3042 in a database table for reference by other stages of the processing3043 system. Currently, the only quantity calculated by Phase 0 is the3044 collection of known moving object ephemerids.3045 3046 At various stages in the IPP analysis, it is necessary to know the3047 location of known moving objects (main belt asteroids, comets,3048 Kuiper-belt objects, any other classes of asteroids) in relation to3049 specific images obtained. If moving object orbits were trivial to3050 calculate, or if the number was limited, this would be a simple3051 problem of three dimensional intersections. However, complete orbits3052 are not trivial and there may be tens of thousands to millions of3053 possible objects of interest. To simplify the task, it is possible to3054 reduce the parameter space of the search by pre-calculating the orbit3055 segments of all objects for a given night and saving fiducial points3056 of the orbit in a database table. Later systems which require the3057 position of objects in a specific image can use linear interpolation3058 between these fiducial points to identify the likely objects, and3059 potentially additional non-linear orbital calculations to refine the3060 positions.3061 3062 The database table of object fiducial positions must include the3063 following information:3064 3065 \begin{itemize}3066 \item object ID3067 \item epoch3068 \item RA at epoch3069 \item DEC at epoch3070 \item dRA at epoch3071 \item dDEC at epoch3072 \item R magnitude?3073 \item date of calculation?3074 \item lifetime?3075 \end{itemize}3076 3077 The input for this calculation is the table of known moving objects3078 and their orbital elements, and the time range for the calculation.3079 If the calculation is slow, Phase 0 could be paralellized by object.3080 If Phase 0 is fast enough (\tbd{minutes?}), the process need not be3081 parallel. The {\tt lifetime} and {\tt date of calculation} allow old3082 Phase 0 entries to be removed when they are not needed. \tbd{This3083 cleaning phase could be a function of Phase 0.} Phase 0 need not be3084 run only for the current night. Any time a specific set of data is to3085 be analysed by the later stages, phase 0 should be run for the3086 appropriate time period. \tbd{Does there need to be a database table3087 with phase 0 runs and time periods defined? this could be the3088 reference used by later phases to decide if phase 0 has been run. they3089 could also trigger the phase 0 run if they notice it has not been run3090 (a job of the scheduler).}3091 3092 \tbd{what is the orbit calculation speed? does it scale with Npts?3093 what is the number of known objects now? in 5 years?}3094 3095 3096 3097 %%% phase 2 metadata3098 \milsection{Metadata}3099 3100 The following metadata associated with the images are required for3101 Phase~2 operation:3102 \begin{itemize}3103 \item The orthogonal transfer (OT) image shifts made during the3104 exposure --- in order to create a convolution kernel;3105 \item Time of observation --- for selecting the appropriate detrend3106 images;3107 \item Filter --- for selecting the appropriate detrend images;3108 \item Telescope identification --- for selecting the appropriate3109 detrend images;3110 \item Exposure time --- for the photometric calibration;3111 \item Detector gain --- for calculating photometric errors and3112 determining the quality of the overscan;3113 \item Detector read noise --- for calculating photometric errors and3114 determining the quality of the overscan;3115 \end{itemize}3116 3117 \milsection{Pixel Masks}3118 \label{ap:masks}3119 3120 This section describes the requirements on Bad Pixel Masks (BPMs).3121 These will consist in of bit masks for each pixel. For Phase 2, flags3122 are required for at least each of the following pixel attributes:3123 \begin{enumerate}3124 \item The pixel is a charge trap;3125 \item The pixel is a bad column;3126 \item The pixel is saturated in the A/D converter;3127 \item The pixel is non-positive in the flat-field;3128 \item The pixel is part of a row that has excess noise; and3129 \item The pixel is determined to be a cosmic ray, based on its3130 morphology.3131 \end{enumerate}3132 3133 Of these, only masks for the charge traps need to be grown by the3134 extent of the OT convolution kernel. For other pixel types,3135 orthogonal transfer of the flux in this pixel will not (necessarily)3136 affect the flux in neighbouring pixels3137 3138 \milsection{Object Catalogs}3139 \label{ap:catalogs}3140 3141 Object catalogs from Phase 2 shall consist of at least the3142 following elements for each object:3143 \begin{enumerate}3144 \item Object centre, with corresponding errors;3145 \item Object magnitude, with corresponding error;3146 \item Object isophotal magnitude, with corresponding error;3147 \item Object FWHM;3148 \item Object elliptical axis lengths; and3149 \item Object position angle for ellipse.3150 \end{enumerate}3151 3152 Though further details may be required for catalogs in Phase~4,3153 the above details are minimum requirements for Phase~2 catalogs.3154
Note:
See TracChangeset
for help on using the changeset viewer.
