Index: /trunk/doc/.cvsignore =================================================================== --- /trunk/doc/.cvsignore (revision 6007) +++ /trunk/doc/.cvsignore (revision 6008) @@ -1,2 +1,1 @@ *.log *.dvi *.aux *.toc *.log *.out -PSDC Index: /trunk/doc/design/.cvsignore =================================================================== --- /trunk/doc/design/.cvsignore (revision 6007) +++ /trunk/doc/design/.cvsignore (revision 6008) @@ -1,14 +1,4 @@ -*.log *.dvi *.aux *.toc *.log *.out *.lof *.tbr *.tbd *.ent -ippSCDps1.pdf -ippSRS.pdf -ippSDRS.pdf -hardware.pdf +*.log *.dvi *.aux *.toc *.log *.out *.lof *.tbr *.tbd *.ent *.pdf notes old -.mana -.status -.dvo -sim ippSRSout.tex ippSRStrace.tex -pdfs -ippSCDps1.pdf Index: unk/doc/design/datasys_cost.tex =================================================================== --- /trunk/doc/design/datasys_cost.tex (revision 6007) +++ (revision ) @@ -1,344 +1,0 @@ -% Copyright (C) 2004 Joshua Hoblitt -% -% $Id: datasys_cost.tex,v 1.12 2004-11-21 23:36:28 jhoblitt Exp $ - -\documentclass[panstarrs,psreport]{panstarrs} -%\usepackage{array} - -%\documentclass[panstarrs]{panstarrs} - -% basic document variables -\title{Pan-STARRS PS1 Data Systems Cost Estimate} -\author{Joshua Hoblitt} -\shorttitle{PS1 Datasys Cost} -\group{Pan-STARRS Algorithm Group} -\project{Pan-STARRS Image Processing Pipeline} -\organization{Institute for Astronomy} -\version{02} -\docnumber{PSDC-xxx-xxx} -% note the use of the docnumber & version number: -% the complete PSDC document number is given by -% \thedocnumber-\theversion - -\begin{document} -\maketitle - -% -- Revision History -- -% provide explicit values for the old versions -% use '\theversion' for the current version (set above) -% use \hline between each table row -\RevisionsStart -% version Date Description -01 & 2004.03.08 & First draft \\ -02 & 2004.03.09 & Add power consumption\\ - & & Tables more consistent\\ -\hline -\RevisionsEnd - -\pagebreak -\tableofcontents - -\pagebreak -\listoffigures - -\pagebreak -\pagenumbering{arabic} - -\section{Scope} - -\subsection{Identification} - -This document establishes the estimated cost of the network and computing -hardware necessary to support PS1. Due to the rapid rate of change in the IT -industry the part numbers specified in this document may change prior to the -actual purchase of equipment but the cost structure is expected to remain -stable. - -\section{Image Processing Pipeline and Data Storage Equipment} - -\subsection{Requirements} - -\subsubsection{CPU} - -Phase 2 and phase 4 dominate the IPP computational processing needs. Assuming -3.0GHz CPUs we need 40 CPUs for phase 2 and 30 CPUs for phase 4. - -\subsubsection{Storage} - -The storage needs for the first year of PS1's operation have been estimated to -be between 700-1100TB. Assuming that 0.5TB SATA disks become available in the -next year we will require between 2000-3000 units. - -\subsection{Hardware} - -A 5u rackmount case has been identified that can support 24 hot-swappable SATA -disks, 1 internal disk, and a dual CPU motherboard. The large data storage -capacity requirement is the driver for the number of complete systems we will -have to acquire. - -We will need between 85 and 132 nodes to support our data requirements. - -\begin{center} -\begin{tabular}{lrrr} -% \begin{tabular}{>{$}l<{$}>{$}r>{$}r>{$}r} -Item &Estimated Price&85 nodes&132 nodes\\ -\cline{1-4} -24 x 0.5TB SATA disks (bulk storage) &\$7,200 &\$61,200 &\$950,400\\ -1 x 0.5TB SATA disk (OS + software) &\$300 &\$25,500 &\$39,600\\ -2 x 3.0GHz CPUs &\$2,000 &\$170,000 &\$264,000\\ -2 x 12-port SATA raid controller &\$1,500 &\$127,500 &\$198,000\\ -1 x Motherboard &\$500 &\$42,500 &\$66,000\\ -4 x 1GB RAM DIMM &\$1,400 &\$119,000 &\$184,800\\ -1 x 5u Rackmount Chassis &\$2,800 &\$238,000 &\$369,600\\ -\cline{1-4} -Total &$\sim$\$16,000&$\sim$\$1,360,000&$\sim$\$2,112,000\\ -\end{tabular} -\end{center} - -\subsubsection{Spares} - -\tbd{TBD} - -\subsubsection{Maintenance} - -\tbd{TBD} - -\section{Network Equipment} - -\subsection{Requirements} - -\subsubsection{Bandwidth} - -It has been estimated that we need a throughput of $\sim604Mb/s$ to support PS1 -at a 30s cadence. A OC-12c Packet over SONET connection can provide enough -bandwidth. - -\subsubsection{Switch ports} - -We will needs to support 32 x Summit Pixel Servers and up to 16 x -Servers/Workstations at the summit. The data center switch should be able to -support up to 192 computation nodes and control computers. - -\subsection{Network Link} - -An OC-12c PoS link from the summit of Haleakala to Kihei on Maui will cost -\$16,970 per month with a 3-year contract or \$16,460 per month with a 5-year -contract. Setup fees will vary by location and are estimated to be in the -range of \$10,000 - \$100,000 per end point (\$20K - \$200K total). - -\subsection{Hardware} - -\subsubsection{Summit Core Switch/Router} - -\begin{center} -Cisco 6513 -\begin{tabular}{lr} -Item &Estimated Price\\ -\cline{1-2} -1 x WS-C6513 Cat 6513 &\$15,250\\ -1 x WS-X6K-S720-6408A &\$33,000\\ -1 x WS-X6548-RJ-45 &\$17,995\\ -1 x WS-C6K-13SLT-FAN2 &\$745\\ -2 x WS-CAC-4000W-US &\$10,000\\ -\cline{1-2} -Total &$\sim$\$76,990\\ -\end{tabular} -\end{center} - -\subsubsection{IPP Core Switch/Router} - -\begin{center} -Cisco 6513 -\begin{tabular}{lr} -Item &Estimated Price\\ -\cline{1-2} -1 x WS-C6513 Cat 6513 &\$15,250\\ -1 x WS-X6K-S720-6408A &\$33,000\\ -4 x WS-X6548-RJ-45 &\$71,980\\ -1 x WS-C6K-13SLT-FAN2 &\$745\\ -2 x WS-CAC-4000W-US &\$10,000\\ -\cline{1-2} -Total &$\sim$\$130,975\\ -\end{tabular} -\end{center} - -\subsubsection{Spares} - -\tbd{TBD} - -\subsubsection{Maintenance} - -\tbd{Cisco SmartNet contracts are typically $\sim5\%$ of the purchase price.} - -\section{Object Database Equipment} - -\subsection{Requirements} - -\tbd{TBD} - -\subsection{Hardware} - -\tbd{TBD} - -\subsubsection{Spares} - -\tbd{TBD} - -\subsubsection{Maintenance} - -\tbd{TBD} - -\section{Moving Object Pipeline Equipment} - -\subsection{Requirements} - -\tbd{TBD} - -\subsection{Hardware} - -\tbd{TBD} - -\subsubsection{Spares} - -\tbd{TBD} - -\subsubsection{Maintenance} - -\tbd{TBD} - -\section{Software} - -\subsection{Lustre} - -\subsubsection{Maintenance} - -ClusterFS, Inc. has given an estimate of \$64,000 per year for a 128 node -cluster. - - -\section{Power Consumption} - -\subsection{Summit Data Systems} - -\begin{center} -\begin{tabular}{lrrr} -Item &Count &Per Unit &Power Consumption\\ -\cline{1-4} -Pixel Server Node &16 &600w &9,600w\\ -Ethernet Switches &4 &540w(high) &2,160w\\ -Core Switch/Router &1 &2,000w &2,000w\\ -OATS Database Server &2 &600w &1,200w\\ -Desktop Computers &3 &230w &690w\\ -OATS Server &1 &600w &600w\\ -PTS Server &1 &600w &600w\\ -20.1" LCD monitors &6 &65w &390w\\ -Managed Power Strips &15 &10w &150w\\ -16-port KVM &1 &50w &50w\\ -32-port Console Server &1 &17w &17w\\ -\cline{1-4} -SubTotal &&&17,457w\\ -UPS &6% loss &&1,047w\\ -\cline{1-4} -Total &&&18,504\\ -\end{tabular} -\end{center} - -\subsection{IPP Data Systems} - -The Pan-STARRS Prototype IPP Computational/Storage Node draws $\sim600w$ when -idle. As this is the idle load and the power consumption of computer -components is rapidly increasing we should budget for $\sim750w$ per node. - -\begin{center} -85 nodes -\begin{tabular}{lr} -Item &Estimated Power Consumption\\ -\cline{1-2} -Nodes &63,750w\\ -Network + Misc. &25,000w\\ -\cline{1-2} -Total &88,750w\\ -\end{tabular} -\end{center} - -\begin{center} -132 nodes -\begin{tabular}{lr} -Item &Estimated Power Consumption\\ -\cline{1-2} -Nodes &99,000w\\ -Network + Misc. &25,000w\\ -\cline{1-2} -Total &124,000w\\ -\end{tabular} -\end{center} - -\subsection{Air Conditioning} - -It takes 1ton of A/C to remove 12kBTU of heat. - -\begin{center} -\begin{tabular}{lr} -Heat Generated &A/C Load\\ -\cline{1-2} -88,750w &25.242tons\\ -124,000w &35.268tons\\ -\end{tabular} -\end{center} - -\subsection{Total Power Consumption} - -Assuming that 1ton of A/C requires 1kw of power. - -\begin{center} -\begin{tabular}{lrr} -Item &85 nodes &132 nodes\\ -\cline{1-3} -Nodes + Net + Misc. &88,750w &124,000w\\ -A/C &25,242w &35,268w\\ -\cline{1-3} -Total &113,992w &159,268w\\ -\end{tabular} -\end{center} - -\subsection{Cost Estimate} - -This Billing estimate is from Maui Electric Company, Ltd. (MECO) for schedule -"J" service with a power factor of 0.9 (MECO refers to this as a power factor -of 90\%) based on Feburary, 2004 rates. - -\begin{center} -\begin{tabular}{lr} -Peak Demand &Estimated Price\\ -\cline{1-2} -115kw &\$13,000 per month\\ -160kw &\$18,100 per month\\ -\end{tabular} -\end{center} - - -\section{Misc} - -\subsection{Hardware} - -\begin{center} -\begin{tabular}{lr} -Item &Estimated Price\\ -\cline{1-2} -Racks/cabinets &\$50,000\\ -Cabling &\$25,000\\ -Tools &\$10,000\\ -\cline{1-2} -Total &$\sim$\$85,000\\ -\end{tabular} -\end{center} - -\subsection{Electrical Wiring} - -Each cabinet will require 4 x 220v/15amp single phase circuits. -\tbd{Material/labor costs are TBD}. - -\bibliographystyle{plain} -\bibliography{panstarrs} - -\end{document} Index: unk/doc/design/hardware-org.fig =================================================================== --- /trunk/doc/design/hardware-org.fig (revision 6007) +++ (revision ) @@ -1,61 +1,0 @@ -#FIG 3.2 -Landscape -Center -Inches -Letter -100.00 -Single --2 -1200 2 -6 6075 2175 9225 5100 -6 6375 2925 8925 4575 -2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 - 7050 4500 6450 4500 6450 3000 7050 3000 7050 4500 -2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 - 7950 4500 7350 4500 7350 3000 7950 3000 7950 4500 -2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 - 8850 4500 8250 4500 8250 3000 8850 3000 8850 4500 --6 -2 2 0 2 0 7 50 0 45 0.000 0 0 -1 0 0 5 - 6150 2250 9150 2250 9150 2700 6150 2700 6150 2250 -2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 - 6750 2700 6750 3000 -2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 - 7650 2700 7650 3000 -2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 - 8550 2700 8550 3000 -4 0 0 50 0 16 24 0.0000 4 360 2730 6375 5025 Static Sky nodes\001 --6 -2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 - 1500 4500 900 4500 900 3000 1500 3000 1500 4500 -2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 - 2400 4500 1800 4500 1800 3000 2400 3000 2400 4500 -2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 - 3300 4500 2700 4500 2700 3000 3300 3000 3300 4500 -2 4 0 2 0 7 50 0 -1 0.000 0 0 7 0 0 5 - 4200 4500 3600 4500 3600 3000 4200 3000 4200 4500 -2 2 0 2 0 7 50 0 45 0.000 0 0 -1 0 0 5 - 750 2250 4350 2250 4350 2700 750 2700 750 2250 -2 1 0 3 0 7 50 0 -1 0.000 0 0 -1 0 0 2 - 4350 2475 6150 2475 -2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 - 1200 2700 1200 3000 -2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 - 2100 2700 2100 3000 -2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 - 3000 2700 3000 3000 -2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 - 3900 2700 3900 3000 -2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 0 0 2 - 1200 2250 1200 975 -2 1 0 1 0 7 50 0 -1 0.000 0 0 -1 1 1 2 - 0 0 1.00 60.00 120.00 - 0 0 1.00 60.00 120.00 - 1200 975 1200 600 -4 0 0 50 0 16 24 0.0000 4 285 1890 1050 2625 OTA switch\001 -4 0 0 50 0 16 24 0.0000 4 285 2190 1425 1050 connection to\001 -4 0 0 50 0 16 24 0.0000 4 360 3210 1425 1485 observatory system\001 -4 0 0 50 0 16 24 0.0000 4 285 1845 1650 5025 OTA nodes\001 -4 0 0 50 0 16 24 0.0000 4 360 2655 6300 2625 static sky switch\001 -4 0 0 50 0 16 18 0.0000 4 210 1335 4575 2400 interswitch\001 -4 0 0 50 0 16 18 0.0000 4 210 405 5025 2775 link\001 Index: unk/doc/design/hardware.tex =================================================================== --- /trunk/doc/design/hardware.tex (revision 6007) +++ (revision ) @@ -1,1251 +1,0 @@ -%%% $Id: hardware.tex,v 1.3 2004-10-29 22:00:08 eugene Exp $ -\documentclass[panstarrs,psreport]{panstarrs} - -% basic document variables -\title{Pan-STARRS Image Processing Pipeline} -\subtitle{Computation Challenge} -\shorttitle{IPP Computational Challenge} -\author{Eugene Magnier, Josh Hoblitt} -\audience{Pan-STARRS PMO} -\group{Pan-STARRS Algorithm Group} -\project{Pan-STARRS Image Processing Pipeline} -\organization{Institute for Astronomy} -\version{00} -\docnumber{PSDC-400-006} - -\begin{document} -\maketitle - -% -- Revision History -- -\RevisionsStart -% version Date Description -DR.00 & 2004/01/30 & Proposal \\ \hline -DR.01 & 2004/02/19 & Fleshed-out, Cleaned-up, Reorganized \\ \hline -DR.02 & 2004/03/15 & Merged with hardware cost doc (JH) \\ \hline -00 & 2004/10/29 & Cleaned up, added to PSDC \\ \hline -\RevisionsEnd - -\pagebreak -\tableofcontents - -\pagebreak -\listoffigures - -\pagebreak -\pagenumbering{arabic} -\section{Overview} - -This document discusses the likely range of the Pan-STARRS Image -Processing Pipeline (IPP) hardware requirements. The hardware -requirements addressed in this document consist of: - -\begin{itemize} -\item Total Disk Volume -\item Total Processing Power -\item Sustained Switch Bandwidth -\item Sustained Node Network I/O -\item Sustained Disk I/O -\end{itemize} - -Even without the complete IPP design, it is possible to identify the -major drivers on the hardware requirements. The total disk volume -requirements are dominated by the need to store raw images for a -certain period, the need to store calibration images for a longer -period, and the need to store the static sky images. Of the various -analysis pipelines, and depending on the data organization as -discussed below, Phase 2 and Phase 4 present the most significant -demands in terms of data I/O throughput on the network. Phase 2 and -Phase 4 also present the most significant CPU demands. In this -discusion, Phase 2 refers to the per-OTA image pre-processing in which -the instrumental signature is removed and a first pass object -detection is performed. Phase 4 refers to the multiple OTA -combination in which the pre-processed images are merged and combined, -in both addition and subtraction, with the static sky image, and up to -three object detection passes are performed. - -This document does not address the hardware requirements implied by -the Phase 0, 1, or 3 stages, nor the load required by the calibration -image creation stages. In the first instance, the operations are only -performed on the metadata and are extremely minimal both in terms of -data I/O and computation requirements. In the second case, the -processing is less time critical than the per-image processing and is -performed only infrequently (once per night to once per week or -month). This document also does not address any hardware requirements -introduced by the metadata manipulation. The software implementation -for metadata storage (RDBMS, FITS tables, etc) will have a very large -impact and will be evaluated along with the needed hardware at a later -date. - -\section{Scenarios} - -We will address the various hardware requirements by referring to a -set of data processing and data organization scenarios. The actual -hardware requirements will depend on design decisions which are not -yet available. It is possible to define the data organization in ways -which will minimize the hardware requirements, but which will increase -the software development effort. We will discuss both the worst-case -data organization scenario, which does not require significant -intelligence in the software systems, and the optimal data -organization scenario, which will require the software to track the -location of data products more carefully. In addition, this document -will address the data requirements of the complete Pan-STARRS pipeline -with 4 telescopes as well as the single-telescope Pan-STARRS-1 scenario -based on the Design Reference Mission [REF]. - -The IPP hardware system must provide both data storage and -computational resources. The IPP requires relativley large amounts of -data storage space, primarily for the image data. Image data is -organized in two categories. First, there is the per-OTA data -- data -associated with specific OTAs, including the raw images, the -calibration images, and temporary processed images at various stages. -Second, there is the data associated with the static sky imagery, -which is in turn organized into smaller sky-cell units. The first -assumption we make is that the hardware is organized into nodes which -provide both data storage and computational resources. The second -assumption we make is that the data storage nodes are divided into two -classes: those which deal with the per-OTA data and those that provide -the static sky storage. In addition, we assume that the computational -tasks related to Phase 2 take place on the per-OTA storage nodes and -the Phase 4 computation takes place on the static sky storage nodes. - -Figure~\ref{layout} shows our basic concept for the hardware -organization for the IPP. This diagram shows the two types of compute -nodes: OTA-level processing and storage nodes (dominated by Phase 2) -and static sky processing and storage nodes (mostly Phase 4). Also -shown are two switches used in this configuration; although it is -currently possible to buy a single switch which would have a -sufficient number of GigE ports for both sections of the PS-1 system, -such a two-switch organization may be needed for the full Pan-STARRS -system. In such a case, the interswitch communication must also meet -the required throughput needs. We discuss the hardware requirements -in the assumption that such an organization will be necessary. - -\begin{figure} -\begin{center} -\begin{tabular}{c} -\includegraphics[height=10cm]{hardware-org.eps} -\end{tabular} -\end{center} -\caption{ -\label{layout} -Schematic IPP hardware organization. -} -\end{figure} - -The way in which the images are distributed among the storage and -compute nodes will largely determine the I/O bandwidth requirements. -For data bandwidth requirements calculations, it is necessary to make -some assumptions about the data organization. For the purposes of -this document, we explore two extreme-case options: -\begin{itemize} -\item Random Data Distribution - OTA \& Sky data is randomly - distributed within the compute node of a given type (ie, OTA data is - randomly distributed among the OTA compute nodes). -\item Optimal Data Distribution - OTA \& Sky data is optimally - distributed to compute OTA/Sky nodes (OTA processing is always on a - machine with local OTA data). -\end{itemize} -A second factor which will have a significant impact on the I/O -requirements is the image storage format for the processed and -calibration images. We have two basic choices: 32 bit floating point -format or 16 bit integer format with appropriate scaling. In the -former case, additional dynamic range is retained, while in the latter -case, we reduce the data volume by a factor of 2. While some may -argue that the higher dynamic range is necessary, arguments can be -made that the 16 bit range is sufficient. (In particular, the 16 bit -data provides a dynamic range far above the expected 1/1000 fractional -accuracy of the flat-field images). A related question is the number -of calibration images needed by the processing system. Since the -complete analysis is not yet defined, this number is difficult to -ascertain. However, we can make a range of assumptions which are -reasonable. We therefore adopt two data volume scenarios to explore -these possibilites: -\begin{itemize} -\item Standard Data Volume - 32 bit data for processed and calibration - images, average of 7 calibration frames per image. -\item Minimal Data Volume - 16 bit data for processed and calibration - images, average of 4 calibration frames per image. -\end{itemize} -In the discussion that follows, we explore the hardware requirements -implied by the collection of four combinations of these two sets of -scenario options. - -\begin{table} -\begin{center} -\caption{Hardware Throughput Tests \label{hardware}} -\begin{tabular}{lrrrr} -\hline -\hline -Test & where \& when & model & result \\ -\hline -node I/O & CFHT 11/2002 & Intel 1000 Gigabit & 35 - 40 MB/s sustained \\ -node I/O & CFHT 2/2004 & Intel 1000 Gigabit & 65 - 70 MB/s sustained \\ -RAID write & CFHT 2/2004 & 3ware RAID cntl + IDE & 110 MB/s sustained \\ -Switch Load & VeriTest & Cisco & 3 GB/s (for 32 ports) \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\section{Existing Hardware Throughput} - -We have collected a few representative tests of various pieces of -modern hardware to give a reference for the throughput capabilities. -A number of hardware configurations have been tested at CFHT for the -Elixir project, and we include here their recent reported hardware -RAID-5 I/O speeds and GigE card speeds. We also have included data -from VeriTest studies of Cisco switch throughput, commissioned by -Cisco for a 32 port GigE switch. These tests are summarized in -Table~\ref{hardware}. - -\begin{table}[b] -\begin{center} -\caption{Data Storage Requirements \label{storage}} -\begin{tabular}{lrrrr} -\hline -\hline - & Standard / PS-4 - & Standard / PS-1 - & Minimal / PS-4 - & Minimal / PS-1 \\ -\hline -Raw data & 300 TB & 75 TB & 300 TB & 75 TB \\ -static sky & 512 TB & 64 TB & 256 TB & 32 TB \\ -calibration frames & 175 TB & 18 TB & 17 TB & 5 TB \\ -metadata db & 2 TB & 2 TB & 0.2 TB & 0.2 TB \\ -object db & 60 TB & 4 TB & 60 TB & 4 TB \\ -\hline -totals & 1050 TB & 163 TB & 633 TB & 116 TB \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\section{Data Storage Requirements} - -The Pan-STARRS IPP data storage requirements may be divided into five -principal areas: raw image data, static sky image data, master -calibration images, the metadata database, and the object database. -We discuss each of these data items and their impact on the data -storage requirements for the IPP, and identify the impact of the -minimal vs standard data storage requirements as well as the -requirements specifically for PS-1. Table~\ref{storage} summarizes -the data storage requirements in the different scenarios. - -\subsection{Raw Data Storage} - -There are two basic image types which will be acquired: night-time -science images and calibration images. The night-time science images -consist of 1Gpix per image, or 2GB in raw format. At nominal cadence, -the 4 telescopes can obtain images at a sustained rate of 1 image per -30 seconds per telescope for the entire night of 10 hours (36000 -minutes). A total of 100 calibration images per night would be a -substantial overestimate of the typical expectation. Combining these -numbers, we can expect to receive a total of 1300 image per telescope -per night, 5200 image total, or 10.4 TB of data per night. The total -data storage requirements for the raw data are governed by the number -of nights' worth of data we are required to keep online. A reasonable -number is one month to allow a full moon's cycle. Thus, for raw image -storage, we require a total of 300 TB data storage. For PS-1, this -number is simply scaled down by a factor of 4. The choice of the -minimal data volume does not affect these numbers because the raw data -is already stored with 16 bit pixels. ({\bf note: the PS-1 design -reference may now require storage of the entire first year of data, -calculated to be 200 TB}). - -\subsection{Static Sky Data Storage} - -The static sky is represented by images with 0.2 arcsec per pixel. -There will be one summed image and one weight image for each of the 6 -filters, each stored in floating point format. At this resolution, -there are 324 Mpix per square degree, and we will observe a potential -total area of 30,000 square degrees. Allowing for 10\% overage for -overlapping tiling, we require a total of 10.7 Gpix to cover the sky -once, or a total of $\sim 512$ TB for the static sky images. In the -minimal data volume scenario, this value is reduced by a factor of 2, -while in PS-1, the reduction is a factor of roughly 8 because we only -intend to store the static sky for the ecliptic plane survey and the -small IPP verification program ({\bf note: this last point is no -longer valid - the PS-1 static sky may require the entire 3pi}). - -\subsection{Calibration Frame Storage} - -The possible required calibration frames consist of the bias, dark, -and mask images, along with one flat, one flat-correction, and -multiple sky/fringe library frames per filter. In fact, not all types -are needed at all stages. For the standard data volume, we assume an -average of 7 calibration frames per image and filter. This results in -a total of 42 master calibration image per telescope. If we intend to -keep all master calibration frames for the project lifetime, and -generate a new master on a weekly basis (a reasonable time-scale), -then we can expect to require a total of 175 TB of calibration image -by the end of the 5 year lifetime of the project. For the case of -PS-1, the time period is only 2 years, and there is only 1 telescope, -resulting in a factor of 10 reduction in the volume. For the minimal -data case, we reduce the volume by another factor of 3.5. We also note -that this is likely to be a drastic overestimate as we are unlikely to -need to regenerate all master calibration frames on a weekly -time-scale. - -\subsection{Metadata Database Storage} - -The metadata data storage requirements are driven by the need to store -the data for the project lifetime. There are two types of metadata -generated at the summit: data associated with images and environmental -data. The environmental data consists of measurements on a regular -cadence, roughly 1 per minute, of a variety of parameters. We suggest -an expected of 1kB per entry, for a total of 2.6 GB over the lifetime -of the project. PS-1 will represent a smaller amount of data per -minute, and also a factor of 2.5 fewer minutes. We suggest PS-1 may -have a total environmental metadata set smaller by a factor of 5. The -additional systems, such as the DIMM, SkyProbe, NIR Sky Camera, and -the LRProbe will have higher data requirements, but should be -considered as separate, self-contained systems. Their data products -are distilled to a limited number of parameters per minute which are -included in the 1kB given above. Furthermore, items such as -guide-star history, if saved, will be saved with the image data and -represents only a small fraction of the total image data volume. Some -subset of the telescope diagnosic information may be a high volume -data product as well, but only retained by the telescope control -system for the purpose of diagnostic studies. Such data will be -excluded from this analysis. - -The image metadata consists of values associated with the FPA (4), the -OTAs (240), and the Cells (15360). Aside from the guide star history, -the total data requirements for each of these entries will be scaled -by the number of bytes required for the metadata from each data level. -Clearly, if the Cell entry is allowed to be large, it will dominate -the total Metadata data volume. If we suggest an expected number of -64 bytes per Cell, 256 B per OTA, and 1k per FPA, we find a total -metadata volume per exposure of roughly 1 MB, completely dominated by -the Cell metadata. With the exposure rates above, we find a total of -metadata volume of 1.8 TB over the lifetime of the project. For PS-1, -the total volume is reduced by a factor of 2.5 (for the shorter -lifetime) and another factor of 4 (for the lone telescope). Neither -data quantity is affected by the minimal vs standard data volume -choice. - -\subsection{Object Database Storage} - -The hardware requirements for the IPP object database are rather -flexible: the total volume depends critically on the depth to which -the object detection analyses are performed (and thus the total number -of object detections) and the number of object parameters which are -measured. We can make very rough estimates that the total number of -detections over the 5 year lifetime of the project may be in the -vicinity of $5\times10^{11}$. We can conservatively estimate the -number of bytes needed to represent each detection as 128 B, resulting -in a total data storage for the object detections of 60 TB. However, -this number depends strongly on the timescale for which the IPP is -required to maintain all object detections, and may potentially be -significantly reduced. For the case of PS-1, the total number of -detections is likely to be reduced by a factor of 4 for the number of -telescopes, and potentially another significant factor ($\sim 4?$) by -limiting the depth of object detections. Again, the minimal data -volume scenario is irrelevant to the object database volume. - -\section{CPU Requirements} - -Phase 2 and Phase 4 dominate the processing requirement, primarily -because they must keep up with the image delivery rate of 1 per 30 -seconds. We have performed benchmarks of a demonstration version for -both the Phase 2 and Phase 4 analyses. - -For the Phase 2, a substantial fraction of the processing time is -consumed by the need to perform FFTs on the images in order to -convolve them with the guide-star kernel, and in the smoothing used -for the object detection process. Additional processing time is -needed by the object detection, deblending, and analysis. Experiments -with the FFTW package show that FFTs may be performed on Intel -processors at rates of approximately 0.25 GHz-sec / Mpix for data sets -of order 1 Megapixel. The FFTs required for the Phase 2 analysis are -performed on the 512$^2$ pixel cells, so these numbers may roughly be -scaled linearly to determine the total time required for OTA -processing. A single FFT on a full OTA, with 64 Cells, therefore -requires roughly 4 GHz-sec. For the full Phase 2 analysis, there are -roughly 4 single direction FFTs required excluding those associated -with object detection; thus the total processing time for these FFTs -is approximately 16 GHz-sec. The addtional analysis steps, excluding -object detection and characterization, account for a small fraction of -this compute time, which we estimate at 10\%. The object detection -stage depends somewhat on the depth to which the analysis is -performed, and the number of measurements made per object. Typical -analysis performed by the Sextractor routine, which performs a -substantial number of per-object analyses, requires 27 GHz-sec for a -full OTA, including the FFTs used for smoothing. We can therefore -assume a total of 50 GHz-sec per OTA for the Phase 2 processing. This -converts to a total of 12000 GHz-sec for a complete major frame. - -For Phase 4, the main computainal tasks are combining the multiple -images, with cosmic-ray rejection, and performing the object detection -tasks. Nick Kaiser has done tests of the Phase 4 image combine and -rejection stages, and finds a total processing time of roughly 96 -GHz-sec for a full stack of 4 OTA images. If we add in an additional -34 GHz-sec for detailed object detection and image differencing, we -find a conservative estimage of 130 GHz-sec for a 4-image OTA stack, -equivalent to 7800 GHz-sec for a major frame. - -For PS-1, the data processing will clearly require a smaller amount of -computational resources because of the lower image rate. However, the -total number of GHz-sec required for the complete analysis of 4 input -images and the combination with the static sky will remain -more-or-less the same. Some reduction in the load may be gained by -reducing the complexity and depth of analysis for PS-1. Depending on -the details and depth of the analysis, we may reduce the computational -load by a factor of 2. - -\begin{table} -\begin{center} -\caption{Data Scenarios (MB per OTA or Sky-cell) \label{scenarios}} -\begin{tabular}{lrrrr} -\hline -\hline - & Random / Standard & Random / Minimal & Optimal / Standard & Optimal / Minimal \\ -\hline -{\em Phase 2 input} & & & & \\ -from summit & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB & $2 \times 32$ MB \\ -input image & 32 MB & 32 MB & {\bf 32 MB} & {\bf 32 MB} \\ -calibration & $7 \times 64$ MB & $4 \times 32$ MB & {\bf 7 $\times$ 64 MB} & {\bf 4 $\times$ 32 MB} \\ -mask image & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ -\hline -network I/O: & 560 MB & 232 MB & 64 MB & 64 MB \\ -disk I/O: & (560 MB) & (232 MB) & 496 MB & 168 MB \\ - & & & & \\ -{\em Phase 2 output} & & & & \\ -output image & 64 MB & 32 MB & {\bf 64 MB} & {\bf 32 MB} \\ -output mask & 16 MB & 8 MB & {\bf 16 MB} & {\bf 8 MB} \\ -image to P4 & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB \\ -mask to P4 & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB \\ -\hline -network I/O: & 200 MB & 100 MB & 120 MB & 60 MB \\ -disk I/O: & (80 MB) & (40 MB) & 80 MB & 40 MB \\ - & & & & \\ -{\em Phase 4} & & & & \\ -input images & $1.5 \times 4 \times 64$ MB & $1.5 \times 4 \times 32$ MB & & \\ -input masks & $1.5 \times 4 \times 16$ MB & $1.5 \times 4 \times 8$ MB & & \\ -static sky & 64 MB & 64 MB & & \\ -static weight & 64 MB & 32 MB & & \\ -\hline -input: & 608 MB & 336 MB & & \\ -output: & 192 MB & 128 MB & & \\ -\hline -\multicolumn{5}{l}{\em Bold-faced entries are access to local-disk} \\ -\multicolumn{5}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\ -\end{tabular} -\end{center} -\end{table} - -\section{Per-Node I/O Requirements} - -Data I/O per node is defined as the number of bytes per second passed -through the node's network adapter. The data throughput for each node -depends strongly on the scenarios identified above. In this section, -we identify the data which is passed between nodes for each of the -different scenarios. Table~\ref{scenarios} lists the per-node data -I/O for the four scenarios. - -For PS-4, there are only 30 seconds of compute time allowed for each -of the Phase 2 and Phase 4 analyses. We use the data I/O volumes and -some assumptions about expected network and disk bandwidth to estimate -the I/O and processing timeline for the four scenarios. From this -analysis, we can judge the total CPU requirements in terms of GHz, not -just GHz-sec. We have assumed that GigE network adapters are capable -of delivering data at 50MB/sec sustained and that a disk RAID can -deliver sustained 100 MB/sec reads and writes. These numbers are -conservative estimates based on recent tests discussed above. Using -these assumptions, Table~\ref{throughput} lists the time allocations -for the complete set of scenarios for the case of PS-4. - -\subsection{Random / Standard Data Scenario} - -In the Random Data Distribution scenario, there is a single CPU -allocated to each OTA in the OTA farm and a single CPU for each Sky -cell process. The OTA data are stored across random machines in the -OTA farm, with the result that every Phase 2 processing requires -network access to the data. For each science OTA image which is -observed, each OTA node will read from the network a total of 560 MB -(the 2 raw images for data storage and the 7 calibration frames, along -with one mask and one raw input image) and write a total of 200 MB -(one processed image and the mask along with the 1.5 processed images -and masks for the Phase 4 analysis). Given the assumption of 50 MB/s -from the network adapter, the total data volume implies an I/O period -of 15.2 seconds. Note that the disk I/O is parallel with the network -I/O and substantially underfills the disk bandwidth. - -\subsection{Random / Minimal Data Scenario} - -In the Random-Minimal, there is a single CPU allocated to each OTA in -the OTA farm and a single CPU for each Sky cell process, and the OTA -data are stored across random machines in the OTA farm. However, the -calibration and the processed science images are stored at 2 bytes per -pixel, the mask is set at 4 bits per pixel, and only 4 calibration -images are assumed. For each science OTA image which is observed, -each OTA node will read from the network a total of 232 MB (the 2 raw -images for data storage and the 4 calibration frames, along with one -mask and one raw input image) and write a total of 100 MB (one -processed image and the mask along with the 1.5 processed images for -the Phase 4 analysis). Given the assumption of 50 MB/s from the -network adapter, the total data volume implies an I/O period of 6.6 -seconds. Again, note that the disk I/O is parallel with the network -I/O and substantially underfills the disk bandwidth. - -\subsection{Optimal / Standard Data Scenario} - -In the Optimal Data Distribution scenario, there is a single CPU -allocated to each OTA in the OTA farm and a single CPU for each Sky -cell process. In addition, all data for the specified OTA are stored -on local disks attached to the same computer as the CPU, with the -result that all Phase 2 I/O is made to a local disk. For each science -OTA image which is observed, each OTA node will read from the network -a total of 2 raw images (one for the original image, one for the -backup copy) and write an average of roughly 1.5 processed images and -masks to the Phase 4 machines for a total of 184 MB of network I/O. -During the processing stage, the OTA node will read from disk a total -of 496 MB (7 calibration frames at 64 MB each, one 16 MB mask, and one -raw science image at 32 MB) and write a total of 80 MB (one processed -image at 64 MB and one mask at 8 MB). Given the assumptions for the -network and disk bandwidths (50 MB/s and 100 MB/s respectively), the -data volumes imply a total I/O period of 9.5 seconds. In this -instance, the network I/O is presumed to be sequential with the disk -I/O. - -\subsection{Optimal / Minimal Data Scenario} - -In the Optimal / Minimal Scenario, the minimal data sizes are used -with the optimal data distribution scheme. In this case, we reduce -the disk I/O volume to 168 read and 40 MB write, and the network -traffic to 124 MB. Given the assumptions for the network and disk -bandwidths, the data volumes imply a total I/O period of 4.6 seconds. -Again, the network I/O is presumed to be sequential with the disk I/O. - -\subsection{Phase 4 Node I/O Requirements / Standard Data Volume} - -Although it is easy to arrange the OTA data in such a way that the -majority of I/O is performed locally, it is not as easy to arrange -this for the Static Sky data used by the Phase 4 analysis. We -therefore make the assumption that the Phase 4 analysis will require -all input OTA data to be loaded across the network, as well as all -Static Sky data. This is somewhat of an overestimate as some of the -Static Sky data will be processed by machines with the data stored -locally, and clever Static-Sky data organization schemes can enhance -this chance. - -In the Phase 4 analysis, the images from the 4 separate telescopes are -combined into a single image, confronted with the appropriate segment -of the static sky, with output difference image and updated static sky -image. If we restrict input access to the individual OTA cells, the -maximum read overhead is 50\% (need to read a 10x10 set of cells for -an 8x8 input image). If the processing is performed on Static Sky -segments equivalent in size to the OTAs, the input data is 608 MB (384 -MB of processed science image, 96 MB of mask images, 64 MB of static -sky image and 64 MB of static sky weight map) while the output data is -192 MB (static sky, weight map, and difference image, each 64 MB). -Thus, we require a total of 800 MB network I/O. Given the network -bandwidth, this implies an I/O period of 16 seconds for Phase 4. - -\subsection{Phase 4 Node I/O Requirements / Minimal Data Volume} - -In the minimal data volume scenario, the Phase 4 analysis volume is -significantly reduced. The total volume of input data is 336 MB (192 -MB of processed science image, 48 MB of input mask, 64 MB of static -sky image and 32 MB of static sky weight map) while the output data is -128 MB (64 MB static sky, 32 MB weight map, and 32 MB difference -image). Thus, we require a total of 464 MB network I/O, which implies -an I/O period of 9.3 seconds. - -\begin{table} -\begin{center} -\caption{Data Throughput for 4 Scenarios \label{throughput}} -\begin{tabular}{lrrrr} -\hline -\hline -& -\multicolumn{1}{c}{Random / Standard} & -\multicolumn{1}{c}{Random / Minimal} & -\multicolumn{1}{c}{Optimal / Standard} & -\multicolumn{1}{c}{Optimal / Minimal} \\ -\hline -Phase 2 per-node network I/O & 15.2 s & 6.6 s & 3.7 s & 2.5 s \\ -Phase 2 per-node disk I/O (read) & (5.6 s) & (2.3 s) & 5.0 s & 1.7 s \\ -Phase 2 per-node disk I/O (write) & (0.8 s) & (0.4 s) & 0.8 s & 0.4 s \\ -Phase 2 CPU total & 14 s : 860 GHz & 23 s : 520 GHz & 20 s : 600 GHz & 25 s : 480 GHz \\ -Phase 4 per-node I/O & 16 s & 9.3 s & & \\ -Phase 4 CPU total & 14 s : 490 GHz & 20 s : 390 GHz & & \\ -Phase 2 switch load & 6.1 GB/s & 2.7 GB/s & 1.5 GB/s & 1.0 GB/s \\ -Phase 4 switch load & 0.8 GB/s & 0.5 GB/s & 0.8 GB/s & 0.5 GB/s \\ -Phase 2 to Phase 4 switch load & 1.1 GB/s & 0.6 GB/s & 1.1 GB/s & 0.6 GB/s \\ -Summit to Phase 2 switch load & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s & 0.5 GB/s \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\section{Switch I/O Requirements} - -The switch I/O requirements are defined by the total number of bytes -per second serviced by the two switches in the system. For the -analysis of the Switch I/O requirements, the choice of data -distribution again has a major impact. We again test the four -scenarios discussed above: Random Data Distribution, Random / Minimal, -Optimal Data Distribution, and Optimal / Minimal. - -\subsection{Random / Standard Data Scenario} - -In the Random Data Distribution scenario, each OTA node needs to read -a total of 560 MB from the network and write a total of 200 MB every -30 seconds. With 240 OTA nodes, this corresponds to a total bandwidth -of 6080 MB/sec, or 49 Gb/sec. Note that this includes the bandwidth -needed to copy data from the summit and make two copies on the OTA -machines, as well as the bandwidth to send the processed image -portions to the Phase 4 machines. The Phase 4 processing adds an -additional 320 MB of network I/O per Sky-Cell group, and there are -roughly 60-70 Sky-cells per exposure set. Thus the Phase 4 processing -adds an additional 750 MB/sec network bandwidth. In the architecture -defined in Figure NN, the Sky nodes and the OTA nodes are each -attached to separate switches. An additional bandwidth requirement is -derived by the need to exchange data between these switches in for -Phase 4. The total amount of data exchanged between these switches is -480 MB per Sky-cell, for a total bandwidth of 1120 MB/sec. In -addition, the connection to the summit is a single, separate line -which needs to support the bandwidth requirement of copying all intial -raw images. In our simple model, each raw image is copied twice, -accounting for a total of 15360 MB every 30 seconds, or a bandwidth -load of 512 MB/sec. (Note that this last is double the actual -bandwidth requirement to the summit: a dedicated local circular buffer -would reduce the need for the second copy to come directly from the -summit.) - -\subsection{Random / Minimal Data Scenario} - -In the Random / Minimal Scenario, the data volumes are significantly -reduced. The total Phase 2 bandwidth contribution is 332 MB over 30 -seconds for 240 nodes (2656 MB/sec) and the residual Phase 4 bandwidth -load is 224 MB per Sky cell over 30 seconds (522 MB/sec). The -inter-switch communication is now 240 MB per sky cell over 30 seconds, -or 560 MB/sec. - -\subsection{Optimal / Standard Data Scenario} - -In the Optimal Data Distribution, the Phase 2 network bandwidth is -reduced significantly to 184 MB per OTA node, for a total of -1.5GB/sec, while the Phase 4 network bandwidth remains unchanged at -750 MB/sec. The inter-switch communication also remains the same at -1.12 GB/sec. - -\subsection{Optimal / Minimal Data Scenario} - -In the Optimal / Minimal Scenario, the total Phase 2 network bandwidth -drops to 124 MB per OTA node, for a total of 1.0GB/sec, while the -Phase 4 network bandwidth is 552 MB/sec. The inter-switch -communication remains the same as the Random/Minimal Scenario at 560 -MB/sec. - -\begin{table}[t] -\begin{center} -\caption{\label{NP2} Phase 2 load per major frame (12000 GHz-sec)} -\begin{tabular}{lrrrr} -\hline -\hline -& Random/Standard -& Random/Minimal -& Optimal/Standard -& Optimal/Minimal \\ -\hline -network I/O (GB) & 182 & 80 & 44 & 30 \\ -PS-1 & & & & \\ - I/O (cpu-sec) & 3640 & 1600 & 880 & 600 \\ - CPU (cpu-sec) & 4000 & 4000 & 4000 & 4000 \\ - \# cpus & 64 & 47 & 41 & 38 \\ -PS-4 & & & & \\ - I/O (cpu-sec) & 1820 & 800 & 440 & 300 \\ - CPU (cpu-sec) & 2000 & 2000 & 2000 & 2000 \\ - \# cpus & 127 & 93 & 81 & 77 \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[b] -\begin{center} -\caption{\label{NP4} Phase 4 load per major frame (7800 GHz-sec)} -\begin{tabular}{lrr} -\hline -\hline -& Standard -& Minimal \\ -\hline -network I/O (GB) & 48 & 28 \\ -PS-1 & & \\ - I/O (cpu-sec) & 960 & 557 \\ - CPU (cpu-sec) & 2600 & 2600 \\ - \# cpus & 30 & 26 \\ -PS-4 & & \\ - I/O (cpu-sec) & 480 & 278 \\ - CPU (cpu-sec) & 1300 & 1300 \\ - \# cpus & 59 & 53 \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\section{Conclusions} - -Table~\ref{throughput} presents one way of analysing the hardware -requirements, making a specific set of assumptions about the number of -nodes for the two phases and the expected network and disk -bandwidths. The important conclusion in this analysis is the implied -number of GHz per processor, given the assumptions laid out. -Phase 2 is specified to have 240 OTA nodes, while Phase 4 is specified -to have roughly 60 static sky nodes. The range of Phase 2 CPU -requirements implies that each CPU needs to have speeds in the range -of 2.0 - 3.6 GHz, which sound very plausible for the year 2007, since -these apply to PS-4. - -Another way to represent this information is to use the total number -of MB I/O and the total number of GHz-sec required for the two stages, -confront these with an assumption for the bandwidth per network -adapter and an assumption for the CPU speed and use those numbers to -calculate the minimum number of nodes (CPUs) needed to sustain the -timing requirements. There are quite a few parameters and options to -choose from. We have assumed that for PS-1, the time between major -frames (4 images combined in Phase 4) is 120 seconds, and 30 seconds -for PS-4. We have also assumed that each CPU has one network adapter -associated with it, and use the numbers of 50 MB/sec for PS-1 era -network adapters and 100 MB/sec for the PS-4 network adapters (since -there has been some steady improvement in GigE hardware over the past -year). We have also assumed each PS-1 CPU is rated at 3 GHz and those -for PS-4 are rated at 6 GHz (somewhat conservative since 3 GHz -machines are already available). Tables~\ref{NP2} and \ref{NP4} show -the load and resulting number of nodes for both Phase 2 and Phase 4 -for both the PS-1 and PS-4 assumptions, using the I/O numbers for all -of the scenarios above. Note that in these discussions, we make the -idealized assumption that the computational and I/O portions of each -process are completely serial. As a result, the CPU is completely -used to perform the I/O during the I/O phase, avoiding any concern -about I/O load on the processor during analysis. - -\end{document} - -\section{Data Throughput Calculations} -\label{datarate} - -The following data throughput discussion is cut from an email by Josh -Hoblitt. - -We can do some quick 'back of the envelope' estimates to figure out -the worst case bandwidth needs. Let's assume that control messages -and db traffic is negligible compared to the image data. Let's also -assume that there isn't even trivial data locality optimization {\em -and} we have to reload all calibration data for every exposure. That -gives us: - -\begin{verbatim} - 4 fully populated focal planes - - 2bytes * ( 4096^2pixels * 1.125overclocks ) * 240otas = 72477573120b - - slew/settle/integrate/read cadence of 30s - - 72477573120b / 30s = 2415919104b/s - - exposure data stored as float - - 4bytes * 4096^2pixels * 240otas = 128849018880b - - 128849018880b / 30s = 4294967296b/s - - best/working stacked stored as float - - 4bytes * 4096^2pixels * 60otas = 32212254720 - - 32212254720b / 30s = 1073741824b/s - - debias, dark, flat, 2x fringe, and 2x sky calibration frames - - 7 * 4294967296b/s = 30064771072b/s - - Which gives us a grand total of: - - [phase 2] - 2415919104b/s summit -> disk (exposure) - + 2415919104b/s non-local disk -> memory (exposure) - + 30064771072b/s non-local disk -> memory (calibration) - + 4294967296b/s memory -> non-local disk (reduced) - ------------------ - 39,191,576,576b/s phase 2 total - - [phase 4] - 4294967296b/s non-local disk -> memory (reduced) - + 1073741824b/s non-local disk -> memory (best) - + 1073741824b/s non-local disk -> memory (working) - + 1073741824b/s memory -> non-local disk (diff) - + 1073741824b/s memory -> non-local disk (working) - ------------------ - 8,589,934,592b/s phase 4 total - - [total] - 39191576576b/s phase 2 total - + 8589934592b/s phase 4 total - ------------------ - 47,781,511,168b/s total is ~48Gb/s -\end{verbatim} - - 48Gb/s is 1/15th of the bandwidth available in a 65xx/SUP720. Given - these numbers I don't think that engineering effort to optimize for a - hierarchical network topology is justifiable. - - To address your concerns about the backplane numbers being accurate, - I can tell you that I have a significant amount of experience with - enterprise class switches from Cisco, Foundry, and HP. I've always - found the switching fabric specs to be accurate from all three of - these manufactures. In addition, as soon as I have equipment - available for use by Pan-Starrs I will generate my own bandwidth - tests. However it should be noted that as demonstrated above that - even our worst case bandwidth needs are rather modest compared to - COTS equipment that is available today. - -\end{document} - - -\subsection{Computer Hardware} - -\subsubsection{Overview} - -This section discusses the Pan-STARRS Image Processing Pipeline (IPP) -PS-1 hardware requirements. The hardware requirements addressed in -this section consist of: - -\begin{itemize} -\item Total Disk Volume -\item Total Processing Power -\item Sustained Switch Bandwidth -\item Sustained Node Network I/O -\item Sustained Disk I/O -\end{itemize} - -Even without the complete IPP design, it is possible to identify the -major drivers on the hardware requirements. The total disk volume -requirements are dominated by the need to store raw images for a -certain period, the need to store calibration images for a longer -period, and the need to store the static sky images. Of the various -analysis stages, Phase 2 and Phase 4 present the most significant -demands in terms of data I/O throughput on the network. Phase 2 and -Phase 4 also present the most significant CPU demands. In this -discusion, Phase 2 refers to the per-OTA image pre-processing in which -the instrumental signature is removed and a first pass object -detection is performed. Phase 4 refers to the multiple OTA -combination in which the pre-processed images are merged and combined, -in both addition and subtraction, with the static sky image, and up to -three object detection passes are performed. - -This document does not address the hardware requirements implied by -Phase 1 or 3, nor the load required by the calibration or reference -catalog creation stages. In the first instance, the operations are -only performed on the metadata and are extremely minimal both in terms -of data I/O and computation requirements. In the second case, the -processing is less time critical than the per-image processing and is -performed only infrequently (once per night to once per week, month or -year). \tbd{The software implementation for metadata storage (RDBMS, -FITS tables, etc) will have a very large impact and will be evaluated -along with the needed hardware at a later date.} - -We will address the various hardware requirements by referring to an -assumed data processing and data organization scenario. The -organization of the data and certain aspects of the data processing -scheme have very large implications for the hardware requirements. In -this analysis, we assume that data types are chosen to minimize the -data volume and that the data is organized to minimize the I/O -bandwidth needs, as defined below. We address the data requirements -of the single-telescope Pan-STARRS-1 scenario based on the Design -Reference Mission \tbd{REF}. - -\subsubsection{Data Organization} - -The IPP hardware system must provide both data storage and -computational resources. The IPP requires relatively large amounts of -data storage space, primarily for the image data. Image data is -organized in two categories. First, there is the per-OTA data -- data -associated with specific OTAs, including the raw images, the -calibration images, and temporary processed images at various stages. -Second, there is the data associated with the static sky imagery, -which is in turn organized into smaller sky-cell units. The first -assumption we make is that the hardware is organized into nodes which -provide both data storage and computational resources. The second -assumption we make is that the data storage nodes are divided into two -classes: those which deal with the per-OTA data and those that provide -the static sky storage. In addition, we assume that the computational -tasks related to Phase 2 take place on the per-OTA storage nodes and -the Phase 4 computation takes place on the static sky storage nodes. - -Figure~\ref{hardware} shows our basic concept for the hardware -organization for the IPP. This diagram shows the two types of compute -nodes: OTA-level processing and storage nodes (dominated by Phase 2) -and static sky processing and storage nodes (mostly Phase 4). Also -shown are two switches used in this configuration; although it is -currently possible to buy a single switch with sufficient number of -ports, this organization represents a minimal configuration for the -PS-1 IPP hardware. In such a case, the interswitch communication must -also meet the required throughput needs. We discuss the hardware -requirements in the assumption that such an organization will be -necessary. - -The way in which the images are distributed among the storage and -compute nodes will largely determine the I/O bandwidth requirements. -For data bandwidth requirements calculations, it is necessary to make -some assumptions about the data organization. We make the assumption -that the OTA data is optimally distributed to the OTA nodes such that -the OTA processing is always on a machine with local OTA data. This -implies that all OTA data from a specific OTA are targetted to a -specific machine. (see below for discussion of data duplication). - -A second factor which will have a significant impact on the I/O -requirements is the image storage format for the processed and -calibration images. We have two basic choices: 32 bit floating point -format or 16 bit integer format with appropriate scaling. In the -former case, additional dynamic range is retained, while in the latter -case, we reduce the data volume by a factor of 2. Since the science -requirements for PS-1 do not specify a need for dynamic range greater -than 16 bits, we assume all images are stored as 16 bit data. - -A third determining factor is the number of calibration images needed -by the processing system. Since the complete analysis is not yet -defined, this number is difficult to ascertain. However, we can make -a reasonable guess at the total number for scaling purposes. We -assume that each frame requires a total of 4 calibration frames on -average - -\begin{table}[b] -\begin{center} -\caption{Data Storage Requirements \label{storage}} -\begin{tabular}{lrrrr} -\hline -\hline -Raw data & 200 TB \\ -static sky & 256 TB \\ -calibration frames & 5 TB \\ -metadata db & 0.3 TB \\ -object db & 4 TB \\ -\hline -total & 116 TB \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Data Storage Requirements} - -The Pan-STARRS IPP data storage requirements may be divided into five -principal areas: raw image data, static sky image data, master -calibration images, the metadata database, and the object database. -We discuss each of these data items and their impact on the data -storage requirements for the IPP for PS-1. Table~\ref{storage} -summarizes the data storage requirements in the different scenarios. - -\paragraph{Raw Data Storage} - -There are two basic image types which will be acquired: night-time -science images and calibration images. The night-time science images -consist of 1Gpix per image, or 2GB in raw format. At nominal cadence, -the PS-1 telescope can obtain images at a sustained rate of 1 image -per 30 seconds for the entire night of 10 hours (36000 seconds). A -total of 100 calibration images per night would be a substantial -overestimate of the typical expectation. Combining these numbers, we -can expect to receive a total of 1300 images, or 2.6 TB of data per -night. The total data storage requirements for the raw data are -governed by the number of nights' worth of data we are required to -keep online. \tbd{for the first year, we are required to keep all -images from the AP and IPV surveys. This amounts to a total of 200 -TB of data}. - -\paragraph{Static Sky Data Storage} - -The static sky is represented by images with 0.2 arcsec per pixel. -There will be one summed image and one weight image for each of the -\tbd{6} filters, each stored with 16 bits of resolution, for a total -of 24 bytes per sky pixel. At this resolution, there are 324 Mpix per -square degree, and we will observe a potential total area of 30,000 -square degrees. Allowing for 10\% overage for overlapping tiling, we -require a total of 10.7 Tpix to cover the sky once, or a total of -$\sim 256$ TB to maintain a single image of the static sky in all 6 -filters. - -\paragraph{Calibration Frame Storage} - -The possible required calibration frames consist of the bias, dark, -and mask images, along with one flat, one flat-correction, and -multiple sky/fringe library frames per filter. In fact, not all types -are needed at all stages. It is very likely that we will not require -bias or dark images, and mask images may be represented by a single -byte per pixel. Nonetheless, it is necessary for us to generate and -store all master calibration frames at least until we prove that they -are not needed. We assume a total of 21 calibration images are -necessary (one flat, fringe, and sky per filter, along with a bias, -dark, and mask). If we intend to keep all master calibration frames -for the project lifetime, and generate a new master on a weekly basis -(a reasonable time-scale), then we can expect to require a total of 5 -TB of calibration image by the end of the 2 years of PS-1. We note -that this is likely to be a drastic overestimate as we are unlikely to -need to regenerate all master calibration frames on a weekly -time-scale. - -\paragraph{Metadata Database Storage} - -The metadata data storage requirements are driven by the need to store -the data for the project lifetime. There are two types of metadata -generated at the summit: data associated with images and environmental -data. The environmental data consists of measurements on a regular -cadence, roughly 1 per minute, of a variety of parameters. We suggest -an expected of 1kB per entry, for a total of 1 GB over the two-year -term of PS-1. The additional systems, such as the DIMM, SkyProbe, NIR -Sky Camera, and the LRProbe will have higher data requirements, but -should be considered as separate, self-contained systems. Their data -products are distilled to a limited number of parameters per minute -which are included in the 1kB given above. Furthermore, items such as -guide-star history, if saved, will be saved with the image data and -represents only a small fraction of the total image data volume. Some -subset of the telescope diagnosic information may be a high volume -data product as well, but only retained by the telescope control -system for the purpose of diagnostic studies. Such data will be -excluded from this analysis. - -The image metadata consists of values associated with the FPA (1), the -OTAs (64), and the Cells (4096). Aside from the guide star history, -the total data requirements for each of these entries will be scaled -by the number of bytes required for the metadata from each data level. -Clearly, if the Cell entry is allowed to be large, it will dominate -the total Metadata data volume. We suggest an expected number of 64 -bytes per Cell, 256 B per OTA, and 1k per FPA, yielding a total -metadata volume per exposure of roughly 0.3 MB, completely dominated -by the Cell metadata. With the exposure rates above, we find a total -of metadata volume of 0.3 TB over the two-year term of PS-1. - -\paragraph{Object Database Storage} - -The hardware requirements for the IPP object database are rather -flexible: the total volume depends critically on the depth to which -the object detection analyses are performed (and thus the total number -of object detections) and the number of object parameters which are -measured. We can make very rough estimates that the total number of -detections over the 2 year lifetime of the project may be in the -vicinity of $10^{11}$. We can conservatively estimate the number of -bytes needed to represent each detection as 128 B, resulting in a -total data storage for the object detections of 12 TB. However, this -number depends strongly on the timescale for which the IPP is required -to maintain all object detections, and may potentially be -significantly reduced. - -\subsubsection{CPU Requirements} - -Phase 2 and Phase 4 dominate the processing requirement, primarily -because they must keep up with the image delivery rate of 1 per 30 -seconds. We have performed benchmarks of a demonstration version for -both the Phase 2 and Phase 4 analyses. - -For the Phase 2, a substantial fraction of the processing time is -consumed by the need to perform FFTs on the images in order to -convolve them with the guide-star kernel, and in the smoothing used -for the object detection process. Additional processing time is -needed by the object detection, deblending, and analysis. Experiments -with the FFTW package show that FFTs may be performed on Intel -processors at rates of approximately 0.25 GHz-sec / Mpix for data sets -of order 1 Megapixel. The FFTs required for the Phase 2 analysis are -performed on the 512$^2$ pixel cells, so these numbers may roughly be -scaled linearly to determine the total time required for OTA -processing. A single FFT on a full OTA, with 64 Cells, therefore -requires roughly 4 GHz-sec. For the full Phase 2 analysis, there are -roughly 4 single direction FFTs required excluding those associated -with object detection; thus the total processing time for these FFTs -is approximately 16 GHz-sec. The addtional analysis steps, excluding -object detection and characterization, account for a small fraction of -this compute time, which we estimate at 10\%. The object detection -stage depends somewhat on the depth to which the analysis is -performed, and the number of measurements made per object. Typical -analysis performed by the Sextractor routine, which performs a -substantial number of per-object analyses, requires 27 GHz-sec for a -full OTA, including the FFTs used for smoothing. We can therefore -assume a total of 50 GHz-sec per OTA for the Phase 2 processing. This -converts to a total of 12800 GHz-sec for a complete major frame. - -For Phase 4, the main computational tasks are combining the multiple -images, with cosmic-ray rejection, and performing the object detection -tasks. Nick Kaiser has done tests of the Phase 4 image combine and -rejection stages, and finds a total processing time of roughly 96 -GHz-sec for a full stack of 4 OTA images. If we add in an additional -34 GHz-sec for detailed object detection and image differencing, we -find a conservative estimage of 130 GHz-sec for a 4-image OTA stack, -equivalent to 7800 GHz-sec for a major frame. - -For PS-1, the typical time for a major frame is $4 \times 30$ seconds. -Some reduction in the load may be gained by reducing the complexity -and depth of analysis for PS-1. Depending on the details and depth of -the analysis, we may reduce the computational load by a factor of 2. - -\begin{table} -\begin{center} -\caption{Data I/O (MB per OTA or Sky-cell) \label{scenarios}} -\begin{tabular}{lrrrr} -\hline -\hline -{\em Phase 2 input} \\ -from summit & $2 \times 32$ MB \\ -input image & {\bf 32 MB} \\ -calibration & {\bf 4 $\times$ 32 MB} \\ -mask image & {\bf 8 MB} \\ -\hline -network I/O: & 64 MB \\ -disk I/O: & 176 MB \\ - & \\ -{\em Phase 2 output} \\ -output image & {\bf 32 MB} \\ -output mask & {\bf 8 MB} \\ -image to P4 & $1.5 \times 32$ MB \\ -mask to P4 & $1.5 \times 8$ MB \\ -\hline -network I/O: & 60 MB \\ -disk I/O: & 40 MB \\ - & \\ -{\em Phase 4} & \\ -input images & $1.5 \times 4 \times 32$ MB \\ -input masks & $1.5 \times 4 \times 8$ MB \\ -static sky & 32 MB \\ -static weight & 32 MB \\ -\hline -input: & 304 MB \\ -output: & 96 MB \\ -\hline -\multicolumn{5}{l}{\em Bold-faced entries are access to local-disk} \\ -\multicolumn{5}{l}{\em parenthesised disk I/O numbers are parallel with the network I/O} \\ -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Per-Node I/O Requirements} - -Data I/O per node is defined as the number of bytes per second passed -through the node's network adapter. The data throughput for each node -depends strongly on the how the data is organized and processed. In -this section, we identify the data which is passed between nodes for -the two stages of the science analysis process. Table~\ref{scenarios} -lists the per-node data I/O for the analysis stages. - -For PS-1, there are 120 seconds of compute time allowed for each of -the Phase 2 and Phase 4 analyses for the collection of four images -which makes up a cannonical major frame. We use the data I/O volumes -and some assumptions about expected network and disk bandwidth to -estimate the I/O and processing timeline for the four scenarios. From -this analysis, we can judge the total CPU requirements in terms of -GHz, not just GHz-sec. We have assumed that GigE network adapters are -capable of delivering data at 50MB/sec sustained and that a disk RAID -can deliver sustained 100 MB/sec reads and writes. These numbers are -conservative estimates based on recent tests discussed below. Using -these assumptions, Table~\ref{throughput} lists the time allocations -for the processing stages. - -\paragraph{Phase 2 Node I/O Requirements} - -In the assumed data distribution scenario, there is a single CPU -allocated to each OTA in the OTA farm and a single CPU for each Sky -cell process. In addition, all data for the specified OTA are stored -on local disks attached to the same computer as the CPU, with the -result that all Phase 2 I/O is made to a local disk. For each science -OTA image which is observed, each OTA node will read from the network -a total of 2 raw images (one for the original image, one for a backup -copy) and write an average of roughly 1.5 processed images and masks -to the Phase 4 machines for a total of 124 MB of network I/O. During -the processing stage, the OTA node will read from disk a total of 176 -MB (4 calibration frames at 32 MB each, one 16 MB mask, and one raw -science image at 32 MB) and write a total of 40 MB (one processed -image at 32 MB and one mask at 8 MB). Given the assumptions for the -network and disk bandwidths (50 MB/s and 100 MB/s respectively), the -data volumes imply a total I/O period of 4.6 seconds. In this -instance, the network I/O is presumed to be sequential with the disk -I/O. - -\paragraph{Phase 4 Node I/O Requirements} - -Although it is easy to arrange the OTA data in such a way that the -majority of I/O is performed locally, it is not as easy to arrange -this for the Static Sky data used by the Phase 4 analysis. We -therefore make the assumption that the Phase 4 analysis will require -all input OTA data to be loaded across the network, as well as all -Static Sky data. This is somewhat of an overestimate as some of the -Static Sky data will be processed by machines with the data stored -locally, and clever Static-Sky data organization schemes can enhance -this chance. - -In the Phase 4 analysis, the images from the 4 separate telescopes are -combined into a single image, confronted with the appropriate segment -of the static sky, with output difference image and updated static sky -image. If we restrict input access to the individual OTA cells, the -maximum read overhead is 50\% (need to read a 10x10 set of cells for -an 8x8 input image). If the processing is performed on Static Sky -segments equivalent in size to the OTAs, the total volume of input -data per node is 304 MB (192 MB of processed science image, 48 MB of -input mask, 32 MB of static sky image and 32 MB of static sky weight -map) while the output data is 96 MB (32 MB static sky, 32 MB weight -map, and 32 MB difference image). Thus, we require a total of 400 MB -network I/O, which implies an I/O period of 8 seconds. - -\begin{table} -\begin{center} -\caption{Data Throughput \label{throughput}} -\begin{tabular}{lrrrr} -\hline -\hline -Phase 2 per-node network I/O & 2.2 s \\ -Phase 2 per-node disk I/O (read) & 1.3 s \\ -Phase 2 per-node disk I/O (write) & 1.2 s \\ -Phase 2 CPU total & 25 s : 128 GHz \\ -Phase 4 per-node I/O & 8 s \\ -Phase 4 CPU total & 112 s : 70 GHz \\ -Phase 2 switch load & 264 MB/s \\ -Phase 4 switch load & 215 MB/s \\ -Phase 2 to Phase 4 switch load & 160 MB/s \\ -Summit to Phase 2 switch load & 70 MB/s \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Switch I/O Requirements} - -The switch I/O requirements are defined by the total number of bytes -per second serviced by the two switches in the system. - -The Phase 2 network I/O is 124 MB per OTA. With 64 OTAs per image, -and 30 seconds average between images, this implies a total of 264 -MB/s switch bandwidth. The Phase 4 network I/O is 400 MB per sky -cell. With 64 cells and 120 seconds between major frames, this is an -average switch bandwidth of 215 MB/s switch bandwidth. The total -switch-to-switch load is 304 MB per OTA, with an average timescale of -120 seconds. With 64 OTAs, this corresponds to 160 MB/s. The -summit-to-Phase 2 switch load is 70 MB/s. - -\begin{table} -\begin{center} -\caption{Hardware Throughput Tests \label{existing-hardware}} -\begin{tabular}{lrrrr} -\hline -\hline -Test & where \& when & model & result \\ -\hline -node I/O & CFHT 11/2002 & Intel 1000 Gigabit & 35 - 40 MB/s sustained \\ -node I/O & CFHT 2/2004 & Intel 1000 Gigabit & 65 - 70 MB/s sustained \\ -RAID write & CFHT 2/2004 & 3ware RAID cntl + IDE & 110 MB/s sustained \\ -Switch Load & VeriTest & Cisco & 3 GB/s (for 32 ports) \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Existing Hardware Throughput} - -We have collected a few representative tests of various pieces of -modern hardware to give a reference for the throughput capabilities. -A number of hardware configurations have been tested at CFHT for the -Elixir project, and we include here their recent reported hardware -RAID-5 I/O speeds and GigE card speeds. We also have included data -from VeriTest studies of Cisco switch throughput, commissioned by -Cisco for a 32 port GigE switch. These tests are summarized in -Table~\ref{existing-hardware}. - Index: unk/doc/design/hardware_mgt.tex =================================================================== --- /trunk/doc/design/hardware_mgt.tex (revision 6007) +++ (revision ) @@ -1,337 +1,0 @@ -%\documentclass[panstarrs,psreport,12pt]{panstarrs} -\documentclass[panstarrs]{panstarrs} - -% basic document variables -\title{Pan-STARRS PS1 Image Processing Pipeline Hardware Management Plan} -\subtitle{System/Subsystem Specification} -\author{Joshua Hoblitt} -\audience{Pan-STARRS Management} -\shorttitle{IPP Hardware Management Plan} -\group{Pan-STARRS IPP Group} -\project{Pan-STARRS Image Processing Pipeline} -\organization{Institute for Astronomy} -\version{DR01} -\docnumber{PSDC-4xx-xxx} -% note the use of the docnumber & version number: -% the complete PSDC document number is given by -% \thedocnumber-\theversion - -\begin{document} -\maketitle - -% -- Revision History -- -\RevisionsStart -% version Date Description -DR00 & 2004.10.28 & First draft \\ -\hline -DR01 & 2004.11.21 & Second draft \\ -\hline -\RevisionsEnd - -% -- TBD / TBR Listing -- -%\TBDsStart -% section % page Date Description -%1.2.3.4 & 2 & xxx-xxx & missing a dingle on a dangle \\ -%\TBDsEnd - -\pagebreak -\tableofcontents - -%\pagebreak -%\listoffigures - -\pagebreak -\pagenumbering{arabic} - -\section{Introduction} - -\subsection{Scope} - -This document presents the strategy for managing the PS1 IPP Cluster through -it's projected life-cycle which will span a period of approximately 3 years. -The plan covers hardware setup, validation, operations, and graceful -decommissioning. The life-cycle has been broken down into three primary phases -which in some cases have been decomposed into smaller stages and steps for -clarity. - -\section{Life-Cycle} - -\subsection{Commissioning} - -In this phase the physical installation, configuration, and testing of the -cluster will be accomplished. This includes testing the functionality of the -nodes, network connectivity, management hardware, and storage media. This -phase is broken down into three discrete stages. - -\subsubsection{Setup} - -This covers the installation of the nodes, the network equipment plus specing -and acquiring the cluster cabling and the management equipment. Followed by -the testing of the integrated system. This stage should cover approximately -75 working days. - -\paragraph{Step 1} - -The IPP nodes will arrive pre-installed into standard 42U racks along with 1-2 -additional racks for the core switch and management equipment. There will also -be a small pool of hard disks included with the shipment for the testing of the -RAID controllers. The shipment will need to be unpackaged, inventoried, -inspected for physical damage, and positioned in the data center. - -Once the racks are in their final configuration and the core switch has been -mounted, the lengths for network and console server cabling will be measured -and the appropriate cabling ordered. The network cabling shall be black PVC or -Plenum rated Category 6 RJ-45 patch cables. The network console server cabling -shall be red PVC or Plenum rated Category 6 RJ-45 patch cables. - -Console server equipment and network managed power strips can then be ordered. -The number and type of network managed power strips will depend on whether 110v -or 220v electricity is available and the physical configuration of the cluster -racks. The console server equipment and managed power strips selected -inter-operate to provide a unified cluster management system. -Cyclades\footnote{Cyclades - www.cyclades.com} has been identified as possible vendor for -this equipment. - -There will be a slack period of approximately one month awaiting order -fulfillment of cabling and equipment. - -\paragraph{Step 2} - -The network and console wiring will be installed and connected to the core -switch, managed power strips, and the console servers. - -The cluster nodes will follow a standardize naming convention in the format of -"ippxxx", where "xxx" is an integer sequentially assigned from the range -[000..999]. Each node shall have one of it's gigabit Ethernet ports connected -to the core switch by a Category 6 network patch cable. The core switch has -four 48-port gigabit Ethernet modules and one fourth of the nodes shall be -connected to each of those modules. The core switch shall be configured such -that all of the Ethernet ports are in the same VLAN. - -The nodes shall have a Linux 2.6 based operating system that meets the -requirements defined in Software Requirements Specification\footnote{Software -Requirements Specification - PSDC-430-005}installed on them. This shall be -accomplished by setting up temporary DHCP and TFTP servers, booting the nodes -from the network via PXE, and installing a local operating system image with -SystemImager\footnote{SystemImager - http://www.systemimager.org/} or another -similar software image installation package. All filesystems should use -\code{ext3} with the obvious exception of system swap space. - -The nodes shall each be assigned a permanent IP address in an RFC -1519\footnote{RFC 1519 - ftp://ftp.rfc-editor.org/in-notes/rfc1519.txt} Class C -equivalent or larger address space. If private IP address space, as defined in -RFC 1918\footnote{RFC 1918 - ftp://ftp.rfc-editor.org/in-notes/rfc1918.txt}, is -to be used for the IPP Cluster then special coordination will need to be made -with the Pan-STARRS Global Systems group due to possible network routing issues -between the IPP Cluster and other PS1 data systems. The IP address allocation -for the nodes shall be sequential from the beginning of the address space with -the first two IP address reserved for the gateway. The IP address of all -of the IPP cluster nodes should be recorded into the \file{/etc/hosts} file on -every node to prevent DNS look-ups during inter-cluster communication. -\tbd{Additional configuration based on some other document that describes IPP -software requirements?} - -One of the nodes shall be designated as the centralized log host for Syslog -and other monitor tools. In particular, \code{3DM}\footnote{3DM - -http://www.3ware.com/products/raid\_management.asp} shall be used for monitoring -the health of the 3ware RAID controllers and attached disks. \code{3DM} sends -warnings and error notifications via SMTP. The log host should be configured -with a dedicated account to receives these messages. - -\begin{figure} -\begin{center} -\begin{tabular}{ll} -Node &IP address\\ -\cline{1-2} -(reserved) &x.x.x.1\\ -(reserved) &x.x.x.2\\ -ipp000 &x.x.x.3\\ -ipp001 &x.x.x.4\\ -ipp002 &x.x.x.5\\ -\cline{1-2} -\end{tabular} -\caption{ \label{addressing} IP Addressing} -\end{center} -\end{figure} - -The IPP Hardware Test Plan should then be carried out. The testing will cover -at least the following items: - -\filbreak -Nodes: - -\begin{itemize} -\item -Power-supplies -\item -Memory -\item -CPUs -\item -PCI device identification -\item -Boot disk -\item -Network interfaces -\item -RAID controllers -\end{itemize} - -\filbreak -Core Switch: - -\begin{itemize} -\item -Memory -\item -Modules -\item -Hardware Diagnostics -\end{itemize} - -\subsubsection{Software Development} - -The cluster enters into a period of partial dormancy where all but 6 of the -nodes are deactivated. The remaining active nodes are to each have 12 disks -from the pool of commissioning disks installed and configured as a single RAID -5 volume mounted as \file{/export/[nodename]/vol0} and exported via NFS as -\code{[nodename]}. These nodes are to remain available for the testing of IPP -software until the cluster enters into it's production phase. The duration of -this stage will vary as Stage 3 won't begging until PS1 is almost ready for -first light. The minimum duration of this stage is 2.5 months. - -\subsubsection{Pre-Production} - -The $1^{st}$ lot of storage disks are ordered and the complete cluster is -brought on line and prepared for production. - -\paragraph{Step 1} - -Approximately 4 months before the IPP Cluster needs to be fully operational the -first of three lots of 600 disks shall be ordered. The disks ordered should be -of the highest capacity commonly available. It is estimated that this will be -at least 400GB per disk when this stage is reached. The disks maybe purchased -loose or pre-installed in drive sleds if possible. The disks must have a 3 -year warranty either from the OEM or the disk manufacturer. There will be a -slack period of approximately two months while the purchase is approved and -delivery is completed. - -\paragraph{Step 2} - -All IPP nodes are activated and subsections x-y of the IPP Hardware Test Plan -are re-run. If the $1^{st}$ lot of disks are purchased lose they will be -concurrently mounted into drive sleds, sled mounting should take approximately -50 hours of labor. The new disks should be installed in sets of 12 on a single -RAID controller in nodes [ipp000..ipp049] and configured as a single RAID 5 -volume mounted as \file{/export/[nodename]/vol0} and shared via NFS by -exporting \file{/export/[nodename]} as \code{[nodename]}. The new disks shall -then be run through the ``Storage'' section of the IPP Hardware Test Plan. - -\begin{figure} -\begin{center} -\resizebox{5in}{!}{ -\includegraphics[angle=0]{pics/ps1_ipp_storage_lots.ps} -} -\caption{ \label{lots} Disk Lots} -\end{center} -\end{figure} - -\subsection{Production} - -During this phase the primary activity will be hardware maintenance. This -includes replacing failed or about to fail disks, memory and other hardware -components. The cluster will need to be checked daily for disks in need of -replacement. Occasionally, new operating system images and software will need -to be installed on the cluster. Additionally, there will be two new lots of -disks purchased in this phase that will be passed through the IPP Storage Test -Plan. The production phase will last approximately two years. - -\subsubsection{2nd Disk Lot} - -Approximately 6 months after the IPP Cluster enters into the Production phase -the $2^{nd}$ lot of 600 storage disks will need to be ordered. It is estimated -that by this time the storage capacity will be at least 600B per disk. The new -disks shall be installed in sets of 12 on a single RAID controller in nodes -[ipp000..ipp049] and configured as a single RAID 5 volume mounted as -\file{/export/[nodename]/vol1} and exported via NFS as part of the original NFS -sharing of \file{/export/[nodename]}. The new lot of disks will also need to -be run through the ``Storage'' section of the IPP Hardware Test Plan. - -\subsubsection{3rd Disk Lot} - -Approximately 14 months after the IPP Cluster enters into the Production phase -the $3^{rd}$ and final lot of 600 disks will need to be ordered. Disk capacity -should be at least 900GB per disk by this time. The new disks shall be -installed in sets of 24 on two RAID controllers in nodes [ipp050..ipp074] and -configured as two RAID 5 volumes (one per controller) mounted as -\file{/export/[nodename]/vol0} \& \file{/export/[nodename]/vol1}. Both of -these volumes will be shared via NFS by exporting \file{/export/[nodename]} as -\code{[nodename]}. This lot will also be run through the ``Storage'' section -of the IPP Hardware Test Plan. - -\subsection{End of Life} - -The stacked static sky images will need to be preserved for transfer to the -PS4 IPP cluster. The static sky data shall be transfered to the 25 nodes -containing lot 3 disks. All but the 25 nodes containing lot 3 disks will be -deactivated. The deactivated nodes shall be stored for possible reuse with -PS4. This phase will last approximate one year. - -\section{Maintenance} - -\subsection{Nodes} - -The IPP Cluster nodes, excluding data storage disks, will be covered under a 3 -parts and labor warranty by the OEM. The significant costs to the Pan-STARRS -project will be the labors costs of installation/removal of hardware -components, the testing associated with the new hardware components, and -possibly the expense of shipping parts back to the OEM. - -The storage disks, depending on the vendor each lot is purchased from, may or -may not be under warranty from the OEM. In the case that a disk warranty is -not extended by the OEM, only disks that are covered under a warranty by the -manufacturer will be acquired. In such a case the Pan-STARRS project will be -required to pay the return shipping expenses unless special terms can be -negotiated directly with the disk manufacturer. - -\subsection{Network Equipment} - -The IPP Cluster core network switch will need to be placed under a -SmartNet\footnote{SmartNet - -http://www.cisco.com/en/US/products/svcs/ps3034/ps2827/ps2978/serv\_datasheet09186a0080092491.html} -contract from Cisco. - -\tbd{The cost scheduled depends on the level of service contract.} - -\section{Schedule} - -\begin{figure} -\begin{center} -\begin{tabular}{ll} -\cline{1-2} -Commissioning &\\ -\hspace{1cm}Stage 1 &75 days (20 days slack)\\ -\hspace{2cm}Step 1 &2 FTE, 5 days\\ -\hspace{2cm}Step 2 &2 FTE, 50 days\\ -\hspace{1cm}Stage 2 &0.1 FTE, 50 days\\ -\hspace{1cm}Stage 3 &\\ -\hspace{2cm}Step 1 &\\ -\hspace{2cm}Step 2 &1 FTE, 30 days\\ -Production &0.5+ (weekends?) FTE\\ -End of Life &0.15 FTE\\ -\cline{1-2} -\end{tabular} -\caption{ \label{manpower} Estimated Manpower} -\end{center} -\end{figure} - -\begin{figure} -\begin{center} -\resizebox{7in}{!}{ -\includegraphics[angle=-90]{pics/ps1_ipp_cluster_timeline.ps} -} -\caption{ \label{timeline} Time-line} -\end{center} -\end{figure} - -\end{document} Index: unk/doc/design/hardware_test.tex =================================================================== --- /trunk/doc/design/hardware_test.tex (revision 6007) +++ (revision ) @@ -1,395 +1,0 @@ -%\documentclass[panstarrs,psreport,12pt]{panstarrs} -\documentclass[panstarrs]{panstarrs} - -% basic document variables -\title{Pan-STARRS PS1 Image Processing Pipeline Hardware Management Plan} -\subtitle{System/Subsystem Specification} -\author{Joshua Hoblitt} -\audience{Pan-STARRS Management} -\shorttitle{IPP Hardware Management Plan} -\group{Pan-STARRS IPP Group} -\project{Pan-STARRS Image Processing Pipeline} -\organization{Institute for Astronomy} -\version{DR01} -\docnumber{PSDC-4xx-xxx} -% note the use of the docnumber & version number: -% the complete PSDC document number is given by -% \thedocnumber-\theversion - -\begin{document} -\maketitle - -% -- Revision History -- -\RevisionsStart -% version Date Description -DR00 & 2004.11.21 & First draft \\ -\hline -\RevisionsEnd - -% -- TBD / TBR Listing -- -%\TBDsStart -% section % page Date Description -%1.2.3.4 & 2 & xxx-xxx & missing a dingle on a dangle \\ -%\TBDsEnd - -\pagebreak -\tableofcontents - -%\pagebreak -%\listoffigures - -\appendix - -\section{IPP Cluster Test Plan} - -\subsection{Scope} - -\subsubsection{Identification} - -The Image Processing Pipeline Cluster hardware, including the processing nodes -and core network switch. - -\subsubsection{System overview} - -Receives, processes, and storages image data generated by the PS1 telescope -system. - -\subsubsection{Document overview} - -This document describes the commissioning test plan for the IPP Cluster -hardware. - -\subsection{Software test environment} - -\subsubsection{MHPCC} - -\subsubsection{Software} - -\subsubsection{Hardware} - -\subsubsection{Installation, testing, and control} - -\subsubsection{Personnel} - -2 FTEs. - -\subsubsection{Planned tests} - -\paragraph{Node} - -\subparagraph{physical inventory} - -\subparagraph{power supplies} - -All 3 power supplies have the green status led lit and have spinning fans. - -\subparagraph{Physical memory} - -Memory passes a complete set of memtest86+ and runs memtest86+ for atleast an -hour. - -\subparagraph{O/S boots} - -The system can boot from a Linux 2.6 based bootable cdrom without having a -kernel panic. - -\subparagraph{CPUs} - -`cat /proc/cpuinfo` show both CPUs at the proper clockrate. - -\begin{verbatim} -processor : 0 -vendor_id : AuthenticAMD -cpu family : 15 -model : 5 -model name : AMD Opteron(tm) Processor 248 -stepping : 10 -cpu MHz : 2189.901 -cache size : 1024 KB -fpu : yes -fpu_exception : yes -cpuid level : 1 -wp : yes -flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow -bogomips : 4325.37 -TLB size : 1088 4K pages -clflush size : 64 -cache_alignment : 64 -address sizes : 40 bits physical, 48 bits virtual -power management: ts fid vid ttp - -processor : 1 -vendor_id : AuthenticAMD -cpu family : 15 -model : 5 -model name : AMD Opteron(tm) Processor 248 -stepping : 10 -cpu MHz : 2189.901 -cache size : 1024 KB -fpu : yes -fpu_exception : yes -cpuid level : 1 -wp : yes -flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow -bogomips : 4374.52 -TLB size : 1088 4K pages -clflush size : 64 -cache_alignment : 64 -address sizes : 40 bits physical, 48 bits virtual -power management: ts fid vid ttp -\end{verbatim} - -\subparagraph{Memory} - -`cat /proc/meminfo` shows the full ammount of memory. - -\begin{verbatim} -MemTotal: 4026236 kB -MemFree: 3140404 kB -Buffers: 277248 kB -Cached: 55476 kB -SwapCached: 0 kB -Active: 301168 kB -Inactive: 34536 kB -HighTotal: 0 kB -HighFree: 0 kB -LowTotal: 4026236 kB -LowFree: 3140404 kB -SwapTotal: 16008764 kB -SwapFree: 16008764 kB -Dirty: 16 kB -Writeback: 0 kB -Mapped: 5576 kB -Slab: 532316 kB -Committed_AS: 9380 kB -PageTables: 448 kB -VmallocTotal: 536870911 kB -VmallocUsed: 8672 kB -VmallocChunk: 536862139 kB -HugePages_Total: 0 -HugePages_Free: 0 -Hugepagesize: 2048 kB -\end{verbatim} - -\subparagraph{PCI} - -`lspci` identifies all of the appropriate hardware. - -\begin{verbatim} -0000:00:06.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8111 PCI (rev 07) -0000:00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-8111 LPC (rev 05) -0000:00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-8111 IDE (rev 03) -0000:00:07.2 SMBus: Advanced Micro Devices [AMD] AMD-8111 SMBus 2.0 (rev 02) -0000:00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-8111 ACPI (rev 05) -0000:00:0a.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) -0000:00:0a.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01) -0000:00:0b.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8131 PCI-X Bridge (rev 12) -0000:00:0b.1 PIC: Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC (rev 01) -0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration -0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map -0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller -0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control -0000:00:19.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration -0000:00:19.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map -0000:00:19.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller -0000:00:19.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control -0000:01:03.0 RAID bus controller: 3ware Inc 3ware Inc 3ware 9xxx-series SATA-RAID -0000:02:03.0 RAID bus controller: 3ware Inc 3ware Inc 3ware 9xxx-series SATA-RAID -0000:02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03) -0000:02:09.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 03) -0000:03:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) -0000:03:00.1 USB Controller: Advanced Micro Devices [AMD] AMD-8111 USB (rev 0b) -0000:03:05.0 Unknown mass storage controller: Silicon Image, Inc. (formerly CMD Technology Inc) SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02) -0000:03:06.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) -0000:03:08.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 10) -\end{verbatim} - -\subparagraph{Internal Harddisk} - -`fdisk -l` shows the internal hard disk - -\begin{verbatim} -Disk /dev/sda: 250.0 GB, 250059350016 bytes -255 heads, 63 sectors/track, 30401 cylinders -Units = cylinders of 16065 * 512 = 8225280 bytes - - Device Boot Start End Blocks Id System -/dev/sda1 1 13 104391 83 Linux -/dev/sda2 14 2006 16008772+ 82 Linux swap / Solaris -/dev/sda3 2007 3999 16008772+ 83 Linux -/dev/sda4 4000 30401 212074065 83 Linux -\end{verbatim} - -\subparagraph{Network} - -both gige interfaces can be configured and a test device pinged. - -\subparagraph{RAID controllers} - -`tw\_cli` shows both RAID controllers and reports there status as OK with no -alarms reported. - -\begin{verbatim} -//alae> alarms - -Ctl Date Severity Alarm Message ------------------------------------------------------------------------------- - -//alae> info c0 u0 - -Unit UnitType Status %Cmpl Port Stripe Size(GB) Blocks ------------------------------------------------------------------------ -u0 RAID-5 OK - - 64K 1629.74 3417817088 -u0-0 DISK OK - p0 - 232.82 488259584 -u0-1 DISK OK - p1 - 232.82 488259584 -u0-2 DISK OK - p2 - 232.82 488259584 -u0-3 DISK OK - p3 - 232.82 488259584 -u0-4 DISK OK - p4 - 232.82 488259584 -u0-5 DISK OK - p5 - 232.82 488259584 -u0-6 DISK OK - p6 - 232.82 488259584 -u0-7 DISK OK - p7 - 232.82 488259584 - -//alae> info c1 u0 - -Unit UnitType Status %Cmpl Port Stripe Size(GB) Blocks ------------------------------------------------------------------------ -u0 RAID-5 OK - - 64K 1629.74 3417817088 -u0-0 DISK OK - p0 - 232.82 488259584 -u0-1 DISK OK - p1 - 232.82 488259584 -u0-2 DISK OK - p2 - 232.82 488259584 -u0-3 DISK OK - p3 - 232.82 488259584 -u0-4 DISK OK - p4 - 232.82 488259584 -u0-5 DISK OK - p5 - 232.82 488259584 -u0-6 DISK OK - p6 - 232.82 488259584 -u0-7 DISK OK - p7 - 232.82 488259584 -\end{verbatim} - -\paragraph{Network hardware} - -\subparagraph{physical inventory} - -\subparagraph{Memory} - -`sh version` reports the proper ammount of memory. - -\begin{verbatim} -#sh ver -Cisco Internetwork Operating System Software -IOS (tm) Catalyst 4000 L3 Switch Software (cat4000-I9K2S-M), Version 12.1(20)EW1, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) -TAC Support: http://www.cisco.com/tac -Copyright (c) 1986-2003 by cisco Systems, Inc. -Compiled Mon 01-Dec-03 19:45 by eaarmas -Image text-base: 0x00000000, data-base: 0x0106BC90 - -ROM: 12.1(12r)EW -Dagobah Revision 86, Swamp Revision 28 - -goat uptime is 43 weeks, 1 day, 5 hours, 42 minutes -System returned to ROM by reload -System restarted at 12:12:06 US/Hawaii Sun Jan 4 2004 -System image file is "bootflash:cat4000-i9k2s-mz.121-20.EW1.bin" - - -This product contains cryptographic features and is subject to United -States and local country laws governing import, export, transfer and -use. Delivery of Cisco cryptographic products does not imply -third-party authority to import, export, distribute or use encryption. -Importers, exporters, distributors and users are responsible for -compliance with U.S. and local country laws. By using this product you -agree to comply with applicable laws and regulations. If you are unable -to comply with U.S. and local laws, return this product immediately. - -A summary of U.S. laws governing Cisco cryptographic products may be found at: -http://www.cisco.com/wwl/export/crypto/tool/stqrg.html - -If you require further assistance please contact us by sending email to -export@cisco.com. - -cisco WS-C4506 (XPC8245) processor (revision 7) with 524288K bytes of memory. -Processor board ID FOX073501MS -Last reset from Reload -5 Ethernet/IEEE 802.3 interface(s) -1 FastEthernet/IEEE 802.3 interface(s) -242 Gigabit Ethernet/IEEE 802.3 interface(s) -403K bytes of non-volatile configuration memory. - -Configuration register is 0x2102 -\end{verbatim} - -\subparagraph{Modules} - -`sh module` shows all the modules with status OK - -\begin{verbatim} -Chassis Type : WS-C4506 - -Power consumed by backplane : 10 Watts - -Mod Ports Card Type Model Serial No. ----+-----+--------------------------------------+------------------+----------- - 1 2 1000BaseX (GBIC) Supervisor(active) WS-X4515 JAE07400KBU - 2 48 10/100/1000BaseTX (RJ45) WS-X4448-GB-RJ45 JAB073904TB - 3 48 10/100/1000BaseTX (RJ45) WS-X4448-GB-RJ45 JAB073904M5 - 4 48 10/100/1000BaseTX (RJ45) WS-X4448-GB-RJ45 JAB06170832 - 5 48 10/100/1000BaseTX (RJ45) WS-X4448-GB-RJ45 JAB0617083X - 6 48 10/100/1000BaseTX (RJ45)V WS-X4548-GB-RJ45V JAB082109AX - - M MAC addresses Hw Fw Sw Status ---+--------------------------------+---+------------+----------------+--------- - 1 000d.ed4a.ac80 to 000d.ed4a.ac81 3.0 12.1(12r)EW 12.1(20)EW1, EAR Ok - 2 000d.bc8d.8360 to 000d.bc8d.838f 1.1 Ok - 3 000d.bc8d.7b50 to 000d.bc8d.7b7f 1.1 Ok - 4 0005.5fa4.38f0 to 0005.5fa4.391f 1.0 Ok - 5 0005.5fa4.3950 to 0005.5fa4.397f 1.0 Ok - 6 0011.20ea.bb00 to 0011.20ea.bb2f 2.0 Ok -\end{verbatim} - -\subparagraph{Diagnostics} - -`sh diag power-on` - -\begin{verbatim} - -Power-On-Self-Test Results for ACTIVE Supervisor - - -Power-on-self-test for Module 1: WS-X4515 - Port/Test Status: (. = Pass, F = Fail) - Reset Reason: Software/User - - -Port Traffic: L2 Serdes Loopback ... - 0: . 1: . 2: . 3: . 4: . 5: . 6: . 7: . 8: . 9: . 10: . 11: . -12: . 13: . 14: . 15: . 16: . 17: . 18: . 19: . 20: . 21: . 22: . 23: . -24: . 25: . 26: . 27: . 28: . 29: . 30: . 31: . - - -Port Traffic: L2 Asic Loopback ... - 0: . 1: . 2: . 3: . 4: . 5: . 6: . 7: . 8: . 9: . 10: . 11: . -12: . 13: . 14: . 15: . 16: . 17: . 18: . 19: . 20: . 21: . 22: . 23: . -24: . 25: . 26: . 27: . 28: . 29: . 30: . 31: . - - -Port Traffic: L3 Asic Loopback ... - 0: . 1: . 2: . 3: . 4: . 5: . 6: . 7: . 8: . 9: . 10: . 11: . -12: . 13: . 14: . 15: . 16: . 17: . 18: . 19: . 20: . 21: . 22: . 23: . -24: . 25: . 26: . 27: . 28: . 29: . 30: . 31: . au: . - - -Switch Subsystem Memory ... - 1: . 2: . 3: . 4: . 5: . 6: . 7: . 8: . 9: . 10: . 11: . 12: . -13: . 14: . 15: . 16: . 17: . 18: . 19: . 20: . 21: . 22: . 23: . 24: . -25: . 26: . 27: . 28: . 29: . 30: . 31: . 32: . 33: . 34: . 35: . 36: . -37: . 38: . 39: . 40: . 41: . 42: . 43: . 44: . 45: . 46: . 47: . 48: . -49: . 50: . 51: . 52: . 53: . 54: . 55: . - - -Module 1 Passed -\end{verbatim} - -\paragraph{Management hardware} - -\pagebreak -\section{IPP Storage Test Plan} - -\end{document} Index: /trunk/doc/design/ippSCD.tex =================================================================== --- /trunk/doc/design/ippSCD.tex (revision 6008) +++ /trunk/doc/design/ippSCD.tex (revision 6008) @@ -0,0 +1,3546 @@ +%%% $Id: ippSCD.tex,v 1.1 2006-01-16 00:39:24 eugene Exp $ +\documentclass[panstarrs]{panstarrs} + +% basic document variables +\title{Pan-STARRS PS-1 Image Processing Pipeline} +\subtitle{System/Subsystem Design Description} +\shorttitle{IPP SSDD} +\author{Eugene A. Magnier, Paul A. Price, Josh Hoblitt} +\audience{Pan-STARRS PMO} +\group{Pan-STARRS Algorithm Group} +\project{Pan-STARRS Image Processing Pipeline} +\organization{Institute for Astronomy} +\version{00} +\docnumber{PSDC-430-011} + +% allow paragraphs to be listed in TOC for now +\setcounter{tocdepth}{4} + +\begin{document} +\maketitle + +% -- Revision History -- +\RevisionsStart +% version Date Description +DR.01 & 2004.01.01 & First draft \\ \hline +DR.02 & 2004.03.05 & Second draft \\ \hline +DR.03 & 2004.03.25 & Section reorganization \\ \hline +DR.04 & 2004.04.13 & Most sections fleshed out \\ \hline +DR.05 & 2004.04.29 & Reorganization for consistency \\ \hline +DR.06 & 2004.10.21 & Major revision in prep of PDR \\ \hline +\RevisionsEnd + +\inserttbd +\inserttbr +\pagebreak + +\tableofcontents +\pagebreak + +\listoffigures +\pagebreak +\pagenumbering{arabic} + +\section{Scope} + +\subsection{Identification} + +This document establishes Software Design Requirements for the +Panoramic Survey Telescope and Rapid Response System (Pan-STARRS) +Image Processing Pipeline (IPP) for the prototype telescope PS-1, and +is a System-level controlled specification/design description document +in the official Pan-STARRS engineering specification tree. + +\subsection{System Overview} + +The Institute for Astronomy at the University of Hawaii is developing +a large optical synoptic survey telescope system, the Panoramic Survey +Telescope and Rapid Response System (Pan-STARRS). The science goals, +priorities, top-level concept of operations with associated +operational requirements, and system performance drivers with +associated system performance requirements are described in the +Pan-STARRS Science Goals Statement (SGS). As described in this +document, The system conceptual design for Pan-STARRS utilizes an +array of four 1.8m telescopes each with a 7 degree$^2$ field of view, +giving the system an \'etendue larger than all existing survey +instruments combined (defined as the product of the collecting area +$A$ multiplied by the field-of-view solid angle $\Omega$). Each +telescope will be equipped with a 1 billion pixel CCD camera with low +noise and rapid read-out, and the data will be reduced in near real +time to produce both cumulative static sky and difference images from +which transient, moving, and variable objects can be +detected. Pan-STARRS will be able to survey up to $\approx 6,000$ +degree$^{2}$ per night to a detection limit of approximately 24$^{th}$ +magnitude. This unique combination of sensitivity and sky coverage +will open up many new possibilities in time domain astronomy including +a major goal of surveying the Potentially Hazardous Object (PHO) +population down to a diameter of $\approx 300$ meters. In addition, +the Pan-STARRS data will be used to investigate a broad range of +astronomical problems of extreme current interest concerning the Solar +System, the Galaxy, and the Cosmos at large. A prototype single +telescope system, PS-1, is being developed as a preliminary step +before construction of the complete four telescope system. + +\begin{tabular}{ll} +Project sponsor:& AFRL, United States Air Force \\ +Acquirer: & University of Hawaii Institute for Astronomy \\ +User: & Astronomical community \\ +Developer: & University of Hawaii Institute for Astronomy, participating \\ + & institutions, and associated subcontractors +\end{tabular} + +\subsection{Document Overview} + +The Pan-STARRS IPP System/Subsystem Design Description (SSDD) contains +the complete design description of the Pan-STARRS PS-1 IPP in order to +achieve the requirements specified by the Pan-STARRS PS-1 IPP Software +Requirements Specification (SRS; PSDC-430-005). The requirements flow +begun in the SGS and SCD and continued in the SRS is used to guide the +design presented here. + +\subsection{Requirements Definitions} + +The Pan-STARRS document naming scheme is PSDC-NNN-MMM-VV, where the VV +entry specifies the document version number. Where documents are +identified without the version number, the latest official version in +that series is implied. + +Open issues (TBDs) in this document are marked {\bf \color{red} in +bold red}. + +Quantities which should be reviewed (TBRs) are marked {\bf +\color{blue} in bold blue}. + +\subsubsection{``Shall''} When used in this specification, the word +``shall'' refers to an explicit requirement of a system component or +the complete system. + +\subsubsection{``Should''} When used in this specification, the word +``should'' refers to a desired characteristic of a system component or +the complete system. + +\subsubsection{``Will''} When used in this specification, the word +``will'' provides information about a characteristic of a related +system component or a complete related system. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\DocumentsInternalSection +PSDC-230-001 & PS-1 Design Reference Mission \\ \hline +PSDC-230-002 & PS-1 System Concept Definition \\ \hline +PSDC-400-006 & The Pan-STARRS IPP Computational Challenge \\ \hline +PSDC-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline +PSDC-430-005 & Pan-STARRS IPP PS-1 Software Requirements Specification \\ \hline +PSDC-430-006 & Pan-STARRS IPP Algorithm Design Document \\ \hline +PSDC-430-007 & Pan-STARRS IPP PSLib Supplementary Design Requirements Specification \\ \hline +PSDC-430-010 & Pan-STARRS IPP Perl Code Conventions \\ \hline +PSDC-430-012 & Pan-STARRS IPP Modules Supplementary Design Requirements Specification \\ \hline +PSDC-430-014 & Pan-STARRS IPP PS-1 Cluster Support \\ \hline +\DocumentsExternalSection +Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ +\DocumentsEnd + +\section{Subsystem Overview} + +The Pan-STARRS Image Processing Pipeline (IPP) performs the image +processing and data analysis tasks needed to enable the scientific use +of the images obtained by the Pan-STARRS telescopes. The primary +goals of the IPP are to process the science images from the Pan-STARRS +telescopes and make the results available to other systems within +Pan-STARRS. It also is responsible for combining all of the science +images in a given filter into a single representation of the +non-variable component of the night sky defined as the ``Static Sky''. +To achieve these goals, the IPP also performs other analysis functions +to generate the calibrations needed in the science image processing +and to occasionally use the derived data to generate improved +astrometric and photometric reference catalogs. It also provides the +infrastructure needed to store the incoming data and the resulting +data products. + +The IPP inherits lessons learned, and in some cases code and prototype +code, from several other astronomy image analysis systems, including +Imcat (Kaiser), the Sloan Digital Sky Survey (REF), the Elixir system +(Magnier \& Cuillandre), and Vista (Tonry). Imcat and Vista have a +large number of robust image processing functions. SDSS has +demonstrated a working analysis pipeline and large-scale database +system for a dedicated project. The Elixir system has demonstrated an +automatic image processing system and an object database system for +operational usage. + +The users of the IPP output are all systems internal to the Pan-STARRS +project. They consist of: 1) the Preferred Science Clients, which +receive specified data products on short timescales. 2) the Moving +Object Processing System (MOPS), which is one of the Preferred Science +Clients, but has the distinction of being a component funded by +Pan-STARRS. It will receive the detections of non-stationary +transient objects. 3) the Published Science Products Subsystem +(PSPS), which will receive all data products of interest to the +community external to the Pan-STARRS data processing systems, and will +act as the long-term archive and publishing clearinghouse. + +The IPP receives data from two Pan-STARRS subsystems: the Camera, from +which it receives the large volume of image data, and OTIS +(Observatory, Telescope and Infrastructure Subsystem), from which it +receives metadata describing the images and the environmental +conditions. The primary IPP hardware system on which the software +operates will probably not be located at the summit. Instead, because +of thermal, power, and space constraints, the hardware will likely be +located in a facility off the mountain. A subset of processing tasks +may eventually be assigned to machines at the summit if justified by +the savings in data transfer time and cost. + +The Pan-STARRS camera produces images consisting of multiple chips +(Orthogonal Transfer Arrays or OTAs), each consisting of multiple +cells (continuous set of pixels). The baseline design for the +Pan-STARRS camera contains 64 chips each with 64 cells. + +This document defines the design requirements of the IPP for the PS-1 +prototype telescope. Much of the IPP design for PS-4 will be +identical to or closely based on the PS-1 implementation. The +software organization and the infrastructure systems are expected to +be identical, with minor improvements in details. The type of +analysis steps to be performed will be nearly identical, with some +additional details added for PS-4 to improve the accuracy. + +Although generally very similar, in terms of the IPP PS-1 differs from +the complete PS-4 system in several specific ways. First, with only +one telescope and camera, the data throughput rate is substantially +reduced to a maximum of 1 64-OTA image per 40 seconds rather than 4. +Since PS-1 is a prototype for testing the Pan-STARRS hardware and +software subsystems, the observing strategy is not a fixed quantity. +The PS-1 Design Reference Mission (PSDC-230-001) provides some +guidelines for the types of observing tests which will probably be +performed, including possibly starting an Astrometric and Photometric +Survey which will eventually cover the entire $3\pi$ steradians of the +sky accessible to PS-4. As a prototype, it is expected that much of +the data collected by PS-1 will be processed multiple times to test +and tune the analysis steps. Compare with PS-4, this difference in +approach has implications for the storage required by PS-1: rather +than delete images soon after they have been used, raw images from +demonstration observations must be stored for at least the first two +years of PS-1 operations. The PS-1 Design Reference Mission is used +as an upper limit for these storage requirements to drive the hardware +design. + +\subsection{System Design Decisions} + +Since Pan-STARRS is a survey project, all data from the telescopes +will be uniformly analyzed by the Pan-STARRS Image Processing Pipeline +(IPP), and the appropriate resulting data products made available to +internal and external science analysis systems as they become +available. The processing performed by the IPP on the science images +will consist of detrending and object detection for the individual +images, combination of multiple overlapping images and further object +detection, subtraction of a reference (static-sky) image and detection +of residual objects, update of the static sky images, and detailed +object analysis of the static sky images. In addition, the IPP will +produce improved astrometric and photometric reference catalogs on an +as-needed basis. The output data products from the IPP consist of the +calibration images, reduced images from the individual telescopes, +combined images, difference images, the static sky image, object +photometry, and reference astrometry and photometry. + +The requirements for the IPP, as identified in the PS-1 IPP SRS +(PSDC-430-005) fall into several broad categories: data analysis +precision, throughput, system reliability, flexibility, testability, +and traceability. The details of the analysis tasks are specified in +order to achieve the precision. The architectural design as discussed +below is motivated by the need for reliability and flexibility. The +hardware organization and the distributed/parallel processing model is +motivated by the throughput requirements. The need for flexibility +and testability drives the software organization. The need for simple +testing procedures drives both the software organization and the +separation of the system architecture into different infrastructure +elements. + +\subsection{Analysis Tasks and Stages} + +Specific programs are required to perform the processing steps listed +above. These can be divided into well-defined analysis stages, each +of which operates on a particular unit of data, such as a single OTA +image or a collection of astronomical objects. Analysis tasks +representing the different analysis stages are performed on the IPP +computer cluster. Note the distinction between the generic analysis +{\em stage} and a specific analysis {\em task}. An analysis stage +represents a type of analysis which is performed, such as the basic +image calibration and object detection analysis. An analysis task is +a particular realization of an analysis stage, e.g., the analysis of +OTA number 61 from exposure 654321 to produce a specific set of output +data products. The analysis stages are discussed in detail in +Section~\ref{sec:AnalysisStages}. + +A particular stage may process individual images, collections of +images, or derived data products. Because of the nature of the image +data, many of the analysis stages can be run in parallel if needed to +increase the processing throughput. For example, the analysis of a +chip in one image does not depend on the results from another chip. + +\subsection{Architectural Components} + +\begin{figure} +\begin{center} +\resizebox{6in}{!}{\includegraphics{pics/IPPoverview}} +\caption{ \label{fig:overview} IPP System Overview} +\end{center} +\end{figure} + +In order to achieve the required functionality, the IPP provides an +infrastructure within which the Analysis Stages described above are +executed. In order to facilitate the subsystem testing, the IPP +software infrastructure has been divided into a number of +clearly-defined architectural software units as follows: + +\begin{itemize} + +\item {\bf Image Server:} This component is a large data store for all + images used by the IPP, including the raw images from the telescope, + the master calibration images, the reference static-sky images, and + any temporary image data products produced by the IPP. The Image + Server accepts the incoming data and stores it until it is no longer + needed by other portions of the IPP. The Image Server is not + restricted to imaging data: it is capable of storing any large data + files which are not well-suited for inclusion in a more structured + relational database, and for which access needs to be widely + available beyond the individual process which created the file. + +\item {\bf Metadata Database:} This component stores the data which is + not directly related to images or astronomical objects, but which is + needed to perform the IPP analyses. The metadata may include the + summary weather information for each night, or details about the + filters, camera, telescopes, etc. Note that the IPP Metadata + Database is not required to retain all archival engineering data + from all of Pan-STARRS; other Pan-STARRS subsystems use their own + internal databases to store engineering metadata and only the + necessary subset is transferred to the IPP Metadata Database. + +\item {\bf Astrometry \& Photometry Database (AP DB):} This component + stores and manipulates astronomical objects detected in various + images, as identified above, including individual measurements of + objects on the images, the summary information about those objects, + and reference object data. It also provides mechanisms for users to + query and manipulate the objects and detections. + +\item {\bf IPP Controller:} In order to achieve the required + processing throughput for the IPP analysis stages, it is necessary + to use distributed computing processes on a large number of + computers. The IPP Controller manages the collection of analysis + tasks performed on these machines. + +\item {\bf IPP Scheduler:} This component is a decision-making + mechanism which guides the operation of the IPP. It evaluates the + currently available collection of data, identifies the necessary + analysis, and assigns the analysis tasks to the IPP Controller. + +\end{itemize} + +The relationship between these software units is shown in +Figure~\ref{fig:overview}. This figure also shows the interactions +between the IPP and other Pan-STARRS systems. The architectural +components are discussed in detail in Section~\ref{sec:ArchComponents}. + +\begin{figure} +\begin{center} +\resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}} +\caption{ \label{fig:hardware} IPP Hardware Organization} +\end{center} +\end{figure} + +\subsection{IPP Hardware Organization} + +The IPP will utilize substantial computer resources, both in terms of +computational power and in terms of data storage and network +bandwidth. The IPP requires relatively large amounts of data storage +space, primarily for the image data. Image data is organized in two +categories. First, there is the per-OTA data -- data associated with +specific OTAs, including the raw images, the calibration images, and +temporary processed images at various stages. Second, there is the +data associated with the static sky imagery, which is in turn +organized into smaller sky-cell units. In addition to image data, +there are the somewhat smaller data entities of the Metadata Database +and AP Database. + +The computer hardware is organized into nodes which provide both data +storage and computational resources. The data storage nodes are +divided into three classes: those which deal with the per-OTA image +data, those that provide the storage for the static sky images, and +those that provide the storage for the other data systems, the +Metadata Database and the AP Database. In addition, the computational +tasks related to the individual images take place on the per-OTA +storage nodes and the processing of stacks of images takes place on +the static sky storage nodes. + +Figure~\ref{fig:hardware} presents the basic concept for the hardware +organization for the IPP. This diagram shows the two types of compute +nodes: (1) OTA-level processing and storage nodes and (2) Static Sky +processing and storage nodes. Also shown are two switches which +divide the network into OTA and Static-Sky portions. In such an +organization, the inter-switch communication must meet the throughput +needs between these network portions (though a single switch may also +be used if its backplane capacity is sufficient). The additional data +systems (Metadata Database and AP Database) are also shown. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{System Design : Architectural Components} +\label{sec:ArchComponents} + +\subsection{IPP Image Server} + +\subsubsection{Corresponding Requirements} + +The Image Server must meet the requirements specified in Section 3.4.1 +of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The specified design +is chosen to meet requirements 3.4.1.3, and 3.4.1.5. The other three +requirements (3.4.1.1, 3.4.1.2, and 3.4.1.4) depend on the volume and +capabilities of the hardware, and are addressed in +Section~\ref{sec:Hardware}. + +\subsubsection{Image Server Overview} + +The IPP Image Server is a repository for all images and other large +data files required by the IPP. Along with the storage hardware, it +provides tools for managing the distribution of these large data files +and for accessing the files. Data files stored by the IPP Image +Server include the raw images, the calibration images, intermediate +processing stage images as needed, final processed images, difference +images, image subsections, and any large non-imaging data files needed +by the IPP. The IPP Image Server must retain the files for as long as +they are needed by the IPP. + +The IPP Image Server is a parallel storage system. It stores data +across a collection of computer nodes, each with their own data +storage resources. Any single file is stored on only a single +computer and storage device. In order to achieve the data throughput +requirements, the IPP Image Server may distribute the images across +the processor nodes in an organized fashion, i.e., associating +specific machines with specific detectors. It is not the +responsibility of the IPP Image Server to determine which computer +should be associated with a specific data concept (Chip / region of +sky), but it must enable the association of a particular file with a +particular machine. + +There are three data concepts relevant to the IPP Image Server: +\begin{itemize} +\item {\bf Storage object:} This represents a single, unique data + entity in the Image Server. + +\item {\bf Instance:} A single copy of the storage object in the Image + Server. In general, a given storage object may have several instances + in the Image Server, normally on different computer nodes. + +\item {\bf File ID:} This is the identifier of a particular storage + object in the Image Server. The file ID is simply a unique string, + equivalent to the filename in a UNIX file system. +\end{itemize} + +The Image Server provides file pointers (in C), handles (in Perl or +Python), or file names corresponding to the instances of the storage +objects. The Image Server provides the data organization but does not +define a file system; it assumes the existence of an appropriate file +system which makes the files visible as local files. This +may be done over many machines with a network file system such as NFS +or GFS. + +The IPP Image Server provides the storage and access mechanisms, but +it does not include any logic or information about the data. The +Image Server does not, e.g., monitor the age of images and delete them +on some schedule. + +As shown in Figure~\ref{fig:ImageServer}, the IPP Image Server +consists of the following components: + +\begin{itemize} +\item Image Server storage hardware +\item Image Server database +\item Image Server daemon +\item Image Server client APIs +\item Image Server maintenance tools (not shown) +\end{itemize} + +\begin{figure} +\begin{center} +\resizebox{4.5in}{!}{\includegraphics{pics/ImageServer}} +\caption{The components of the IPP Image Server.} +\label{fig:ImageServer} +\end{center} +\end{figure} + +\subsubsection{IPP Image Server Client APIs} + +Clients interact with the IPP Image Server via a small number of C +APIs. Bindings are also provided for Perl and Python and UNIX shell +commands in some cases. The client commands are: + +\begin{itemize} +\item {\tt new object}: create a new storage object in the Image + Server. This function takes as input the file ID and returns a + C-style file pointer or a Perl file handle to the instance of the + storage object. The arguments to the function include an optional + node name on which the new storage object must be located. If this + target is not given, the Image Server places the new storage object + on an appropriate machine from the pool, though the details need to + be specified. + +\item {\tt open object}: open an instance of an existing storage + object, as identified by the file ID. This function may also + specify the node on which the object should be opened (if an + instance of the object is not stored on that node, the function + returns an error). On success, the function returns a file pointer. + +\item {\tt find object}: return a list of filenames in the UNIX name + space associated with the storage object identified by the given + file ID. Since there are in general multiple instances for a given + storage object, this function returns the collection of all + available instances. These may be freely opened by the client + server using the standard \code{fopen} functions. + +\item {\tt stat object}: returns status information about the + specified storage object, including the number of instances of the object. + +\item {\tt replicate object}:a new instance of the given storage + object. The target node may be optionally specified, otherwise an + appropriate node is selected. + +\item {\tt cull object}: removes one of the instances of the storage + object. The input parameters may optionally specify the target + machine to delete. + +\item {\tt delete object}: deletes all instances of the storage object + and sets the storage object status to {\tt deleted}. +\end{itemize} + +\subsubsection{IPP Image Server Daemon} + +The Image Server client requests are mediated via the Image Server +daemon. Communication between the clients and the server is via SOAP +implementing the commands above. The identity of the machine on which +Image Server daemon runs is part of the Image Server configuration +information. + +\subsubsection{IPP Image Server Database} + +The IPP Image Server daemon uses a database to store the information +about the data storage objects, their instances, and the available +hardware resources. A {\tt mysql} database engine is used to manage +the database table. The database tables defined for the Image Server +are listed in Table~\ref{tab:ImageServerTables}, and their contents are +listed in Appendix~\ref{sec:ImageServerTableContents}. This database +engine need not be the same one used for other IPP subsystems. +% +\begin{table}[ht] +\begin{center} +\caption{Image Server Database Tables\label{tab:ImageServerTables}} +\begin{tabular}{ll} +\hline +\hline +{\bf Table Name} & {\bf Description} \\ +\hline +\code{storage_object} & all storage objects known to Image Server \\ +\code{instance} & all instances of all storage objects \\ +\code{volume} & data storage devices known to Image Server \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\subsubsection{IPP Image Server Storage Hardware} + +The IPP Image Server manages data across a collection of computers and +possibly on multiple storage devices on those computer nodes. The +Image Server maintains a table of the available data volumes. The +Image Server tracks information about each volume such as the total +capacity, the current capacity, the association between computer and +data volume. + +\subsubsection{IPP Image Server Maintenance Tools} + +The IPP Image Server provides a collection of administration tools +which allow for maintenance. These are operations which may be +automatically scheduled by the IPP or which may be initiated by a +human via a command-shell interface. The maintenance functions +include migrating data between nodes to re-balance the available space +(this would only occur for instances which have not been placed on a +specific node by the client API). Other functions include checking +for file corruption, which involves sweeping all files on a data +volume and comparing the calculated file checksum to the currently +recorded value. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Metadata Database} +\label{sec:Metadata} + +\subsubsection{Corresponding Requirements} + +The Metadata Database must meet the requirements specified in Section +3.4.2 of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The specified +design is chosen to meet requirements 3.4.2.1, 3.4.2.2, 3.4.2.3, +3.4.2.4, 3.4.2.5. + +\subsubsection{Overview} + +The IPP Metadata Database acts as a repository for non-pixel data +needed by the IPP subsystems. This includes the image metadata, the +environmental data, system configuration data and system reference +data. The Metadata Database is required to save the non-ephemeral +data for the lifetime of the project for future reference and +additional analysis. The Metadata Database may be used in close +coupling with the analysis pipelines to store temporary data either +within or between stages of the analysis. In this scenario, the +analysis pipeline will interact directly with the database. However, +database latency may make this scenario impractical, in which case the +database may be used for long-term storage only. In this scenario, +the data produced by analysis pipelines which is destined for the +Metadata Database may be collected and inserted by a separate, +dedicated process. Metadata which is large in volume or poorly +structured may also be stored in an appropriate container file (FITS +Table, FITS Header, XML File) in the Image Server with the Metadata DB +providing pointers to these files. + +The IPP Metadata Database is a simple database system, consisting of a +number of simple tables without extensive inter-table links. The +\code{mysql} database engine will be used to drive the database. + +\begin{table}[hb] +\begin{center} +\caption{Metadata Database Tables\label{tab:MetadataDBTables}} +\begin{tabular}{ll} +\hline +\hline +{\bf Table Name} & {\bf Description} \\ +\hline +Weather & Details on the weather, including internal temperatures. \\ +SkyProbe Transparency & Analysis of SkyProbe B \& V data. \\ +SkyProbe Absorption & Analysis of SkyProbe A data. \\ +SkyProbe Emission & Analysis of SkyProbe E data. \\ +DIMM & Summary of DIMM data analysis. \\ +NIR & Summary statistics from NIR camera. \\ +Dome Status & The time history of the dome status. \\ +Telescope Status & The time history of the telescope status. \\ +Raw FPAs & Information about the raw FPA exposures. \\ +Pending Science Chips & Science images to be processed and status. \\ +Processed Science Chips & Science images which have been migrated to the processed state. \\ +Observation Group & Details about a group of associated observations. \\ +Observation Frame & Major frame information. \\ +Science Processing stats & Details on processed cells. \\ +Chip / Sky overlaps & List of overlaps between sky cells and detectors. \\ +Processed Sky-Cell stats & Details of the sky cell processing. \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\subsubsection{Metadata Tables} + +Table~\ref{tab:MetadataDBTables} lists the Metadata tables identified to +date for the Metadata Database. The contents of these tables are +outlined in Appendix~\ref{sec:MetadataTableContents}, with examples for +the data entries and their data types in many cases. Additional +tables will be added as necessary as the data analysis scripts are +fleshed out in detail. The Metadata Database, with a flat data +organization, is flexible enough to add additional information as it +is identified. + +\subsubsection{Metadata Queries} + +The IPP provides simple queries to the Metadata Database tables using +auto-coded APIs. These queries return a single row or a collection of +rows based on the primary key. The format of the API is identical for +all Metadata tables. New tables and APIs can be added to the IPP +system by adding to the auto-code table description files. The +auto-code API includes read and write access permissions to be set +for each table independently. See Appendix~\ref{sec:AutocodeIO} for +further information. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{AP Database} + +\subsubsection{Corresponding Requirements} + +The AP Database must meet the requirements specified in Section 3.4.3 +of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The specified design +is chosen to meet requirements 3.4.3.1 and 3.4.3.2. In order to meet +the throughput requirements, the AP Database will be distributed +across 10 Nodes independent of the Image Server Nodes. An alternative +organization of the database which will be studied will have the AP +Database co-located with the Image Server Phase 4 Nodes. + +\subsubsection{Overview} + +The AP (Astrometry \& Photometry) Database is a CSCI which stores data +related to astronomical objects derived from various sources with a +variety of associations. The AP Database deals with two related +concepts: {\em objects} and {\em detections}. The {\em objects} are +descriptions of astronomical objects while the {\em detections} are +the specific measurements of those objects, typically measured from +astronomical images. A collection of {\em detections} may be used to +derive average quantities which describe a particular {\em object}. A +third class of measurement to be considered are those supplied by +external references. Such measurements may be treated as {\em +detections}, with the caveat that access to the raw measurements and +metadata are usually unavailable: the reported measurements and errors +must be accepted as they are reported. + +The AP Database stores the collections of detections which were +derived from specific images from any of the analysis stages. It +provides a mechanism to determine the image from which a specific +detection was derived, and in conjunction with the Image Server locate +the corresponding data file. The AP Database also makes it possible +to extract all detections derived from a specific image and to +determine quantities such as the pixel coordinates of the detection on +the image. + +The AP Database also has the capability to associate multiple +detections of a specific object. Several major classes of objects +will be present, each of which must be handled correctly. + +First, the most distant stars, compact galaxies, and QSOs will have +nearly fixed locations relative to other distant stars, with only +small deviations for individual measurements. The association between +multiple detections of such objects is made on the basis of their +coincident positions. The AP Database determines the average position +of the object and the deviations of the individual detections from +that average on the basis of the ensemble of individual detection. + +Second, solar system objects do not have a fixed location. Detections +of such objects are linked by their orbits, and depend on both the +position and the time of the image. The AP Database does not attempt +to make this link; this is the role of the MOPS system. However, it +has the ability to accept identifications made externally with +specified detections and to return the identifier of the moving object +associated with the specific detections. These associations also +include descriptive information such as the offset of the detection +from the predicted location of the detection based on the orbit. This +functionality is required to allow the AP Database to ignore known +moving object detections from other types of queries. + +Third, objects in the general vicinity of the solar system fall in +between these first two classes of objects. Their proper motion and +parallax response is significant enough ($>0.2$ arcsec in 1 year) that +they are not well-described by an average location and a collection of +offsets. These objects are described by a distance and a proper +motion vector. The AP Database provides the association between the +specific detections and an average object which includes finite +parallax and proper motion. + +Fourth, many detections, especially in their initial states, will not +be associated with a specific astronomical object of any of the above +classes and are treated as orphans. Most of these will be spurious +(not representing real objects), some will be from solar system +objects for which orbits are not yet determined, some will be from +faint stars near the detection limits, and some will be from +short-term transients which have only been detected once. The AP +Database maintains these detections until they have been associated +with one of the objects above. The AP Database provides mechanisms by +which individual detections may be migrated back and forth between the +orphan state and association with an astronomical object. + +For every object, and all orphaned detections, the AP Database also +provides the capability to determine the images containing the +location of the object but for which no detection was made. The +minimum set of information which must be carried for these +non-detections is the image and the associated object or orphan. + +The AP Database also stores the relationships between various +photometric systems and the evolution of that relationship. It +provides mechanisms to convert between the measured instrumental +magnitude of a detection with a specific filter, detector, and +telescope, and at a particular time and the implied magnitude in the +average Pan-STARRS photometry system, given a determined set of +calibrations. It also provides the capability to convert magnitudes +in one system to the magnitudes in another system; an example of such +a conversion is between the average Pan-STARRS filter systems and the +various reference systems appropriate for those filters. + +\begin{figure} +\begin{center} +\resizebox{4.5in}{!}{\includegraphics{pics/APDB}} +\caption{AP DB components} +\label{fig:APDBComponents} +\end{center} +\end{figure} + +The AP Database provides interfaces to extract lists of objects and +detections based on various query parameters. It provides the +capability to extract all detections associated with a specific +object, all non-detections of that object, all non-detections of an +orphan, and summary statistics from these collections. It will also +return all objects or detections within specified spatial regions +including regions bounded by great circles (RA,DEC; GLAT,GLON; +ELAT,ELON) and regions described by a location and a search radius. +It will also return the image parameters associated with a specific +detection including image coordinates of the detection, exposure time, +time and date of the detection, etc. + +As shown in Figure~\ref{fig:APDBComponents}, the IPP AP Database +consists of the following components: + +\begin{itemize} +\item AP Database database tables +\item AP Database database engine +\item AP Database servers +\item AP Database client APIs +\end{itemize} + +\subsubsection{AP Database Tables} + +Table~\ref{tab:APDBTables} lists the tables used by the AP Database. The +contents of these tables are outlined in +Appendix~\ref{sec:APDBTableContents}. Below, the use of these tables by +the AP Database software is discussed below. Several of the tables +are not just simple tables in the database but are instead table +groups divided into many subtables, each of which represents a portion +of the sky (a {\tt region}). These subtables may also be distributed +across different computers to distribute the processing load. + +\paragraph{Images Table Group} + +The {\tt Images} table group lists all of the images which provided +the data in the AP Database. These tables are subdivided by region on +the sky. In general, the images listed in this table correspond to +the Chips. This group of tables includes sufficient astrometric +parameters to represent the coordinates of the detections to a +sufficient accuracy. Parallel to the Images table is the Mosaic +table. This table is very similar to the Images table, but defines +the Mosaic which corresponds to a group of Images. The parameters +include the astrometric information needed to define the camera +distortion. + +\paragraph{Image Overlaps Table Group} + +The specific subtable of {\tt Images} which contains a given image is +the one which contains the center pixel of that image. An additional +table group, {\tt Image Overlaps} (with the same subtable organization +as the {\tt Images} subtables), lists images which overlap that +specific subtable. Thus, given a particular coordinate, in order to +find that images which overlap that coordinate, it is necessary to +search the images in the {\tt Images} subtable which includes that +coordinate, and all images in the {\tt ImageOverlaps} subtable for +that coordinate. + +\begin{table}[hb] +\begin{center} +\caption{AP Database Tables\label{tab:APDBTables}} +\begin{tabular}{ll} +\hline +\hline +{\bf Table Name} & {\bf Description} \\ +\hline +Images & The images that have objects in the DB. \\ +Image Overlaps & Image regions which are touched by specific images. \\ +Objects & The objects --- average properties of multiple detections of the same object. \\ +Average Magnitudes & Average photometry in multiple filters \\ +Solar System Objects & Identification of solar system objects \\ +Matched Detections & Detections of sources in an image identified with an Object. \\ +Orphaned Detections & Detections of sources in an image not identified with an Object. \\ +Non-detections & Non-detections of objects in an image. \\ +Regions & spatial distribution of tables \\ +Filters & Filters understood by the system. \\ +Photcodes & Transformations between different photometric systems \\ +Zero Points & History of Zero-point \& Airmass terms \\ +Distortion Models & History of Optical Distortion terms \\ +Database Hosts & computers used to store the tables \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\paragraph{Objects Table Group} + +The {\tt Objects} table group (also divided by region) stores the +average parameters for each astronomical object. Certain details of +this table have not yet been specified. In particular, objects with +significant parallax and/or proper motion may potentially be stored in +a distinct table. Solar system object identifications, to the extent +average properties are maintained in the AP Database, will certainly +be stored in a separate table. + +\paragraph{Average Magnitudes Table Group} + +A related table, also divided into the same regions, is the {\tt +Average Magnitudes} table. In this table, there are multiple rows per +object, one for each of the primary filters of interest for which +photometric averaging is performed. This organization makes the +number of primary (averaged) filters a configurable value. + +\paragraph{Matched Detections Table Group} + +The {\tt Matched Detections} table stores all of the measurements of +astronomical objects on specific images. This table includes all +detections associated with the average {\tt Objects}. As discussed +below, bright objects (above a configuration-specified signal-to-noise +level) are defined object even if only one detection has been found at +that position. Faint orphaned objects are not added to this list or +the list of objects. The different types of detections (P2, +P4$\Delta$, P4$\Sigma$) are distinguished by their photometry codes. +(This is only valid if the AP Database does not store different +quantities for these types of detections.) + +\paragraph{Orphaned Detections Table Group} + +The {\tt Orphaned Detections} table stores the detections which have +not been correlated with an existing object. This table is only +populated for objects below a configuration-specified signal-to-noise +limit (e.g., 5$\sigma$). Bright orphaned detections are assigned an +object and added to the {\tt Matched Detections} table. + +\paragraph{Non-detections Table Group} + +The {\tt Non-detections} table stores information about detection +failures for each object. If an image is added to the database which +overlaps an object but the object is not detected, an entry is made in +this table. In practice, this table may store only the most recent +non-detection and the total number, or a similar reduced set of +non-detection statistics. + +\paragraph{Regions Table} + +The {\tt Regions} table is used to subdivide the tables of images, +objects, and detections, etc, as discussed above. The AP Database +divides the sky into a hierarchy of regions (portions of the sky) each +of which is in turn subdivided into smaller portions. Since nearly +all interactions with the AP Database performed by the IPP are limited +in spatial coverage, subdividing the tables allows a specific +interaction to search only a small subset of the data. The table of +images is the smallest of the three; the table of detections is likely +to be the largest. As a result, the {\tt Images} table group will be +subdivided at a shallow hierarchical level, while the {\tt Objects} +and {\tt Detections} are subdivided on deeper (more finely sampled) +levels. The {\tt Regions} table defines the boundaries of the sky +regions and specifies if the region corresponds to an {\tt Images} +table, an {\tt Objects} table, and/or a {\tt Detections} table. It +also specifies which regions in the next level of the hierarchy are +contained by the region, and which parent region it belongs to. In +addition to improving the spatial access to the image, object, and +detection data, the {\tt Regions} table allows for multiple computers +to serve the database tables. The region file specifies the machine +which stores the specific table. Figure~\ref{fig:APDBRegions} +illustrates schematically the subdivision of the sky and the +association between different levels of the hierarchy with different +subtables. + +\begin{figure} +\begin{center} +\resizebox{6in}{!}{\includegraphics{pics/APDBRegions}} +\caption{AP DB Regions and Image / Object tables} +\label{fig:APDBRegions} +\end{center} +\end{figure} + +\paragraph{Other Reference Tables} + +The {\tt Filters} table identifies all of the physical filters +(specific pieces of glass) known to the system. A related table, {\tt +Photcodes}, defines relationships between photometry systems. A +photometry system may consist of a detector, telescope, and specific +filter, or it may be a derived photometry system. The {\tt Database +Machines} table identifies all of the computers available to the AP +Database. + +\subsubsection{AP Database servers} + +The AP Database functions on a group of computers, with portions of +the tables stored on separate machines, as described above. The +association between a machine and the corresponding table or part of +the sky is defined by the Region table. Each machine has a +corresponding AP Database server which runs on that machine to +interact with the tables available on that machine. Two possible +interaction models are considered. + +{\bf Option A:} A client chooses one of the machines and sends its +query or data to that machine. The server then uses the region table +to determine which machines contain the relevant portion of the sky. +Data to be added to the database is divided into corresponding region +chunks and sent to the appropriate servers. Queries are redirected to +the appropriate server(s). The original server may collect the +results and return them to the original client. + +{\bf Option B:} The client downloads the region table and performs the +division of the data into appropriate subsets. The subsets are then +sent to the corresponding servers by the client. + +The differences between these models is small. The first option may +make the code more testable, placing all of the logic in the servers +and making each server symmetric. The smaller tables (ie, Region, +Filters, etc) could either be downloaded from a single server or +replicated to all AP DB servers. For these reasons, Option A will be +used for the PS-1 IPP.. + +\subsubsection{AP Database engine} + +The backend database engine for the AP Database stores the tables and +provides them to the servers on demand. The AP Database will use a +\code{mysql} database engine for this function. + +\subsubsection{AP DB Client operations} + +The AP Database client interactions consist of a collection of basic +queries of the database, along with more complex operations to perform +particular tasks. The complex operations are listed below. + +\paragraph{Insert Image \& Detection Set (addstar)} + +One of the most basic operations needed by the AP Database is to +insert a collection of detections derived from a specific image, and +add the definition of that image to the database. This operation is +critical in terms of the processing throughput. After the detections +have been assigned to the appropriate regions, they are matched +against all objects in the {\tt Objects} table. Matches are performed +only on the basis of positional coincidence, using a matching radius +which may depend on the image astrometry errors, or may be a fixed +distance. Any matched detections are added to the {\tt Matched +Detections} table. Any unmatched detections brighter than the Faint +Detection cut-off are specified as a new {\tt Object} and also added +to the {\tt Matched Detections} table. Any faint unmatched detections +are added to the {\tt Orphaned Detections} table. This division is +important because it allows the automatic association of new +detections with existing bright objects while limiting the I/O volume +required to make the detections. In general, there will be many fewer +{\tt Objects} than {\tt Detections}, and there will be fewer bright +orphans than faint orphans. + +\paragraph{Insert Reference Objects (addrefs)} + +This operation is very similar to the previous one. A collection of +reference objects are added to the database as a collection of +detections. The reference photometry should in general be given its +own photometry code. The reference data is different from the image +detection set because the associated image information is not +included. Thus, no corresponding images are added to the database. + +\paragraph{Determine Relative Photometry in region (relphot)} + +This operation uses the overlaps of images and multiple observations +of the same objects to determine the relative photometry zero-points +for a collection of images. This is a task that wil be run much more +infrequently than the object insertion tasks. + +\paragraph{Determine Consistent Photometry Zero Points (uniphot)} + +This operation uses the time history of relative photometry zero +points for images and the spatial overlap information to determine a +best set of image zero points which have a specific time scale for the +atmospheric stability. + +\paragraph{Determine Distortion and Static Astrometry Model (mosastro)} + +This operation uses the reference and image detections to determine an +optical distortion model for the camera and static astrometry model +components. The astrometry model includes: (1) field distortion +introduced by the telescope optics, which is a smoothly-varying +function of the field position relative to the center of the telescope +boresite coordinates. (2) focal plane geometry, which includes the +chip positions and rotations in the focal relative to the boresite, +along with chip-dependent plate-scale modifications needed to +represent tilts or warps of the individual detectors relative to the +ideal flat focal plane. . + +\begin{table} +\begin{center} +\caption{AP Detection Classes \& Object Parameters\label{tab:APdetections}} +\begin{tabular}{lrrrr} +\hline +\hline +Object Parameter & P2 & P4S & P4D & SS \\ +\hline +PSF x,y, covar, $\alpha,\delta$ & + & + & + & + \\ +PSF mag, $\sigma_{\rm mag}$ & + & + & + & + \\ +star/gal sep & + & + & + & + \\ +$\sigma_x$, $\sigma_y$, $\theta$ & + & + & + & + \\ +local sky data & + & + & + & + \\ +Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ +S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ +W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ +exp. spaced aps., Poisson noise, variance & - & - & - & + \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\subsubsection{Throughput} + +The AP Database design partly driven by the need to make the +detection-object associations quickly and to processes the incoming +detections at a sufficiently high rate to meet the throughput +requirements. For each upload of the object detections from a +complete FPA, the AP Database must match roughly $1.4 \times 10^{6}$ +detections from an FPA with roughly $6.4 \times 10^{6}$ objects, +including orphaned bright detections. This corresponds to roughly 640 +MB, if each object uses 100 bytes for its descriptive informations +(more than is currently specified in the Object table). With a +throughput of 100 MB/s for reads from a RAID, the AP Database can +perform the data read in a fraction of a second if the data is +distributed across 10 computers. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Controller} +\label{sec:Controller} + +\subsubsection{Corresponding Requirements} + +The Controller must meet the requirements specified in Section 3.4.4 +of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The design must meet +requirements 3.4.4.1 - 3.4.4.7. In particular, the Controller / Node +Agent architecture is chosen to control the I/O flow between the +Controller and the individual processes so that blocking on the I/O +from many remote processes does not saturate the Controller +processing. + +\subsubsection{Overview} + +\begin{figure} +\begin{center} +\resizebox{4.5in}{!}{\includegraphics{pics/Controller}} +\caption{Schematic illustration of the Controller components} +\label{fig:Controller} +\end{center} +\end{figure} + +The IPP uses a group of computers to store and process images and to +manipulate collections of detections. These computers perform any of +a large number of analysis stages or other processing tasks without +significant interprocess communication. It is necessary to have a +mechanism which initiates computing tasks on the different computers, +which monitors the tasks as they are executed, which handles the +output and the errors from these tasks, and which reacts to the +failure of any of the computing nodes. The system responsible for the +tasks in the IPP is the IPP Controller. + +The IPP Controller interacts with the collection of computers under +its management and with other subsystems in the IPP. The IPP +Controller receives a variety of inputs from other subsystems, +described below, and initiates actions such as adding a new process to +the queue of pending tasks. The IPP Controller also provides +information to other subsystems on demand about its processing history +and current state. Each physical computer may have multiple +processors; since the IPP Controller is managing processing tasks, it +treats each processor independently. It is up to the system +configuration if each computer needs to reserve one of its CPUs to +manage background tasks or if the IPP Controller should attempt to +send one task per CPU and let the operating system handle the I/O +load. The relationship between the different components of the +Controller is illustrated in Figure~\ref{fig:Controller} and discussed +below. + +\subsubsection{Nodes} + +The Controller maintains a table of available processing computers +(`Nodes') and tracks the status of these Nodes. Nodes managed by the +IPP Controller are allowed to be in one of several states, and the IPP +Controller must interact with it in an appropriate way for each of +those states. A Node may be {\tt alive}, {\tt dead} or {\tt off}. +If the Node is {\tt alive}, it responds to commands from the IPP +Controller and may be used for tasks subject to other constraints. If +it is {\tt dead}, the Node is not responsive and must not be used +for executing tasks. The IPP Controller must identify Nodes which +have died (not responding) and occasionally test them to see if they +are {\tt alive} again. Nodes which are {\tt off} are not +available for tasks and must not be tested. Nodes may be set to +the {\tt off} or {\tt dead} states by external subsystems; it is the +responsibility of the IPP Controller to return a Node to the {\tt +alive} state if possible. + +The IPP Controller must honor requests (normally from the users) to +change the mode of any computing node on demand between {\tt off} and +{\tt dead}. This would normally be done after a Node has been +rebooted and is released to the IPP Controller for its use. It must +also be able to change the list of allowed tasks as requested by +external commands. + +Two example scenarios illustrate the transition between these states, +and the basic concept of operations for the IPP Controller. First, +imagine a computer crashes. At this point the IPP Controller should +detect that the Node is no longer responsive and mark it as {\tt +dead}. It should occasionally try to re-establish communication with +the Node, potentially with longer and longer delays between attempts. +A human could be notified if the Node seems to remain {\tt dead} for a +very long time. In another scenario, a person needs to work on a +Node. They notify the IPP Controller that the machine is {\tt off}, +perhaps with a prior notification that the machine should be prepared +to go off. When work on the machine is complete, it should be placed +in the {\tt dead} state. Only when the person is done working and +testing the machine, and tells the IPP Controller that the machine is +now {\tt dead} can the IPP Controller attempt to re-start +communications and re-new processing operations on that Node. + +\subsubsection{Node Agents} + +When the Controller starts, it attempts to launch a Node Agent on each +of the available processing Nodes. Nodes which are not responsive are +marked as {\tt dead} so they may be re-tried. A Node Agent runs on +each of the individual nodes to execute the tasks as directed by the +Controller. The Node Agents communicate with the Controller via a +socket connection. + +A Node Agent (which is only running on a Node in the {\tt alive} +state) may be in one of four modes: {\tt idle}, {\tt busy}, {\tt +done}, {\tt crash}. A Node Agent which is {\tt busy} currently has a +task assigned to it which is executing. The IPP Controller may only +assign one task to a Node at a time. A Node Agent which is in the +{\tt idle} state may have a task assigned to it. When the Node Agent +detects that a tasks has finished, it changes to either the {\tt done} +or {\tt crash} states depending on the outcome of the process +execution. The IPP Controller must also respect a list of task +restrictions which may require specific tasks to run on specific CPUs +or exclude specific tasks from specific CPUs. + +A task being executed by the Node is run in the UNIX user space as a +forked process. The Node Agent must monitor the standard error and +standard output of the executing task and save them in separate +buffers. If the process exits or dies, the Node Agent must detect +this result and change state appropriately. The Node Agent must +respond to various commands from the Controller, as follows: + +\paragraph{Report status} + +The Node Agent returns its state ({\tt idle}, {\tt busy}, {\tt done}, +{\tt crash}) and the exit status of the current processing task, if +available. The reported exit state, if the process has completed +without crashing, is the UNIX exit state reported by the task: 0--256 +with 0 indicating a successful completion. + +\paragraph{Report stdout} + +Send and flush the current stdout buffer. The Node Agent will return +the complete contents of the stdout buffer via a buffered write and +flush the buffer when it is finished. The Node Agent will not accept +more data on the stdout buffer from the current processing task until +the send is complete and the buffer is flushed. The daemon must +accept all of the buffer output. + +\paragraph{Report stderr} + +Identical to `report stdout', but for stderr. + +\paragraph{Kill task } + +The Node Agent should send a kill signal (\code{KILL} or \code{TERM}) +to the current processing task. When the processing task has exited, +the Node Agent should set its state to {\tt crash}. + +\paragraph{Clear task} + +The Node Agent should set its state {\tt idle}. If a processing stage +is currently running, it should be killed (\code{KILL} or \code{TERM}) +before the task is cleared. + +\paragraph{Start processing stage} + +The Node Agent forks a specified command. The command should be a +standard UNIX command without command line redirection or +backgrounding. The task is run with the same user ID as the Node +Agent, which is also the same user ID as the Controller. + +\subsubsection{Tasks} + +The IPP Controller accepts tasks from other IPP subsystems. The task +requests include the specific command to be executed and are in the +form of a UNIX command which could be performed on any of the +computing nodes. Any input or output data in the commands must be a +valid resource regardless of the node on which the task is executed. +Input and output data resources must be unique where necessary to +avoid conflicts. It is the responsibility of the task to wait for +network lags (ie, NFS delays). The IPP Controller gives each task a +unique identifier, which is returned to the requesting entity. The +requestor may then use that ID to obtain status information on that +task or to send control signals to the specific task. + +Task requests may specify a desired node for the task execution. The +IPP Controller attempts to honor the request if the node is {\tt +alive}, but will execute it on another node if the requested one is +{\tt dead} or {\tt off}. Even if a node is {\tt alive}, the IPP +Controller will choose another node if the specified task is not +allowed on the requested node. In all other cases, the IPP Controller +waits until the currently executing processes, and processes with +higher priority, are completed before executing the specified task on +the requested node. + +Task requests may specify an urgency level. The IPP Controller +determines the priority of the task on the basis of both the urgency +and the age of the request. An executing task must be completed on a +CPU before any new task is started on that CPU, regardless of +priority. The urgency levels range from 0 to 2. Tasks with an +urgency of 1 are scheduled whenever they reach the top of the stack. +Tasks with an urgency of 2 are sent immediately to the top of the +stack. Tasks assigned a priority of 0 are maintained in the queue and +never executed. + +It may be useful for the Controller to distinguish between tasks +dominated by I/O and tasks dominated by data processing. It is +possible that one of each of these types of tasks may be sent to the +same node without significantly impacting the system performance. +Alternatively, it may be necessary to limit a single machine with 2 +CPUs to only one of each of these types of tasks (i.e., one processor +will be working on I/O while the other is working on processing). +Such details will be studied by the IfA IPP Team. + +The IPP Controller monitors the output streams from the executing +tasks and the exit status of the tasks. Each task is associated with +a log file, to which all output is written. The status, including the +exit status, of each task is maintained by the IPP Controller so that +other subsystems may determine if specific tasks have started or +completed. + +\subsubsection{Controller Interfaces} + +The IPP Controller must accept commands from other IPP subsystems. +These commands include those which govern the processing of specified +tasks, those which govern the behavior of specific computing nodes, +and those which request information from the IPP Controller. The IPP +Controller must be able to halt the execution of a specified task, +delete an unexecuted task from the task list, change the priority of +tasks, and change the requested nodes for tasks. The IPP Controller +must also be able to stop the current execution of a task and push it +to the end of the queue and also change its priority. + +The IPP Controller must respond to informational requests regarding the +collection of machines and their states as well as the collection of +tasks and their states. The IPP Controller must monitor the execution +times of the different tasks and provide summary statistics. Finally, +the IPP Controller must respond to three top-level commands: {\tt finish}, +{\tt stop} and {\tt abort}. When {\tt finish} is requested, no more +new tasks are accepted on the stack of task, and when all tasks in the +stack have completed, the IPP Controller must exit. When {\tt stop} is +requested, the currently executing tasks must be completed at which +point the IPP Controller must exit, but tasks remaining in the stack which +have not been started are flushed. When {\tt abort} is issued, the +IPP Controller immediately kills all executing tasks and exits. + +The IPP Controller and the IPP Image Server have related needs for +information from the combined storage-and-processing nodes regarding +which nodes are available. It is not yet clear if this information is +best stored in a single location (either IPP Controller or IPP Image +Server), which provides the information to other systems on demand, or +if both systems should maintain the information. Also, it may be +necessary to distinguish nodes which are available for processing from +those that are available to serve data as part of the IPP Image +Server. + +The Controller maintains three tables of processing jobs: pending +stages, active stages, and completed stages. The pending stages are +those which have not yet been performed. The active stages are those +currently being performed on one of the remote nodes. The completed +stages are those which have finished, either successfully or with an +error state. The Controller daemon monitors the collection of remote +clients and sends them new pending stages when they become free. + +The IPP Controller provides a mechanism for users (either other +programs or humans) to interact with it. The user interface provides +commands to check the current processing job queues, the tables of +successful and failed jobs, to stop or delete jobs, etc. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Scheduler} + +\subsubsection{Corresponding Requirements} + +The Scheduler must meet the requirements specified in Section 3.4.5 of +the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The design must meet +requirements 3.4.5.1 - 3.4.5.7. In particular, the Task / Test +division is chosen to prevent the Scheduler from blocking while an +analysis process is performed. Scheduling requirements will be met by +defining appropriate Test periods for the different Tasks. + +\subsubsection{Overview} + +The IPP is responsible for a variety of analysis jobs: processing of +the science images through several stages; routine assessment of the +detrend (instrumental calibration) images used in processing the +science images; construction of replacement detrend images when +needed; generation of astrometric and photometric reference catalogs +based on the collected dataset; and the performance of test analysis +programs. At any point, decisions need to be made about which of +these tasks should be performed, based on an analysis of the contents +of the metadata database, the requirements of the people monitoring +the IPP, and the near-term observing plans. The IPP Scheduler is the +mechanism that assesses these various inputs to guide the decisions +and initiate the actions. + +The IPP Scheduler acts as an interface between several components of +the IPP and also between the IPP and external agents such as OTIS and +the users who must monitor the behavior of the IPP. The IPP Scheduler +may be viewed as the central brain of the IPP. +Figure~\ref{fig:Scheduler} illustrates the design of the IPP +Scheduler. + +\subsubsection{Scheduler Tasks and Tests} + +The IPP Scheduler performs two types of actions. 'Tasks' are +long-running programs which are executed by the Controller. These are +not only background tasks, but are distributed computing tasks. +Examples of these include the science analysis tasks (eg, Phase 1, 2, +3, 4), the Calibration construction tasks, and data copy tasks (such +as copying images and metadata from the summit system). 'Tests' are +short-running programs which are used to decide which tasks should be +run. Tests should be designed to return immediately ($< 100 ms$) and +are not run in the background; the Scheduler will block until the test +is complete. The IPP Scheduler daemon, which runs continuously, +performs tests (eg, queries of the IPP Metadata Database, queries of +OTIS, checks of the IPP hardware status, etc). Based on these tests, +the daemon defines appropriate tasks and sends them to the Controller. +When tasks are completed, their results may be used by the Scheduler +to update the external systems (update the Metadata Database), or the +tasks themselves may send their results directly to the Metadata +Database or other subsystems. Based on the successful completion (or +not!) of the tasks, and the new state of entries in the Metadata +Database, the Scheduler can define new tasks. + +\begin{figure} +\begin{center} +\resizebox{6in}{!}{\includegraphics{pics/Scheduler}} +\caption{ \label{fig:Scheduler} IPP Scheduler} +\end{center} +\end{figure} + +The IPP Scheduler sends tasks to the IPP Controller for execution. +While the IPP Scheduler chooses the tasks to be performed, it is the +IPP Controller's responsibility to manage the specific tasks executing +on a given processing node. This division of responsibilities allows +the different functionalities of the IPP Scheduler and the IPP +Controller to be isolated and encapsulated. With this separation, the +IPP Controller does not information about the details of the tasks it +executes, while the IPP Scheduler does not need to monitor the +computer hardware. + +Communication between the IPP Scheduler and the IPP Controller is +bi-directional; the IPP Scheduler sends tasks to the IPP Controller, +while the IPP Controller informs the IPP Scheduler of the outcome of +those tasks. For the PS-1 IPP, the IPP Scheduler and the IPP +Controller are distinct, interacting software components. The +interface mechanisms are described in Section~\ref{sec:interfaces}. + +\subsubsection{Task Rules} + +The IPP Scheduler takes as input a collection of rules which define +the dependency of tasks on certain tests. The IPP Scheduler must +choose between several types of analysis tasks based on those rules +and on results of the tests. The timescale on which different tasks +(and their related tests) are executed may vary from 10s of seconds to +hours, days, or even as long as a week. The list of tasks which the +IPP Scheduler must decide between, and the relevant timescale, follow: +\begin{itemize} +\item moving data from the Summit pixel server ($\sim 30$ second timescales) +\item running the science analysis stages ($\sim 30$ second timescales) +\item testing the validity of the current detrend images ($\sim$ + nightly) +\item constructing new detrend images ($\sim$ weekly) +\end{itemize} +The scheduler may be viewed as a complex state machine. The goal is +to design the scheduler so that rules may be specified independently +from the engine which parses the rules to determine which specific jobs +to send to the controller. + +\subsubsection{User Interface} + +The IPP Scheduler shall possess a user interface which allows a human +operator, or other processes, to monitor the current state of the +Scheduler. Users have the option to specify that a particular task or +set of tasks is of higher or lower urgency (as defined in +Section~\ref{sec:Controller}) than the norm, or to schedule a +particular tasks on a different timescale from the basic rule. + +The IPP Scheduler defines the operating state of the IPP and shares +the same set of states: +\begin{itemize} +\item active state +\item interactive state +\item paused state +\end{itemize} +When the IPP Scheduler is in the {\em active state}, it performs the +most appropriate of all possible tasks at a particular time. When the +IPP Scheduler is in the {\em interactive state}, it performs only a +specific requested action regardless of the outcome of the decision +trees. In addition, in the interactive state, the IPP Scheduler must +only perform the requested actions and not attempt to perform the +other normally-required actions. The only exception to this exclusion +is that, in the interactive state, data is still copied from the +summit system. An additional IPP state is the {\em paused state}, +intended for tests or maintenance, in which case the IPP Scheduler +does not perform even the data copy tasks. Every task is performed on +demand by the user. A user command sets the IPP Scheduler in one of +these three states, {\em active}, {\em interactive}, and {\em paused}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{System Design : Science Analysis Tasks and Stages} +\label{sec:AnalysisStages} + +This section describes the design of the science analysis stages which +perform the fundamental image analysis steps of the IPP. The IPP +science image processing stages perform analyses on the night-sky +science images to extract the science data from these images. These +consist of: +\begin{itemize} +\item Phase 1, the image processing preparation stage, +\item Phase 2, the image reduction stage +\item Phase 3, the exposure analysis stage +\item Phase 4, the image combination stage. +\end{itemize} +These analysis tasks must process the images in a timely manner so +that the incoming data stream will not overload the IPP Image Server. +The decision to execute a specific pipeline for a specific dataset is +made by the Scheduler, which sends the information to the Controller. +The Controller executes the pipeline for the data on an appropriate +machine and monitors the success or failure of the processing stage. + +The analysis stages are written as UNIX commands, which may be +executed by the IPP Controller, or may be executed individually by +hand. This option makes testing of the complete analysis system much +easier because the individual analysis stages may be tested +independently of each other and the IPP infrastructure. + +As part of this design model, the analysis stages have several methods +for accepting and returning the input and output data and for defining +optional choices in the analysis. All of the analysis stages load an +analysis recipe, which defines the details of that analysis. The +recipe includes the location of the data sources (from the metadata, +from the image headers, from other external files, or supplied +directly), which steps to employ, and how to assign optional +parameters. For example, in the discussion of the Phase 2 analysis +below, the recipe file may specify {\em if} a bias subtraction should +be applied, {\em where} to find the overscan region and {\em which} +bias image, {\em if any}, to apply. + +The recipe is loaded as part of the runtime configuration information +loaded when the analysis script starts. Four levels of runtime +configuration information are defined. The {\tt site} configuration +defines values specific to the particular installation of the +software. For example, the name of the machine which hosts the +Metadata Database or a default path for data files could be part of +the {\tt site} configuration. Multiple installations or versions of +the IPP software would need to have separate {\tt site} configuration +entries. For example, a version of the IPP installed at the IfA would +use a different computer for the Image Server from the live IPP +installation running on the Pan-STARRS cluster. The {\tt base} +configuration defines general data sources which may be needed by any +portion of the IPP. The list of known telescopes or filters might be +an example. The {\tt camera} configuration consists of information +which defines the parameters relevant to the cameras known by the IPP. +For example, the default layout of the detectors or the names of +specific header keyword values would be defined for each camera in a +camera-specific configuration collection. Finally, each analysis +script loads its own recipe. The location of this configuration +information may be a collection of configuration files available on +disk or some subset of the information may be stored in the Metadata +Database. The source of these configuration entries can be overridden +when the script is executed, and individual configuration values may +also be specified on the command line. Examples of the recipe and +other runtime configuration options are given in +Appendix~\ref{sec:RuntimeConfig}. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Phase 1: image processing preparation} + +The Phase 1 analysis stage is performed on each science exposure (each +complete FPA image) to calculate basic astrometric data needed by the +later stages. Phase 1 uses the static (pre-determined) telescope +distortion model and a table of nominal OTA positions and rotations, +combined with the guide star pixel and celestial coordinates, to +determine the correct telescope bore-sight, field rotation and +magnification. The guide star coordinates are loaded from the +Metadata database. These calculations are performed by comparing the +observed guide star detector coordinates with the known astrometric +positions of these same stars as reported by an external astrometric +reference. The accuracy of the resulting astrometric solution is +expected to be $\sim 1$ arcsec across the field, sufficient in later +stages to match the vast majority of astrometric reference stars with +their detections with minimal effort. + +In some circumstances, science images may have no guide stars. This +may occur in the Pan-STARRS system if the detectors are not run in OTA +mode, for example for short snapshot images. This may also be the +case if the IPP is being run on non-Pan-STARRS data. In such a +circumstance, the Phase 1 stage uses the provided boresight +coordinates, exposure time, and camera zero-point to predict the pixel +coordinates of known bright stars expected to be found on the +detectors. It then extracts a large box ($\sim$ 30 $\times$ +30\arcsec) around these locations and performs extremely basic object +detection to determine the detector coordinates of those bright stars +which are not saturated but which are significantly above the +background level. By targeting known locations in the image files, +only a small amount of data will have to be read. + +If the image has invalid coordinates or no detectable bright stars, +Phase 1 fails and reports a descriptive error. + +Given the above astrometric solution, the Phase 1 analysis stage +constructs a table of the overlaps between the science image to be +processed and the static sky images that must be constructed. This +table will be used to guide the processing of the static sky in Phase +4. The overlaps should be generously calculated so that small errors +in astrometry at Phase 1 will not cause any valid static sky / science +image pairs to be missed because of the astrometric error at this +phase. It is acceptable for a small number of invalid overlaps to be +identified as these will be excluded in Phase 4. Static Sky cells +which do not have sufficient science image overlap ($< 5\%$) need not +be processed because the few new measured pixels do not add +significantly to the Static Sky. + +\subsubsection{Examples} + +Examples of Phase 1 as called from the command line, with different +types of images: + +\begin{verbatim} +Phase1 -file filename.fits [FPA is single fits file] +Phase1 -list filename.list [FPA is collection of files] +Phase1 -imdb ID [FPA is single file in image server] +Phase1 -FPA ID [FPA identifier from metadata db] +\end{verbatim} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Phase 2 : image reduction} + +\subsubsection{Overview} + +Phase 2 processing within the Pan-STARRS image processing pipeline is +the detrend stage, where the images from the detector are processed to +remove instrumental signatures. This analysis is performed on +individual chips, which can be identified as the data entity which has +a single, continuous astrometric solution. + +Phase 2 consists of the following operations, some of which as noted +may be skipped by the recipe: +\begin{itemize} +\item Load science image +\item Identify appropriate detrend images +\item Load detrend images +\item Form OT kernel +\item Convolve detrend images with the OT kernel +\item Bias/dark/overscan subtraction +\item Mask bad pixels +\item Trim overscan +\item Non-linearity correction +\item Flat-field +\item Mask diffraction spikes and optical ghosts +\item Subtract sky +\item Find and photometer objects in the image +\item Identify CRs by morphology +\item Determine PSF model +\item Improve astrometry +\item Extract Bright object postage stamps +\end{itemize} +The steps are explained in detail below. + +\subsubsection{Load Images} + +The Phase 2 analysis must load the science image to be analyzed into +memory, as well as the corresponding metadata (either from the image +header and/or from the IPP Metadata Database). It must use the +metadata for the image, along with information from the processing +recipe, to determine the appropriate detrend images to be used for +this analysis. The Metadata Database stores the information necessary +to associate a specific science image with one of the registered +master detrend images for each type. These images are also loaded by +the Phase 2 analysis (note that the design of Phase 2 may perform the +actual loading of pixels in blocks or groups to minimize the memory +impact). + +\subsubsection{Form OT Kernel \& Convolve with Detrend Images} + +Science images which have been obtained with Orthogonal-Transfer +Guiding have had their pixel response smoothed by the image correction +motion. For these images, some of the detrend images need to be +convolved by the same OT kernel, so that they accurately represent the +effective pixel response. The detrend images which must be convolved +include: the flat-field image, the high-spatial-frequency fringe +images, and dark images, if they are used. The appropriate kernel for +each cell of an OTA must be determined from the guide star history, +extracted from the IPP Metadata Database or from the image header. If +the OT kernel is not available, but the image metadata notes that it +should be, the convolution is skipped, with a warning. + +The convolution method depends on the size and structure of the OT +kernel. If the kernel is small ($< 5x5$ pixels), direct convolution +may be employed. If the kernel is large, but may be decomposed using +Gaussians, then it may be convolved using a decomposition method. + +The module convolves each of the dark frame, flat-field, and the +fringe frame(s) by the OT convolution kernel. Specific flags in the +static bad pixel mask are also grown by the outline of the OT +convolution kernel (see Section~\ref{sec:masks}). + +\subsubsection{Bias Correction / Overscan Subtraction} + +The image bias must be subtracted. Since different detectors behave in +different ways, several options for modeling the bias are available. +The bias is measured from the image overscan region. The bias +subtraction method must be capable of subtracting a single constant +from the complete image, or to subtract a 1-D bias which varies as a +function along the overscan. The function used to represent the +overscan region may be a spline or a Chebychev polynomial derived from +the data values along the overscan. The values used to determine both +the single constant or the inputs to the spline and polynomial fits +are derived from groups of pixels on the basis of one of several +statistics, including the sample and robust mean, median, and modes. +In the case of a single constant, all of the overscan pixel values are +used in the calculation of this statistic. In the case of the 1-D +functional representation, the input values to the fit must represent +the coordinate along the overscan, with the statistic derived from the +pixels in the perpendicular direction at each location. +Sigma-clipping on the input data values must be an option. + +\subparagraph{Flag bad and saturated pixels} +\label{sec:masks} + +A static bad pixel mask is used to identify pixels which are known to +be bad in the camera. This mask is then processed with the science +image. Bad pixels which are charge traps are grown by the extent of +the OT convolution kernel. Bad pixels above a charge trap (i.e.\ bad +columns) must not be grown, since they were not affected by pixel +shifting, but only became bad at read-out. + +Pixels which are saturated in the A/D converter, or with a signal +level at which the response is excessively non-linear, must also be +masked, and this area must be grown by an additional pixel to mask +excess charge spillover. + +The bad pixel mask is carried with the science images. Different bits +are set to identify different reasons for masking the pixel. Flags +are required for at least each of the following pixel attributes: + +\begin{itemize} +\item The pixel is a charge trap; +\item The pixel is a bad column; +\item The pixel is saturated in the A/D converter; +\item The pixel is non-positive in the flat-field; +\item The pixel is part of a row that has excess noise; and +\item The pixel is determined to be a cosmic ray, based on its +morphology. +\end{itemize} + +\subsubsection{Trim} + +The image is trimmed to remove the non-imaging pixels, such as the +overscan and any pre-scan pixels, along with those pixels near the +edges that have been compromised due to OT operation. The definition +of the imaging area of the detector is determined from the camera +configuration data or from the metadata associated with the image, +with the choice a user-configurable option. + +The input science and mask frames are additionally trimmed by the +extent of the OT convolution kernel in each direction ($+x$, $-x$, +$+y$, $-y$). Within the PSLib image handling functions, the trim +function is a virtual operation which simply marks the boundaries of +the trimmed image but does not remove the corresponding memory. This +allows the later corrections to work with untrimmed correction images +and still apply the correct pixels. At the end of Phase 2, the only +the trimmed portions of the output images are written out to disk. + +\subsubsection{Non-Linearity Correction} + +If required, the science image (after bias correction) must be +corrected for the effects of non-linearity through a provided +polynomial fit to the pixel data values or a numeric lookup table as a +function of pixel flux. The choice to apply the correction must be +set by the analysis recipe. + +\subsubsection{Flat field Correction} + +The science image (after bias correction and non-linearity correction) +must be corrected for sensitivity variations as a function of +position, dividing by a flat-field image. The mask is also modified +for zero-valued pixels in the flat-field image. + +\subsubsection{Sky \& Fringe subtraction} + +After the science image has been flat-fielded, the flux contribution +of the sky (from both continuum emission and the line emission that +causes fringing) must be subtracted from the image. The subtraction +needs to remove background (technically, foreground) variations which +are not celestial but generated in the atmosphere or by more localized +scattering. This background should include the contribution from the +zodiacal light. This background subtraction does not address the +artifacts generated by bright stars: bleeding columns, ghosts, or +other localized reflection effects. This process also produces a +super-binned image of the background map which may be used as a +debugging diagnostic. + +This analysis step temporarily masks objects on the image using the +astrometric solution from the boresight and fits for the sky +background, consisting of a polynomial to model the continuum, and the +fringe frame(s) to model the fringes from sky emission lines. If the +concentration of objects in the image is too high to reliably fit the +sky background, the background solution from an exposure close in time +and airmass to the current object image is used. The output is the +sky-subtracted object image. + +\subsubsection{Detect and Measure objects} + +After the image have been processed by the preceding steps, the Phase +2 analysis performs a basic object detection analysis. Objects on the +flat-fielded object image are found, and general parameters are +measured. Object detection is performed at several stages by the IPP, +with different object parameters measured in each case. +Table~\ref{tab:APdetections} gives a list of the different detection +stages and the object parameters measured for those stages. For the +Phase 2 analysis, the object parameters are: the object centroid and +the position covariance matrix, the instrumental PSF magnitude and +error, local background level and error, a measurement of the +star-galaxy separation, and a measurement of the object shape +($\sigma_x, \sigma_y, \theta$). The detection threshold must be +configurable, and be a function of the average background flux or the +image noise map. Minimal object classification must be performed to +distinguish objects which are consistent with a single PSF, objects +which are inconsistent, and objects which are saturated. The +resulting collection of detected objects are saved in the AP Database +along with the relevant image metadata (\ie filter, exposure time, +etc). In addition, this process constructs a model of the +point-spread-function (PSF) as a function of position in the image. +This PSF model is saved as part of the image metadata. + +\subsubsection{Identify CRs by morphology} + +Charged particles in the detector frequently cause features which do +not have the morphology of astronomical objects. In a well-sampled +image, these may be easily identified by the sharpness of the image. +In a near critically-sampled image, these `cosmic rays' may be +indistinguishable from stellar objects. This analysis step makes +morphological identification of cosmic rays if the imaging data is +sufficiently well sampled. The identified cosmic rays are masked with +a configurable growth factor so that additional pixels beyond the +detected pixels in the feature are also masked. + +\subsubsection{Perform Astrometry} + +The detected objects are matched with known astrometric reference +objects, using reference object coordinates which have been adjusted +for proper motion. The matches are then used to improve the +astrometric parameters for the individual OTAs. The OTA astrometric +parameters which are determined may include terms up to 3rd order in +position, though the terms which are actually fitted are +user-configurable. The Cell astrometric parameters are not allowed to +vary at this stage. The fit must be robust, rejecting outlier matches +(either stars with poorly determined proper motion or spurious +matches). The resulting astrometric solution is consistent across the +OTA field to within 1.0 arcsec. + +\subsubsection{Perform Photometry} + +If possible (if a local photometry reference exists), the Phase 2 +analysis determines a photometry zero point for each image. To do +this, it extracts the appropriate reference objects (from the AP +Database) and matches the stars between the two samples. The +zero-point is derived on the basis of a static atmospheric absorption +model (eg, linear airmass slope). + +\subsubsection{Bright object postage stamps} + +The IPP must have the capability of extracting regions surrounding a +specified subset of objects from the flattened images. These postage +stamp images must be saved for additional use by client science +pipelines. The identification of these objects must be on the basis +of a set of rules applied to the object's magnitude and position. The +postage stamps are not restricted in shape to simple rectangles, but +may represent more complex regions. They are written to the Image +Server. The outputs are these postage stamps and pixel masks, which +are sent to the IPP Pixel Server. + +%\begin{figure} +%\begin{center} +%\resizebox{6in}{!}{\includegraphics{pics/phase2}} +%\caption{ \label{fig:phase2} Phase 2 dataflow - this diagram is old: update} +%\end{center} +%\end{figure} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Phase 3 : exposure analysis} + +The Phase 3 system operates on the combined Phase 2 results from an +FPA to determine improved solutions for the image calibrations and to +provide the parameters needed by Phase 4. The Phase 3 output is saved +by the Metadata Database, and consists largely of improved values of +the calibrations already determined by Phase 2. The analysis +performed by this pipeline consists of: + +\begin{itemize} +\item improved astrometric solution based on comparison between + objects in the images and the astrometric reference. +\item improved background model based on the full telescope field, or + fields. +\item photometric solution based on comparison to photometric + standards +\item FPA-wide PSF analysis +\end{itemize} + +In the Phase 2 analysis, the astrometric solutions were determined +independently for each chip. These solutions are limited by the +assumption of a static distortion and by the accuracy of the +astrometric reference. In the phase 3 analysis, the astrometric +solutions of the complete FPA images are improved by combining the +astrometry for all chips. The astrometry model consists of a +projection of the celestial coordinates to the telescope boresite +center, followed by a rotation to the average rotation of the FPA and +adjustment for the central plate scale. The free parameters in this +stage are the boresite coordinates ($R_o, D_o$), the field rotation +($\theta_o$) and the plate scale ($\rho_o$), and are fitted in Phase +1. These tangent plane coordinates are then distorted by the optical +distortion model, consisting of $N^{\rm th}$ order polynomials in two +dimensions. Finally, the focal plane coordinates are mapped to the +individual chip coordinates, including the chip position and rotation, +as well as possible higher order terms representing warping of the +individual detectors. A first pass at the chip positions is +calculated in Phase 2, while the complete set of parameters is fitted +as a whole during Phase 3. The fitting process is determined in a +robust and stable way by fitting the gradient of the distortion as a +function of field position, removing the degeneracy of the distortion +coefficients on the chip position parameters. + +In the Phase 2 analysis, the background is determined based only on +the available sky in a single chip. However, the background +structures are normally correlated on the scale of the FPA, so an +improved background solution can be determined by combining the +information from many chips. A high-order polynomial model of the +background may be used and subtracted from all chips. + +The Phase 3 photometric improvement is made by comparing the +photometered objects from Phase 2 with the corresponding objects in a +local reference catalog. This analysis may only be performed if a +local reference is available. Note that improved relative photometry +calculations may be performed in the absence of a reference catalog on +the basis of image overlaps in the AP Database {\em after} the +detections have been added to the Database. Such a relative +photometry analysis is outside the scope of Phase 3 and will likely be +performed as an independent analysis process. Given the presence of a +local photometry reference, the zero point variations across the field +may be measured, and possibly modeled. If the zero-point variations +are excessive, then the image is marked as non-photometric by the +analysis. + +In the Phase 4 analysis, the $N$ FPA images are optimally combined to +create a single image of the sky with bad-pixel and cosmic-ray +rejection. This combination requires the calculation of a set of PSF +kernels to convert each of the input images to a single, common PSF. +These PSF kernels are determined from the per-chip PSFs measured in +Phase 2. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Phase 4 : image combination} + +\subsubsection{Overview} + +Phase 4 is the image combination stage, in which multiple images of +the same portion of the sky are merged and confronted with the static +sky image. Phase 4 operates on the smallest data unit of the static +sky, the sky cell, along with the associated pixels from a collection +of images which have been processed through phases 1--3. The size and +exact representation of a static sky cell are yet to be determined. +The working concept is that the static sky cells contain roughly the +same number of pixels as an OTA (4k x 4k) and represent a portion of a +local tangent plane projection. In order to meet the image +degradation requirements, the pixel scale of the static sky is planned +to be 0.2\arcsec, somewhat smaller than the 0.3\arcsec\ raw image +pixel scale. + +For each sky cell, the corresponding pixels are extracted from the +exposures being processed and mapped to the projection of the sky +cell. The pixels from the multiple input processed images are combined +into a single, cleaned image. Outlier pixels may be optionally +rejected at this stage (optionally, since moving objects will be +rejected in images obtained over a wide range of times). This image +is then confronted with the static sky cell data to produce a +difference image. Residual objects in the difference image above a +threshold are detected and excised from the original cleaned image. +The remaining pixels are added to the existing static sky image. +Object detection must be performed on the difference ($P4\Delta$) and +cleaned ($P4\Sigma$) images. + +\subsubsection{Image Warping and Matching} + +The analysis first maps the detector images to the sky cell using the +specified linear transformations, then combines the images with strong +rejection criteria and uses the combined sky cell image to identify +artifacts in the original detector images. It is desirable that the +artifacts are masked in the detector plane (i.e.\ before mapping to +the sky cell) so that they are not smeared out by the mapping; +alternatively, the CR mask needs to be grown by an additional pixel. +The mapped and masked detector images are then optimally combined +using the specified weighting. Both sets of combinations use the +photometric calibration for the images to set the relative scales of +the input images. The limiting magnitude for the combined sky cell +image should also be estimated. +% + +\subsubsection{Static Sky Subtraction} + +The corresponding static sky image is subtracted from the combined +image stack. In this step, it is necessary to match the image kernel +between the input image and the static sky image. This will be done +by solving for a best-fit image kernel which minimizes the difference +image using a technique equivalent to the Allard-Lupton method. One +modification for the IPP is to represent the kernel as a combination +of independent pixels rather than represent the components of the +image difference kernel as a combination of Gaussians. This method +also automatically determines a photometric match between the static +sky image and the input science image. + +\subsubsection{Object Detection and Measurement} + +Objects in the difference image are detected and a specific set of +object parameters are measured from these detections. +Table~\ref{tab:APdetections} gives a list of the different detection +stages and the object parameters measured for those stages. For the +Phase 4 difference image (P4$\Delta$), the measured object parameters +consist of: the object centroid and the position covariance matrix, +the instrumental PSF magnitude and error, local background level and +error, a measurement of the star-galaxy separation, and a measurement +of the object shape ($\sigma_x, \sigma_y, \theta$). The detection +threshold must be configurable, and be a function of the average +background flux or the image noise map. Detections must be performed +for both positive and negative (static sky only) sources. Minimal +object classification must be performed to distinguish objects which +are consistent with a single PSF, objects which are inconsistent, and +objects which are saturated. The resulting collection of detected +objects are saved along with the relevant image metadata (\ie filter, +exposure time, etc). + +Objects in the cleaned, summed image are detected and a specific set +of object parameters are measured from these detections. +Table~\ref{tab:APdetections} gives a list of the different detection +stages and the object parameters measured for those stages. For the +Phase 4 summed image (P4$\Sigma$), the measured object parameters +consist of: the object centroid and the position covariance matrix, +the instrumental PSF magnitude and error, local background level and +error, a measurement of the star-galaxy separation, a measurement of +the object shape ($\sigma_x, \sigma_y, \theta$), the Petrosian radius, +magnitude, axis ratio, and angle; and the S\'ersic radius, magnitude, axis +ratio, angle, and parameter $\nu$. The detection threshold must be +configurable, and be a function of the average background flux or the +image noise map. Minimal object classification must be performed to +distinguish objects which are consistent with a single PSF, objects +which are inconsistent, and objects which are saturated. The +resulting collection of detected objects are saved along with the +relevant image metadata (\ie filter, exposure time, etc). In this +measurement, objects at known positions will also be measured even if +they have not been detected. + +Objects which are detected in both of the Phase 4$\Sigma$ and Phase +4$\Delta$ images are saved to the AP Database, along with the relevant +image metadata (\ie filter, exposure time, etc). In the process of +adding these objects to the database, the transients which are +correlated with previous detections of an object (and those which are +not) will automatically be determined. A subset of these transient +objects are sent, along with their associated metadata, to the MOPS +and other preferred science client pipelines. + +\subsubsection{Static Sky Update} + +It is essential that the static sky image (which may have been +painstakingly accumulated over many months) not be corrupted by adding +in transient sources, or data that is of suspect quality (due, e.g., +to an error upstream in the processing). + +Object analysis of the static sky images is {\em not} a part of the +Phase 4 analysis. This processing is envisioned to take place +relatively infrequently (perhaps weekly or even monthly) and is +scheduled as a separate analysis task, probably run during the day at +a time when the computing infrastructure is not under significant load. + +%\begin{figure} +%\begin{center} +%\resizebox{6in}{!}{\includegraphics{pics/phase4}} +%\caption{ \label{fig:phase4} Phase 4 dataflow} +%\end{center} +%\end{figure} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{System Design : Calibration Image Processing} + +The Calibration Analysis Stages construct calibrations from the +relevant input data. Some of these combine multiple raw input images +together, after processing, to create a high-quality high-signal +master calibration image. Some of the calibrations are used to +correct other calibrations. Each of the calibration stages must also +provide the tools to test the quality of the input data against +existing master calibration data and to test the consistency of +multiple measurements of the calibration. + +The Calibration analysis stages may be performed on whatever +timescales are appropriate and necessary to maintain the quality and +relevance of the calibration images. The specific calibration data +which must be constructed in the calibration analysis stages is listed +below. + +The IPP must generate basic calibration images using the raw bias, +dark, and flat-field (dome or twilight) images obtained by the +telescope as the input. The analysis of these images requires +relatively simple stacking of the input set of images. Outlier +rejection, both of complete input images as well as pixels within the +input stack, must be performed. In addition, each type of image +requires an appropriate normalization which may depend on the data +levels in other detectors in the input set. Each of these calibration +stages must be able to determine from the input stack if the relevant +calibration image needs to be updated and perform an initial test to +see which input images are consistent and valid. + +\subsection{Bias Images} + +Bias images may be needed to correct for structure in the bias. The +IPP must have the capability of constructing a master bias image from +a stack of raw bias frames. The input bias images, representing +offsets from the overscan level, are processed by subtracting the +overscan, including 1D structure if needed. + +The master bias frame construction uses outlier image and outlier +pixel rejection to construct a single high-quality bias frame. The +statistic used to determine pixel values from the input stack can be +set by the user to be one of the following: the sample mean, median, +and mode, robust mean, median, and mode, and the clipped mean and +median. Testing of the input images consists of constructing residual +images, in which the master bias is applied to the input images. +These images may be included or excluded from an additional iteration +of the stack on the basis of their pixel-to-pixel statistics. + +\subsection{Dark Images} + +Dark images may be needed to correct for structure in the dark +current. The IPP must have the capability of constructing a master +dark image from a stack of raw dark frames. The input dark images are +first corrected for the bias using whatever method is appropriate for +the science images. Master dark frames depend on their exposure time. +As such, the input dark frames must have a limited range of exposure +times, and the output dark frame includes the exposure time as part of +its associated metadata. + +The master dark frame construction uses outlier image and outlier +pixel rejection to construct a single high-quality dark frame. The +statistic used to determine pixel values from the input stack can be +set by the user to be one of the following: the sample mean, median, +and mode, robust mean, median, and mode, and the clipped mean and +median. Testing of the input images consists of constructing residual +images, in which the master dark image is applied to the input images. +These images may be included or excluded from an additional iteration +of the stack on the basis of their pixel-to-pixel statistics. A +collection of master dark frames with a range of exposure times are +used to determine the scaling of the dark frame as a function of +exposure time. + +\subsection{On-Off Dark Images for Light Leaks} + +A type of image which may be necessary for calibrations will be pairs +of images taken at night with the shutter closed with and without the +dome shutter closed. Such a pair of images can be used to determine +any light-leak in the camera which may contribute additional flux +across the mosaic. + +\subsection{Flat-Field Images} + +Master flat-field images must be constructed from a collection of +input flat-field images. The input flat-field images may be obtained +from any of the standard sources: the dome, the twilight sky, and the +night-time sky. The choice of flat-field input image must be +determined experimentally from observations during the commissioning +phase of the telescope. The IPP flat-field construction system must +be capable of handling any of these sources. + +An appropriate set of input images is selected on the basis of their +flux levels, time of observations, and the observing conditions. The +input flat-field images are processed (bias and dark corrected if +needed) and the resulting images are stacked. The master flat-field +construction uses image and pixel outlier rejection to construct a +single high-quality master flat-field frame. The statistic used to +determine pixel values from the input stack can be set by the user to +be one of the following: the sample mean, median, and mode, robust +mean, median, and mode, and the clipped mean and median. Testing of +the input images consists of constructing residual images, in which +the master flat-field image is applied to the input images. These +images may be included or excluded from an additional iteration of the +stack on the basis of their pixel-to-pixel statistics. + +\subsection{Mask Images} + +Preliminary bad-pixel mask images are generated on the basis of +comparison between raw flat-field images and a cleaned, stacked +master. The mask creation system accepts a collection of flat-field +images and identifies pixels which are consistently poorly flattened. +Pixels which are under-responsive are also identified as pixels to be +masked. + +\subsection{Sky \& Fringe Frames} + +Fringe-correction frames must be generated to remove the fringe +pattern caused by thin-film interference in the top layers of CCDs, +particularly in the redder passbands. Fringe correction frames may be +constructed on the basis of observations of the night-sky in the +appropriate filters or on the basis of dome fringe lamp observations. +The choice of the appropriate source will be determined experimentally +on the basis of data obtained during the commissioning phase. The IPP +must be capable of handing either source. The images are first +flattened to remove the pixel-to-pixel sensitivity variations of the +detector. The combination of multiple input fringe frames may not be +simply stacked since the amplitude of the fringe pattern varies +independently of other variations in the image. The amplitude of the +fringe pattern in the input frames is measured and the images scaled +to normalize the fringe amplitude to a consistent range (-1 to +1) for +all input images before they are combined with one of the standard +combination statistics (mean, median, mode, etc). The quality of the +input frames is tested by flattening the input image and applying the +master fringe-frame. The resulting residual image statistics are used +to select or exclude specific input images. + +\subsection{Shutter Correction Map} + +Shutter correction map images may be generated based on the timing +measurements of the shutter itself, or on the basis of dome-flat +images of decreasing exposure times down to the shortest available +exposures. + +\subsection{Low-k Sky Models} + +Large-scale background structure in images which is not caused by +thin-film interference must also be detected and corrected. Models of +this background structure may be a necessary input to the correction +procedure. The IPP must have the capability of generating image +models of the large-scale structure patterns observed with the +telescope + +\subsection{Flat-Field Correction Frame} + +Flat-field images, whether constructed from the dome, twilight, or +night-sky images, do not perfectly correct the detector response in a +consistent fashion across the full field of the camera. The IPP must +have the capability of generating flat-field photometric correction +frames on the basis of the measured photometry of objects which are +moved to a variety of locations on the detector in a sequence of +images. The flat-field correction frames analysis stage makes use of +targeted observations following a specified dither pattern, and +extracts the photometered objects from the AP Database to determine +the necessary photometric corrections. The resulting image is applied +to the master flat-field image. Testing of the correction is +performed by applying the correction to the basic master flat-field +image, applying that flat-field image to the dithered photometry +observations, and performing the object detections. Comparison of the +photometry of individual stars at different locations on the mosaic +will demonstrate the consistency of the flat-field image. + +\subsection{Non-Linearity Correction} + +The IPP must have the capability of constructing a correction for +non-linearity in the detectors. These frames are constructed from +exposures of a uniform source with a range of exposure times. The +non-linearity correction frames provide polynomial correction +coefficients or a lookup table describing the correction. There is +likely to be a single non-linear correction for each OTA detector, or +potentially for each Cell. The IPP must handle these two cases. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{System Design : Miscellaneous Tasks} + +This section discusses additional operations which are performed by +the IPP but which do not fall under the analysis of the science images +or the creation of the calibration images. + +\subsection{Retrieval} + +The retrieval stage simply retrieves images from an external source +(ordinarily OTIS at the Summit, but it could conceivably be some other +external source) and store it in the Image Server. + +\subsection{Static Sky Analysis} + +The IPP is responsible for performing object detection and analysis on +the static sky. This analysis is performed continuously (every day or +week) on those portions of the sky within 15\degree\ of the sun. In +this analysis, the object measurement is much more detailed than those +performed in the real-time analysis. The currently envisioned +parameters to be measured for every object are listed in +Table~\ref{tab:APdetections}. The parameters include the object centroid +and the position covariance matrix, the instrumental PSF magnitude and +error, local background level and error, a measurement of the +star-galaxy separation, a measurement of the object shape ($\sigma_x, +\sigma_y, \theta$), the Petrosian radius, magnitude, axis ratio, and +angle; the S\'ersic radius, magnitude, axis ratio, angle, and +parameter $\nu$, and a collection of annular aperture flux +measurements, all of which are also measured for the P4$\Sigma$ +images. In addition, the galaxy-shape parameters $Gamma_1 \& +\Gamma_2$, along with the complete `polarization' terms are measured, +as well as a collection of annular aperture flux and variance +measurements. Another unique feature of the static sky analysis is +that the object detection may be performed simultaneously on all +filters, in which case the locations and other parameters may be more +strongly constrained by simultaneously fitting between all bands. The +analysis to be performed may be substantially more complex than the +real-time analysis because of the relatively low data rate. Instead +of needing to process thousands of images per night ($\sim 350$Mpix +per second), it is only necessary to process the complete sky in a +year, or an average rate of $\sim$2 Mpix per second, or $< 1$\% of the +object analysis in the other analysis stages. + +\subsection{AstroRef: Astrometric Reference Catalog creation} + +\tbd{needs to be fleshed out substantially} + +This processing stage shall use many observations over a given time +period to fit a consistent global astrometric solution, resulting in a +high quality and internally-consistent astrometric catalog that may be +published. + +\subsection{PhotoRef: Photometric Reference Catalog creation} + +\tbd{needs to be fleshed out substantially} + +This processing stage shall use many observations over a given time +period to fit a consistent global photometric solution, resulting in a +high quality and internally-consistent photometric catalog that may be +published. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Software Hierarchy} + +In order to facilitate testing and development, and to encourage +flexibility, the IPP will be built in a layered fashion. The lowest +level functions will be written in C and collected together into a +Pan-STARRS library. These library functions will be used to write +more complex modules. The modules will be written in C but will make +use of SWIG to make their functionality available within other +languages. In particular, the modules can be tied together with a +program in C or through the use of a high-level language such as Perl, +Python, or Tcl employing the SWIG interfaces. For the high-level +functions in the operational system, the IPP will make use of Perl as +the scripting language to provide the required flow-control to tie the +modules together. + +This approach satisfies the requirement that complicated low-level +analysis steps run fast, while preserving flexibility for coding the +high-level wrappers for which the speed requirements are not so +stringent. + +\subsection{External Libraries} + +Pan-STARRS will employ several external libraries to save duplicating +functionality that is already available. These external libraries +will be wrapped by the Pan-STARRS Library, insulating the project from the +implementation details of the external libraries. Examples of the +external libraries are FFTW and SLALib. + +\subsection{Pan-STARRS Library} + +The Pan-STARRS Library will consist of C structures describing the +basic data types needed by the IPP and C functions which perform the +basic data manipulation operations. Note that a subset of the library +functions will be provided with SWIG interfaces as well to allow for +their use in the creation of the processing stages. Examples of the +Pan-STARRS Library are Fourier transforms and transforming between +pixel and celestial coordinates. The details of the Pan-STARRS +Library are specified in the document Pan-STARRS IPP PSLib +Supplementary Design Requirements Specification (PSDC-430-007), which +also addresses coding requirements detailed in the IPP PS-1 SRS +(PSDC-430-005), Section 3.3. + +\subsection{IPP Modules} + +The IPP analysis stages are broken down into modules which represent +specific functional operations. The modules will be written in C +using the Pan-STARRS Library functions and will be grouped into a +Pan-STARRS Module Library. The modules will be provided with SWIG +interfaces to all public APIs for their use in processing stages. +Examples of modules are overscan subtraction and image combination. +Some modules (e.g.\ find objects on an image) will be used by multiple +stages. The details of the Pan-STARRS Modules are specified in the +document Pan-STARRS IPP Modules Supplementary Design Requirements +Specification (PSDC-430-012), which also addresses coding requirements +detailed in the IPP PS-1 SRS (PSDC-430-005), Section 3.3. + +\subsection{IPP Stages} + +The major IPP processing tasks are organized into stages, which +consist of multiple modules. Each stage represents a collection of +complex operations performed on a single data entity. Each stage +therefore represents the maximum amount of effort which can be +performed in serial without interaction between parallel threads. The +stages will be written in Perl, linking the modules together. +Examples of stages are Phase 2 (detrend images) and Phase 4 (combine +images from multiple telescopes and search for transients). + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Interfaces} +\label{sec:interfaces} + +\subsection{Internal Interfaces} + +Internal interfaces consist of interactions between the analysis +scripts and the IPP Metadata Database, Image Server or AP Database. +There are also interfaces between the IPP Scheduler, Controller, and +the Metadata Database. + +The science and calibration image processing pipelines make requests +for images from the Image Server, metadata from the Metadata Database, +and push their results back onto the Image Server and Metadata +Database. The Scheduler specifies analysis tasks and sends them to +the Controller, and determines the next action based on the contents +of the Metadata Database. The various subsystems specify the API for +client / server interactions, and are discussed in their individual +section. Commands will be sent using either text-based commands, SOAP +or an equivalent protocol. The format of the exchanged data may be in +one of several forms discussed below. + +FITS Images will be used to transport images between the components of +the IPP. Non-standard FITS images representing triangular collections +of pixels may be used to store the static sky images. + +FITS Tables will be used to store and transport tabular data, +especially large queries from database subsystems. The Auto-coding +technique discussed in Appendix~\ref{sec:AutocodeIO} is used to define many +different table interactions. + +XML files will be used to store and transport data which is not +well-suited to the rectangular form of FITS Tables. Hierarchical data +concepts and variable-length structures fall in this class. Examples +include mosaic astrometry description information and configuration +information. + +SQL queries and C wrappers of SQL queries will be used as the direct +interface to the databases. + +Within IPP and Pan-STARRS in general, process-to-process communication +will be defined through auto-coded APIs which support a limited and +validated communication protocol. The APIs will be coded based on a +table which defines the allowed command set and the grammar to be +used. This mechanism will allow a single code block to define +inter-process communication methods for many Pan-STARRS subsystems, +including, within the IPP, the Scheduler-Controller communications. + +\subsection{External Interfaces} + +This subsection describes the interfaces between the IPP and other +Pan-STARRS systems and the external clients. The interfaces are +illustrated in Figure~\ref{fig:overview}. + +\subsubsection{OTIS} + +The IPP Scheduler may query OTIS for a list of new images and +metadata. The locations of those images in the Summit Pixel Server is +sent back as a table, and all metadata may be sent to the IPP as a +collection of FITS Tables. The IPP also may send quality assessment +information for each FPA and major frame by writing out FITS tables +and notifying OTIS of the presence of the new tables. + +\subsubsection{Camera} + +Images are pulled from the Summit Pixel Server, part of the Camera +team's purview. The locations of the files are sent by OTIS. IPP may +grab these via {\tt http} commands or via {\tt NFS} or another network +file exchange protocol. The IPP notifies OTIS (and Camera) when each +image has been received. + +\subsubsection{PSPS} + +Data will be sent to PSPS from the IPP as part of a daily or weekly +analysis process on the Static Sky. The data will be pushed from the +IPP to PSPS when they are available. The data to be transfered +include: +\begin{itemize} +\item Static Sky images - to be transferred as FITS images or + FITS triangular image regions. +\item Postage Stamps - to be transferred as FITS images. +\item Metadata tables - to be transferred as FITS tables +\item Detections \& Object associations - to be transferred as FITS tables. +\end{itemize} + +\subsubsection{MOPS} + +Data will be sent to MOPS from the IPP as part of the Phase 4 +analysis. The data will be pushed from the IPP to MOPS when they are +available. The data to be transfered include: +\begin{itemize} +\item Image Metadata tables - to be transferred as FITS tables +\item Orphaned Detections - to be transferred as FITS tables +\end{itemize} + +\subsubsection{Other Preferred Client Science Pipelines} + +These cannot be completely defined until the Clients are defined and +their requirements are specified. The expectation is that the data +products will be the same as for the MOPS. The data will be pushed +from the IPP to the Client Science Pipeline when they are available. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Computer Hardware} +\label{sec:Hardware} + +\subsection{PS-1 Cluster Design} + +The PS-1 IPP computer system is designed as a cluster of 'fat bricks': +computers with both processing power and large amounts of local disk +storage. These computers are large rack-mount boxes with space for +10s of disks (24 and 36 disk cases are available) and a motherboard +with two CPUs and two Gig-E ethernet ports. One set of machines is +specified for storage and processing of the individual OTAs up through +Phase 2 (the `OTA nodes'), another set of machines are specified for +storage of the Static Sky and processing of data from Phase 3 and +Phase 4 (the `Sky nodes'). Other machines will be necessary to +support the Metadata DB and the AP DB. + +The IPP PS-1 SRS (PSDC-430-005) specifies the processing throughput +requirements for the IPP. Benchmark tests of the IPP processing +algorithms have been used to drive the design needed to achieve the +throughput requirements. The details of this study are presented in +the IPP Computational Challenge (PSDC-400-006), summarized here. The +analysis measures the processing time (excluding I/O) for both Phase 2 +and Phase 4 on an Intel Pentium 4 processor, and expresses the +processing time in GHz-seconds, under the assumption that a machine +with the same architecture and twice the processor speed will perform +the same analysis in half the time. This is probably a valid +assumption within a limited range on hardware using the same +architecture. Independent tests show that 32-bit Pentium processors +perform somewhat slower (up to a factor of 2) than equivalently rated +64 bit Opteron processors. This discrepancy makes the measured +numbers somewhat conservative, and compensates for the simplified +analysis performed. The benchmarks show that the Phase 2 analysis +takes 12000 GHz-seconds for a complete major frame (4 FPAs) while the +Phase 4 analysis takes 7800 GHz-seconds for the same major frame. + +The total data I/O required for each processing node, both locally to +disk and across the network to other machines, has also been measured. +These numbers in turn depend on whether the data is optimally stored +on the OTA nodes (raw images matched to their calibration images) or +if the data are randomized across the storage nodes. There are also +differences in the analysis for the number of bits per pixel and the +number of calibration images used in the processing. For PS-1, the +`minimal' data set is appropriate, resulting in a total Phase 2 I/O of +21 GBs per major frame and a total Phase 4 I/O of 36 GBs. The +randomized numbers are used as a conservative estimate, under the +assumption the network, not local disk access, is the dominant I/O +bottleneck. + +The analysis assumes each CPU (rated at 2.2 GHz) is associated with +one RAID array (maximum throughput 110 MB/sec) and one network +controller (maximum throughput 70 MB/s). In this case, given the CPU +load and I/O throughput above, Phase 2 will require a total of 190 +seconds of I/O and 5500 seconds of processing distributed across the +cluster. Likewise, the Phase 4 analysis will require a total of 330 +sec of I/O and 3500 seconds of processing. Given the 160 seconds +available per major frame, these numbers imply a total of 63 +processors are needed to keep up with the processing and I/O load. + +The other major driver on the IPP PS-1 cluster is the data storage +requirements. It is necessary to store the raw images from the entire +AP Survey, the MOPS Verification Program (MVP) and the IPP +Verification Program (IVP), and to have storage enough to represent +the Static Sky by the end of the two year mission. These storage +requirements as a function of time are shown in +Figure~\ref{fig:StorageProfile}. Based on the PS-1 Design Reference +Mission (PSDC-230-001), by the end of the second year, the total +storage requirements for raw images and the Static Sky will be 850 TB, +along with and an additional 55 TB needed for the AP DB storage + +To meet these requirements, the IPP cluster is designed to use fat +bricks which will be capable of holding 24 disks each. The 5U / 24 +disk rack mount computer cases are one of the highest density +solutions currently available. A 4U / 36 disk box is also available +and will be considered. The disk purchases will be staggered in three +waves. Before PS-1 goes on the sky, the first 1/3 of the disks (600 +disks total) will be purchased. Since the lead time for disks is +fairly short, the purchase will be made only when other portions of +Pan-STARRS are clearly on a timeline to success. After 9 months +(tentatively 2006 September), the next 1/3 of the disks will +purchased, and the remaining disks 9 months after that (tentatively +2007 June). Using conservative estimates of the available disk sizes +at these purchase dates (400 GB, 600 GB, and 900 GB), and allocating 1 +of 12 disks to the RAID and 10\% of the volume to file system and +binary Gigabyte overheads, the disk purchases outlined above result in +a total volume after the last purchase of 950 TB. This meets the +requirements with 10\% spare excess. The disk volume profile is also +shown in Figure~\ref{fig:StorageProfile} and shows that the disk space +will be available in the time it is required. + +The total number of computers to be purchased is 80. This provides +the 1800 disk slots and more than enough processors to meet the +processing requirements. This also leaves 5 live spare machines. + +There are two details which are not included in the analysis above: +compression and replication. Compression of the older raw data will +reduce the volume requirements by a factor of roughly two. However, +replication of the data across the network is necessary to ensure the +data against catastrophic failures on a single machine. Replication +doubles the total data space needed. These two factors will tend to +cancel each other, and are ignored in the calculations above. + +The IPP PS-1 clusters will have the following allocations of computers +from this cluster: +\begin{itemize} +\item Phase 2 Nodes: 32 +\item Phase 4 Nodes: 30 +\item AP Database: 10 +\item Metadata Database: 1 +\item Image Server Database: 1 +\item Controller / Scheduler: 1 +\end{itemize} +This distribution meets the projections for computational power for +each of these data systems, and leaves 5 computers as live spares for +redundancy. + +\subsection{PS-1 Cluster Expected Reliability} + +With 80 computers and 1920 disks, component failures are inevitable. +The cluster design and management must be chosen to minimize their +impact on operations and data integrity. + +There are several factors which reduce the cluster's exposure to +hardware failures. First, the use of RAID controllers and RAID-5 +striping of the data will protect the data on a single RAID set +against the failure of a single disk in the array. Second, +duplication of data across the cluster will protect against +catastrophic failures of the array (loss of two disks, loss of the +array controller card). Finally, the flexibility of the distributed +computing plan minimizes the impact the loss of individual machines +has on operations by making changes in the data and processing +assignments on the cluster a trivial matter. + +The components which are most likely to fail in the experience of our +team are, in order: hard drives, RAID controllers, ram, power +supplies, and other components. The hard drive and RAID controller +failure rates are by far the dominant concerns as they potentially +affects the data integrity. + +Most sources (REFS: UCSD article, Samsung White Paper) currently imply +hard disk failure rates (MTBF) in the range 400,000 hours and 500,000 +hours. These are used as an upper limit, with the more historically +conservative value of 100,000 hours used instead. With 1920 disk, +this MTBF implies a failure of one disk every 2.2 days. Since the +disks are in a RAID which reports the disk failures immediately and +drops the array into degraded mode, these failures will not have a +huge impact on the operations, and recovery time is only 10s of +minutes. This failure rate implies that the maintenance plan must +include checks for hard disk failures on a daily basis, and should +make use of email notification and early warning information (ie, +SMART messages). + +A catastrophic failure for the array would require two of the 12 disks +to fail before the first failed disk is replaced. Assuming that +maintenance is poor and it is possible for a disk to take 1 week to +be replaced, the probability of a catastrophe is 1.8\% each time the +first disk fails. Combined with the disk failure rate, RAID +catastrophes are expected 6 times over the 2 year operation of PS-1. +These numbers can be used as a guideline for the level of support +needed to avoid these RAID failures. Note that these 6 failures +should not cause loss of data since the data is duplicated across the +cluster, but they require over 1 day for recovery (as the entire array +must be replicated across the network). + +A detailed IPP computer cluster commissioning and maintenance plan is +specified in the document `Pan-STARRS PS-1 IPP Cluster Support' +(PSDC-430-014). + +\begin{figure} +\begin{center} +\resizebox{6in}{!}{\includegraphics[angle=-90]{pics/ps1_ipp_storage.ps}} +\caption{ \label{fig:StorageProfile} Storage Profile} +\end{center} +\end{figure} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\clearpage +\appendix +\section{Image Server Database Table Contents} +\label{sec:ImageServerTableContents} + +Tables~\ref{tab:ImageServerTables:SO} - \ref{tab:ImageServerTables:VOL} list +the basic contents of the Image Server database tables. + +\begin{table}[bh] +\begin{center} +\caption{Storage Object Table Contents\label{tab:ImageServerTables:SO}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ +\hline +\code{so_id} & integer & internal storage object identifier \\ +\code{ext_id} & string & external storage object identifier (file ID) \\ +\code{comment} & string & user description of object \\ +\code{epoch} & date/time & last date of access \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Instance Table Contents\label{tab:ImageServerTables:INT}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ +\hline +\code{ins_id} & integer & internal instance identifier \\ +\code{so_id} & integer & key to storage object table \\ +\code{uri} & string & location in hardware collection \\ +\code{sha1sum} & string & checksum information \\ +\code{assigned_location} & boolean & is location user-specified? \\ +\code{epoch} & date/time & last date of access \\ +\code{atime} & date/time & last date of access \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Volume Table Contents\label{tab:ImageServerTables:VOL}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ +\hline +\code{vol_id} & integer & internal volume identifier \\ +\code{uri} & string & node name \\ +\hline +\end{tabular} +\end{center} +\end{table} +\clearpage + +\section{Metadata Database Table Contents} +\label{sec:MetadataTableContents} + +Tables~\ref{tab:WeatherTable} -- \ref{tab:overlaps} list the basic contents of +each of the Metadata Database tables listed in Section~\ref{sec:Metadata}. + +\begin{table}[bh] +\begin{center} +\caption{Weather Table: some sample weather points\label{tab:WeatherTable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time the weather information was measured. \\ +Temperature 01 & float & The external temperature \\ +Temperature 02 & float & The temperature at top of the dome \\ +Temperature 03 & float & The temperature on the primary mirror \\ +Humidity & float & The relative humidity. \\ +Pressure & float & The (external) atmospheric pressure. \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{SkyProbe Transparency Table (sample entries)\label{tab:SkyprobeBVTable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time the SkyProbe image was taken. \\ +Filter & string & Filter used for SkyProbe image. \\ +Transparency & float & The derived transparency. \\ +Number of stars & int & The number of stars used to measure the transparency. \\ +Astrometry & coords & The astrometry used on the SkyProbe image. \\ +Exposure time & float & The exposure time of the SkyProbe image. \\ +Sky brightness & float & The measured sky (surface) brightness, counts / second \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Skyprobe Line Absorption Table (sample entries)\label{tab:SkyprobeATable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time the LRProbe observation was taken. \\ +Disperser ID & string & ID of the dispersing element \\ +Atm Component 1 & float & The strength of the 1st atmospheric component. \\ +Atm Component 2 & float & The strength of the 2nd atmospheric component. \\ +Atm Component 3 & float & The strength of the 3rd atmospheric component. \\ +Disperser ID & string & ID of the dispersing element \\ +Number of stars & int & Number of stars used to measure the absorptions. \\ +Astrometry & coords & The astrometry used on the LRProbe image. \\ +Exposure time & float & The exposure time of the LRProbe image. \\ +Sky brightness & float & The measured sky (surface) brightness, in physical units. \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Skyprobe Line Emission Table (sample entries)\label{tab:SkyprobeETable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time the LRProbe observation was taken. \\ +Disperser ID & string & ID of the dispersing element \\ +Atm Component 1 & float & The strength of the 1st atmospheric component. \\ +Atm Component 2 & float & The strength of the 2nd atmospheric component. \\ +Atm Component 3 & float & The strength of the 3rd atmospheric component. \\ +Continuum & float & The strength of the continuum emission. \\ +Disperser ID & string & ID of the dispersing element \\ +Exposure time & float & The exposure time of the LRProbe image. \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{DIMM Measurements Table\label{tab:DimmTable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time the DIMM observation was taken. \\ +$\sigma_x$ & float & Raw dispersion in $x$. \\ +$\sigma_y$ & float & Raw dispersion in $y$. \\ +FWHM & float & Dervied seeing full width at half maximum. \\ +RA & float & The coordinates of the measured star. \\ +DEC & float & The coordinates of the measured star. \\ +Exposure time & float & The exposure time of the DIMM observation. \\ +Telescope ID & string & source of the DIMM data \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Near IR Wide-field Camera Results Table\label{tab:NIR-Table}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time the NIR observation was taken. \\ +Sky brightness & float & The sky (surface) brightness in the NIR observation. \\ +Sky variance & float & The variance in the sky (surface) brightness. \\ +Astrometry & coords & The astrometry used on the NIR image. \\ +FOV X & float & field width \\ +FOV Y & float & field height \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Dome Status Table\label{tab:DomeStatusTable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time for which the dome status is valid. \\ +Azimuth & float & The azimuth of the dome. \\ +Open status & boolean & Whether the dome is open or not. \\ +Lights status & boolean & Whether lights are on in the dome or not. \\ +Track status & boolean & Whether dome is tracking telescope or not. \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Telescope Status\label{tab:TelescopeStatusTable}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Time & date/time & The time for which the telescope status is valid. \\ +Guide status & enum & The status of the guiding. \\ +Altitude & float & The telescope altitude. \\ +Azimuth & float & The telescope azimuth. \\ +RA & float & The telescope Right Ascension (ICRS $\approx$ J2000). \\ +Dec & float & The telescope Declination (ICRS $\approx$ J2000).\\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Raw FPA Images\label{tab:RawFPAs}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +ID & string & FPA image ID \\ +RA & float & Coordinates of the boresight (i.e. telescope pointing). \\ +DEC & float & Coordinates of the boresight (i.e. telescope pointing). \\ +Filter & string & Filter used for the exposure. \\ +Image Type & enum & image exposure type \\ +Exposure time & float & Exposure time for the image. \\ +Airmass & float & Airmass at which the image was taken. \\ +ObsFrame ID & int & Observation frame identification number, ties FPAs into major frame \\ +ObsGroup ID & int & Observation group identification number, ties FPAs into observing group \\ +Observer & string & The name of the observer, or the version of the telescope scheduler software. \\ +Program & string & The observing program being executed. \\ +Nchips readout & int & Number of detector chips read out \\ +Camera & string & Identification of camera source \\ +Telescope & string & Telescope used for observation \\ +Astrometry & coords & The astrometry used for the FPA. \\ +Chip Metadata & string & metadata resource file \\ +Cell Metadata & string & metadata resource file \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Pending Science Chips\label{tab:PendingChips}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +FPA ID & string & FPA image ID \\ +Chip ID & string & Chip identification number. \\ +Proc Status & enum & Current Processing Status. \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Processed Science Chips\label{tab:ProcessedChips}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +FPA ID & string & FPA Image ID \\ +Chip ID & string & Chip identification number. \\ +Status & enum & Current Processing Status. \\ +Residual Stats & float & quality statistics. \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Observation Group Information\label{tab:OBSGroup}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +ObsGroup ID & string & Identification number for the observation group. \\ +Number of images & string & Number of images in the observation group. \\ +Type & string & Type of observation. \\ +Status & string & Status of the observation group. \\ +etc & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Observation Frame Information\label{tab:OBSFrame}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +ObsFrame ID & string & Identification number for the observation frame. \\ +Number of images & string & Number of images in the observation group. \\ +Type & string & Type of observation. \\ +Status & string & Status of the observation group. \\ +etc & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Science Processing Stats\label{tab:PSStats}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Chip ID & string & The chip identification number. \\ +State & string & The state of the processing. \\ +ObsFrame ID & string & The major frame the chip belongs to. \\ +ObsGroup ID & string & The observation group the chip belongs to. \\ +P1 astrom & string & The Phase 1 astrometry results file. \\ +P2 astrom & string & The Phase 2 astrometry results file. \\ +P3 astrom & string & The Phase 3 astrometry results file. \\ +N guide stars & string & Number of guide stars used for the exposure. \\ +Astrometry stats & string & Summary statistics for astrometry (number of stars, $sigma_x$, $sigma_y$) \\ +Astrom catalog & string & The reference catalog that was used for the astrometry. \\ +Bias method & string & Method used to correct the bias. \\ +Bias stats & string & Summary statistics for bias \\ +Flat-field image & string & The flat-field image that was applied. \\ +Kernel data & & A description of the OT kernel. \\ +Flat-field stats & & Summary statistics for flat-field (sigma of sky). \\ +Mask image & string & The mask image that was applied. \\ +Mask method & string & The algorithm used to mask the bad pixels. \\ +Fringe images & string & The fringe model images that were used. \\ +Fringe stats & & Summary statistics for fringes (fringe amplitude, sky sigma) \\ +Object stats & & Summary statistics for object detection (number of objects, depth, other input parameters). \\ +Photometry data & & photometry information: magnitude zero point and other corrections. \\ +Photometry stats & & Summary statistics for the photometry (number of stars, $sigma_m$) \\ +Photom catalog & string & The reference catalog that was used for the photometry. \\ +PSF stats & & Summary statistics of the PSF. \\ +Software ver & string & Versions of each of the modules used in the processing. \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Chip / Sky overlaps\label{tab:overlaps}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Chip ID & string & The identification number of the chip. \\ +Sky Cell ID & string & The identification number of the sky cell. \\ +State & string & Processing state of overlap \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Processed Sky-Cell stats\label{tab:ProcessedSky}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Input Chips & string & Identification numbers of the chips used to produce the sky cell. \\ +PSF adjustments & string & Adjustments to the PSF. \\ +CR rejection stats & string & Statistics from the CR rejection (number of CRs, distribution, limiting flux). \\ +Image comb params & string & Parameters used for the image combination. \\ +Diff image params & string & Parameters used for the image differencing. \\ +Average weight & string & The weight of the reference image \\ +P4D object stats & string & Summary statistics of the object detection \\ +P4S object stats & string & Summary statistics of the object detection \\ +Software versions & string & Software versions of modules used in the sky cell processing. \\ +Processing stats & string & Summary statistics of the processing (CPU time, etc). \\ +\hline +\end{tabular} +\end{center} +\end{table} +\clearpage + +\section{AP Database Table Contents} +\label{sec:APDBTableContents} + +\begin{table}[bh] +\begin{center} +\caption{Images\label{tab:images}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Image ID & & \\ +time/date & & \\ +Exposure Time & & \\ +Nstars & & \\ +NX & & \\ +NY & & \\ +photcode & & \\ +Mcal & & \\ +Mcal error & & \\ +Mcal chisq & & \\ +Airmass & & \\ +Astrometry & & \\ +PSF & & \\ +flags & & \\ +Camera & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Image Overlaps\label{tab:ImageOverlaps}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Image ID & & \\ +Region Table & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Objects\label{tab:Objects}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +ID & & \\ +$\alpha$ & & \\ +$\delta$ & & \\ +$\mu_{\alpha}$ & & \\ +$\mu_{\delta}$ & & \\ +$\sigma_{\alpha}$ & & \\ +$\sigma_{\delta}$ & & \\ +$\chi^2$ position & & \\ +$N_{\rm det}$ & & \\ +$N_{\rm miss}$ & & \\ +flags & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Average Magnitudes\label{tab:AveMags}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +object ID & & \\ +$M_{\rm int}$ & & \\ +$M_{\rm ext}$ & & \\ +$\chi^2_{\rm mag}$& & \\ +$\sigma_{\rm mag}$& & \\ +photcode & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Solar System Objects\label{tab:SSObjs}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +SSO ID & & \\ +$N_{\rm det}$ & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Matched Detections\label{tab:Detections}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +$\alpha$ & & \\ +$\delta$ & & \\ +$\sigma_{\alpha}$ & & \\ +$\sigma_{\delta}$ & & \\ +$M_{\rm inst}$ & & \\ +$M_{\rm cal}$ & & \\ +$\sigma_{\rm mag}$& & \\ +photcode & & \\ +type & & \\ +flags & & \\ +time/date & & \\ +airmass & & \\ +$\sigma_{x}$ & & \\ +$\sigma_{y}$ & & \\ +$\theta$ & & \\ +object ID & & \\ +exptime & & \\ +sky & & \\ +$\sigma_{\rm sky}$& & \\ +etc & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Orphaned Detections\label{tab:Orphans}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +$\alpha$ & & \\ +$\delta$ & & \\ +$\sigma_{\alpha}$ & & \\ +$\sigma_{\delta}$ & & \\ +$M_{\rm inst}$ & & \\ +$M_{\rm cal}$ & & \\ +$\sigma_{\rm mag}$& & \\ +photcode & & \\ +type & & \\ +flags & & \\ +time/date & & \\ +airmass & & \\ +$\sigma_{x}$ & & \\ +$\sigma_{y}$ & & \\ +$\theta$ & & \\ +exptime & & \\ +sky & & \\ +$\sigma_{\rm sky}$& & \\ +etc & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Non-detections\label{tab:NonDetects}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +object ID & & \\ +$N_{\rm non-det}$ & & \\ +last time/date & & \\ +last mag & & \\ +faintest time/date & & \\ +faintest mag & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Regions\label{tab:Regions}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +$\alpha_0$ & & \\ +$\alpha_1$ & & \\ +$\delta_0$ & & \\ +$\delta_1$ & & \\ +Region ID & & \\ +Parent ID & & \\ +Nchildren & & \\ +Images & & \\ +Objects & & \\ +Detections & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Filters\label{tab:Filters}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Filter ID & & \\ +Filter name & & \\ +Photcode & & \\ +$\lambda_0$ & & \\ +$\delta_\lambda$ & & \\ +$\epsilon$ & & \\ +transmission curve& & \\ +time/date & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Photcodes\label{tab:Photcodes}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Photcode & & \\ +Telescope & & \\ +Camera & & \\ +Detector & & \\ +Filter & & \\ +Nominal ZP & & \\ +airmass terms & & \\ +color terms & & \\ +Target & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Zero Point History\label{tab:Zpts}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Photcode & & \\ +start Time/date & & \\ +end Time/date & & \\ +Zero Points & & \\ +airmass & & \\ +color & & \\ +error & & \\ +N measurements & & \\ +N stars & & \\ +photom ref set & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Distortion History\label{tab:Distortions}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +Camera & & \\ +Telescope & & \\ +distortion terms & & \\ +time/date & & \\ +residuals / error & & \\ +N stars & & \\ +N images & & \\ +astrom ref set & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{table}[bh] +\begin{center} +\caption{Database Hosts\label{tab:APDBHosts}} +\begin{tabular}{lll} +\hline +\hline +{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ +\hline +machine name & & \\ +machine ID & & \\ +\hline +\end{tabular} +\end{center} +\end{table} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\section{Software Runtime Configuration Issues} +\label{sec:RuntimeConfig} + +The IPP Software requires extensive runtime configuration information. +This includes default parameters for analysis to be performed, +descriptions of how a particular analysis is performed, locations of +data sources, and so forth. The IPP may store this information in the +Metadata Database or in configuration files available to the user. +Both methods are implemented in the current design. In either method, +the necessary parameters are identical. This section discusses the +contents of specific portions of the runtime configuration. + +\subsection{Camera Definition Information} + +Every camera which may be analysed by the IPP has differences in how +the data is represented. The IPP is built with the flexibility to +handle data from many different cameras, not just the Pan-STARRS +Gigapix cameras. This is partly to allow testing of the analysis +system on data from other telescopes, such as MegaPrime on CFHT and +Suprime on Subaru, but also to allow us to adapt to changes in the +design of the Gigapix cameras themselves. It also means the IPP +software may be used by astronomers for other analysis projects beyond +the IPP. + +Most cameras provide extensive descriptive information in the FITS +image headers when the images are read out. Typically, the location +and orientations of the individual detectors are defined by keywords +such as DATASEC and DETSEC. Other variations on these words are used +for cameras which place the pixels from multiple amplifiers in the +same FITS data segment. Other parameters, such as astrometric +information or exposure times, are stored in headers as well. It is +possible to use these header keywords to guide the analysis software, +but there are two difficulties. + +First, it is very common for different keywords to be used by +different cameras, sometimes even the same camera may use different +keywords for the same information at different times (major readout +software upgrades, for example, can be accompanied by keyword +revisions). In addition, within Pan-STARRS and the IPP, it is +necessary to have the capability to refer to the Metadata database as +the authoratative sources of some of these entries rather than the +image headers. Given this circumstance, it is at least necessary to +define the appropriate source for a given data concept appropriate to +data from a specific camera. + +The second problem arises when actually performing an analysis. In +many circumstances, the software needs to know what data to expect +even when an appropriate camera image is not available. This is +particularly true for a camera which is composed of multiple chips and +multiple amplifiers. It is a frequent circumstance than some subset +of the chips or amplifiers will either be unavailable or are invalid +for one reason or another. It is important for the software to have a +guide for what data should be available from a perfect readout of the +given camera so decisions can be made how to handle data which is not +complete. This is also important to validate that a particular +dataset, which appears to be from a known camera, actually corresponds +to that camera and has all of the necessary information where +expected. + +In order to facilitate the operation of the IPP with a variety of +cameras, and to allow the software the flexibility to change the +camera defintion dynamically, the IPP includes a collection of +software runtime configuration information which defines a given +camera. This information is represented below in the form of the +PSLib Metadata Config file, but may be stored in the Metadata Database +or in an alternate format as appropriate. + +The a single camera is represented as a Focal Plane Array (FPA), +divided into Chips, divided into Cells. For a single FPA, all imaging +data is stored in a FITS file or a collection of FITS files. Software +needs to know where in a given file or set of files to find a +particular Cell, what Cells to expect, what chips to expect, and the +relationships between those entities, etc. + +A single camera configuration file (or dataset) represents the +description of a complete FPA. In the configuration file, any +parameters which are specific to the complete FPA are placed on their +own lines. These include the definition of the keywords or database +locations. An incomplete example is given below. + +\begin{verbatim} +NCELL S32 NN +NCHIP S32 NN +EXPTIME-SRC STR HD:EXPTIME # need to specify PHU vs EXTNAME +EXPTIME-KEY STR EXPTIME +DATE-KEY STR DATE-OBS +DATE-FMT STR YYYY/MM/DD + +TYPE CELL FILENAME EXTNAME CHIP DATASEC BIASSEC +CELL.nn CELL @ROOT@CELL AMP00 CHIP.00 CF:[0,0:0,0] HD:BIASSEC +CELL.01 CELL @ID/@ID@CELL.fits AMP01 CHIP.00 DB:??? +\end{verbatim} + +\subsection{Analysis Recipe Information} + +In order to maintain flexibility in the analysis details, the IPP uses +recipes to define how a particular analysis is implemented. Each +major analysis script (eg, Phase 2) has its own recipe configuration +information, which may be stored in the Metadata Database or in the +form of the PSLib Metadata Config file. This configuration +information includes all of the user configurable parameters. Many of +these may specify a specific value, or they may specify lookup methods +(database locations, or header locations). The specifies of each +depends on the context. Below is an example recipe file for the bias +subtraction portion of Phase 2, giving several alternative options for +certain entries. Note that, for example, the overscan subtraction may +be specified as using a particular region given in the recipe file, or +on the basis of a particular header keyword. + +\begin{verbatim} +# BIAS: +BIAS.IMAGE STR NONE +BIAS.IMAGE STR FILE:bias.fits +BIAS.IMAGE STR DB:BEST +BIAS.IMAGE STR DB:CLOSE + +BIAS.OVERSCAN STR HD:BIASSEC +BIAS.OVERSCAN STR CF:[0,16:0,2048] +BIAS.OVERSCAN STR NONE + +BIAS.OVERSCAN.STATS STR MEDIAN +BIAS.OVERSCAN.STATS STR MEAN + +BIAS.OVERSCAN.FIT STR SPLINE +BIAS.OVERSCAN.FIT.NPTS S32 5 + +BIAS.OVERSCAN.FIT STR POLYNOMIAL +BIAS.OVERSCAN.FIT.ORDER S32 3 +BIAS.OVERSCAN.FIT.NBIN S32 5 +\end{verbatim} + +\section{I/O Code Autogeneration} +\label{sec:AutocodeIO} + +The IPP includes a number of data collections which have multiple +representations. A software tool will be used to automatically +generate code to provide I/O APIs to read and write these data and to +define the data structures used to carry them within a program. +Within the IPP, examples of these different data entities include +database tables (ie, in the Metadata Database), FITS Tables (to +exchange bulk data), and XML (to exchange more complete datasets). + +I/O API Autocode template (example.def): +\begin{verbatim} +Name Example +Table EXAMPLE +EXTNAME EXAMPLE + +KEY XVALUE + +# name format unit comment +XVALUE F32 pixels "x coordinate" +BINNING S32 fraction "binning factor" +NAME STR[32] string "description of entry" +\end{verbatim} + +Running autocode on such a file would generate an output header and C +files \code{example.h, example.c} with the following structure and APIs: + +\begin{verbatim} +typedef struct { + psF32 XVALUE; // x coordinate + psS32 BINNING; // binning factor + char NAME[32]; // description of entry +} Example; + +psMetadata *psFITSTableInitExample (); +psExample *psFITSTableLoadExample (char *filename, int *Nrows); +bool psFITSTableSaveExample (char *filename); + +psMetadata *psDatabaseTableInitExample (); +psExample *psDatabaseTableLoadExample (char *filename, int *Nrows); +bool psDatabaseTableSaveExample (char *filename); +psExample *psDatabaseTableLoadExampleRow (char *filename, psF32 XVALUE); +\end{verbatim} + +%\bibliographystyle{plain} +%\bibliography{panstarrs} + +\input{glossary.tex} + +\end{document} Index: unk/doc/design/ippSDRS.tex =================================================================== --- /trunk/doc/design/ippSDRS.tex (revision 6007) +++ (revision ) @@ -1,3546 +1,0 @@ -%%% $Id: ippSDRS.tex,v 1.15 2004-11-30 23:16:03 eugene Exp $ -\documentclass[panstarrs]{panstarrs} - -% basic document variables -\title{Pan-STARRS PS-1 Image Processing Pipeline} -\subtitle{System/Subsystem Design Description} -\shorttitle{IPP SSDD} -\author{Eugene A. Magnier, Paul A. Price, Josh Hoblitt} -\audience{Pan-STARRS PMO} -\group{Pan-STARRS Algorithm Group} -\project{Pan-STARRS Image Processing Pipeline} -\organization{Institute for Astronomy} -\version{00} -\docnumber{PSDC-430-011} - -% allow paragraphs to be listed in TOC for now -\setcounter{tocdepth}{4} - -\begin{document} -\maketitle - -% -- Revision History -- -\RevisionsStart -% version Date Description -DR.01 & 2004.01.01 & First draft \\ \hline -DR.02 & 2004.03.05 & Second draft \\ \hline -DR.03 & 2004.03.25 & Section reorganization \\ \hline -DR.04 & 2004.04.13 & Most sections fleshed out \\ \hline -DR.05 & 2004.04.29 & Reorganization for consistency \\ \hline -DR.06 & 2004.10.21 & Major revision in prep of PDR \\ \hline -\RevisionsEnd - -\inserttbd -\inserttbr -\pagebreak - -\tableofcontents -\pagebreak - -\listoffigures -\pagebreak -\pagenumbering{arabic} - -\section{Scope} - -\subsection{Identification} - -This document establishes Software Design Requirements for the -Panoramic Survey Telescope and Rapid Response System (Pan-STARRS) -Image Processing Pipeline (IPP) for the prototype telescope PS-1, and -is a System-level controlled specification/design description document -in the official Pan-STARRS engineering specification tree. - -\subsection{System Overview} - -The Institute for Astronomy at the University of Hawaii is developing -a large optical synoptic survey telescope system, the Panoramic Survey -Telescope and Rapid Response System (Pan-STARRS). The science goals, -priorities, top-level concept of operations with associated -operational requirements, and system performance drivers with -associated system performance requirements are described in the -Pan-STARRS Science Goals Statement (SGS). As described in this -document, The system conceptual design for Pan-STARRS utilizes an -array of four 1.8m telescopes each with a 7 degree$^2$ field of view, -giving the system an \'etendue larger than all existing survey -instruments combined (defined as the product of the collecting area -$A$ multiplied by the field-of-view solid angle $\Omega$). Each -telescope will be equipped with a 1 billion pixel CCD camera with low -noise and rapid read-out, and the data will be reduced in near real -time to produce both cumulative static sky and difference images from -which transient, moving, and variable objects can be -detected. Pan-STARRS will be able to survey up to $\approx 6,000$ -degree$^{2}$ per night to a detection limit of approximately 24$^{th}$ -magnitude. This unique combination of sensitivity and sky coverage -will open up many new possibilities in time domain astronomy including -a major goal of surveying the Potentially Hazardous Object (PHO) -population down to a diameter of $\approx 300$ meters. In addition, -the Pan-STARRS data will be used to investigate a broad range of -astronomical problems of extreme current interest concerning the Solar -System, the Galaxy, and the Cosmos at large. A prototype single -telescope system, PS-1, is being developed as a preliminary step -before construction of the complete four telescope system. - -\begin{tabular}{ll} -Project sponsor:& AFRL, United States Air Force \\ -Acquirer: & University of Hawaii Institute for Astronomy \\ -User: & Astronomical community \\ -Developer: & University of Hawaii Institute for Astronomy, participating \\ - & institutions, and associated subcontractors -\end{tabular} - -\subsection{Document Overview} - -The Pan-STARRS IPP System/Subsystem Design Description (SSDD) contains -the complete design description of the Pan-STARRS PS-1 IPP in order to -achieve the requirements specified by the Pan-STARRS PS-1 IPP Software -Requirements Specification (SRS; PSDC-430-005). The requirements flow -begun in the SGS and SCD and continued in the SRS is used to guide the -design presented here. - -\subsection{Requirements Definitions} - -The Pan-STARRS document naming scheme is PSDC-NNN-MMM-VV, where the VV -entry specifies the document version number. Where documents are -identified without the version number, the latest official version in -that series is implied. - -Open issues (TBDs) in this document are marked {\bf \color{red} in -bold red}. - -Quantities which should be reviewed (TBRs) are marked {\bf -\color{blue} in bold blue}. - -\subsubsection{``Shall''} When used in this specification, the word -``shall'' refers to an explicit requirement of a system component or -the complete system. - -\subsubsection{``Should''} When used in this specification, the word -``should'' refers to a desired characteristic of a system component or -the complete system. - -\subsubsection{``Will''} When used in this specification, the word -``will'' provides information about a characteristic of a related -system component or a complete related system. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\DocumentsInternalSection -PSDC-230-001 & PS-1 Design Reference Mission \\ \hline -PSDC-230-002 & PS-1 System Concept Definition \\ \hline -PSDC-400-006 & The Pan-STARRS IPP Computational Challenge \\ \hline -PSDC-430-004 & Pan-STARRS IPP C Code Conventions \\ \hline -PSDC-430-005 & Pan-STARRS IPP PS-1 Software Requirements Specification \\ \hline -PSDC-430-006 & Pan-STARRS IPP Algorithm Design Document \\ \hline -PSDC-430-007 & Pan-STARRS IPP PSLib Supplementary Design Requirements Specification \\ \hline -PSDC-430-010 & Pan-STARRS IPP Perl Code Conventions \\ \hline -PSDC-430-012 & Pan-STARRS IPP Modules Supplementary Design Requirements Specification \\ \hline -PSDC-430-014 & Pan-STARRS IPP PS-1 Cluster Support \\ \hline -\DocumentsExternalSection -Posix Standard & Open Group Based Specifications Issue 6, IEEE Std 1003.1, 2003 \\ -\DocumentsEnd - -\section{Subsystem Overview} - -The Pan-STARRS Image Processing Pipeline (IPP) performs the image -processing and data analysis tasks needed to enable the scientific use -of the images obtained by the Pan-STARRS telescopes. The primary -goals of the IPP are to process the science images from the Pan-STARRS -telescopes and make the results available to other systems within -Pan-STARRS. It also is responsible for combining all of the science -images in a given filter into a single representation of the -non-variable component of the night sky defined as the ``Static Sky''. -To achieve these goals, the IPP also performs other analysis functions -to generate the calibrations needed in the science image processing -and to occasionally use the derived data to generate improved -astrometric and photometric reference catalogs. It also provides the -infrastructure needed to store the incoming data and the resulting -data products. - -The IPP inherits lessons learned, and in some cases code and prototype -code, from several other astronomy image analysis systems, including -Imcat (Kaiser), the Sloan Digital Sky Survey (REF), the Elixir system -(Magnier \& Cuillandre), and Vista (Tonry). Imcat and Vista have a -large number of robust image processing functions. SDSS has -demonstrated a working analysis pipeline and large-scale database -system for a dedicated project. The Elixir system has demonstrated an -automatic image processing system and an object database system for -operational usage. - -The users of the IPP output are all systems internal to the Pan-STARRS -project. They consist of: 1) the Preferred Science Clients, which -receive specified data products on short timescales. 2) the Moving -Object Processing System (MOPS), which is one of the Preferred Science -Clients, but has the distinction of being a component funded by -Pan-STARRS. It will receive the detections of non-stationary -transient objects. 3) the Published Science Products Subsystem -(PSPS), which will receive all data products of interest to the -community external to the Pan-STARRS data processing systems, and will -act as the long-term archive and publishing clearinghouse. - -The IPP receives data from two Pan-STARRS subsystems: the Camera, from -which it receives the large volume of image data, and OTIS -(Observatory, Telescope and Infrastructure Subsystem), from which it -receives metadata describing the images and the environmental -conditions. The primary IPP hardware system on which the software -operates will probably not be located at the summit. Instead, because -of thermal, power, and space constraints, the hardware will likely be -located in a facility off the mountain. A subset of processing tasks -may eventually be assigned to machines at the summit if justified by -the savings in data transfer time and cost. - -The Pan-STARRS camera produces images consisting of multiple chips -(Orthogonal Transfer Arrays or OTAs), each consisting of multiple -cells (continuous set of pixels). The baseline design for the -Pan-STARRS camera contains 64 chips each with 64 cells. - -This document defines the design requirements of the IPP for the PS-1 -prototype telescope. Much of the IPP design for PS-4 will be -identical to or closely based on the PS-1 implementation. The -software organization and the infrastructure systems are expected to -be identical, with minor improvements in details. The type of -analysis steps to be performed will be nearly identical, with some -additional details added for PS-4 to improve the accuracy. - -Although generally very similar, in terms of the IPP PS-1 differs from -the complete PS-4 system in several specific ways. First, with only -one telescope and camera, the data throughput rate is substantially -reduced to a maximum of 1 64-OTA image per 40 seconds rather than 4. -Since PS-1 is a prototype for testing the Pan-STARRS hardware and -software subsystems, the observing strategy is not a fixed quantity. -The PS-1 Design Reference Mission (PSDC-230-001) provides some -guidelines for the types of observing tests which will probably be -performed, including possibly starting an Astrometric and Photometric -Survey which will eventually cover the entire $3\pi$ steradians of the -sky accessible to PS-4. As a prototype, it is expected that much of -the data collected by PS-1 will be processed multiple times to test -and tune the analysis steps. Compare with PS-4, this difference in -approach has implications for the storage required by PS-1: rather -than delete images soon after they have been used, raw images from -demonstration observations must be stored for at least the first two -years of PS-1 operations. The PS-1 Design Reference Mission is used -as an upper limit for these storage requirements to drive the hardware -design. - -\subsection{System Design Decisions} - -Since Pan-STARRS is a survey project, all data from the telescopes -will be uniformly analyzed by the Pan-STARRS Image Processing Pipeline -(IPP), and the appropriate resulting data products made available to -internal and external science analysis systems as they become -available. The processing performed by the IPP on the science images -will consist of detrending and object detection for the individual -images, combination of multiple overlapping images and further object -detection, subtraction of a reference (static-sky) image and detection -of residual objects, update of the static sky images, and detailed -object analysis of the static sky images. In addition, the IPP will -produce improved astrometric and photometric reference catalogs on an -as-needed basis. The output data products from the IPP consist of the -calibration images, reduced images from the individual telescopes, -combined images, difference images, the static sky image, object -photometry, and reference astrometry and photometry. - -The requirements for the IPP, as identified in the PS-1 IPP SRS -(PSDC-430-005) fall into several broad categories: data analysis -precision, throughput, system reliability, flexibility, testability, -and traceability. The details of the analysis tasks are specified in -order to achieve the precision. The architectural design as discussed -below is motivated by the need for reliability and flexibility. The -hardware organization and the distributed/parallel processing model is -motivated by the throughput requirements. The need for flexibility -and testability drives the software organization. The need for simple -testing procedures drives both the software organization and the -separation of the system architecture into different infrastructure -elements. - -\subsection{Analysis Tasks and Stages} - -Specific programs are required to perform the processing steps listed -above. These can be divided into well-defined analysis stages, each -of which operates on a particular unit of data, such as a single OTA -image or a collection of astronomical objects. Analysis tasks -representing the different analysis stages are performed on the IPP -computer cluster. Note the distinction between the generic analysis -{\em stage} and a specific analysis {\em task}. An analysis stage -represents a type of analysis which is performed, such as the basic -image calibration and object detection analysis. An analysis task is -a particular realization of an analysis stage, e.g., the analysis of -OTA number 61 from exposure 654321 to produce a specific set of output -data products. The analysis stages are discussed in detail in -Section~\ref{sec:AnalysisStages}. - -A particular stage may process individual images, collections of -images, or derived data products. Because of the nature of the image -data, many of the analysis stages can be run in parallel if needed to -increase the processing throughput. For example, the analysis of a -chip in one image does not depend on the results from another chip. - -\subsection{Architectural Components} - -\begin{figure} -\begin{center} -\resizebox{6in}{!}{\includegraphics{pics/IPPoverview}} -\caption{ \label{fig:overview} IPP System Overview} -\end{center} -\end{figure} - -In order to achieve the required functionality, the IPP provides an -infrastructure within which the Analysis Stages described above are -executed. In order to facilitate the subsystem testing, the IPP -software infrastructure has been divided into a number of -clearly-defined architectural software units as follows: - -\begin{itemize} - -\item {\bf Image Server:} This component is a large data store for all - images used by the IPP, including the raw images from the telescope, - the master calibration images, the reference static-sky images, and - any temporary image data products produced by the IPP. The Image - Server accepts the incoming data and stores it until it is no longer - needed by other portions of the IPP. The Image Server is not - restricted to imaging data: it is capable of storing any large data - files which are not well-suited for inclusion in a more structured - relational database, and for which access needs to be widely - available beyond the individual process which created the file. - -\item {\bf Metadata Database:} This component stores the data which is - not directly related to images or astronomical objects, but which is - needed to perform the IPP analyses. The metadata may include the - summary weather information for each night, or details about the - filters, camera, telescopes, etc. Note that the IPP Metadata - Database is not required to retain all archival engineering data - from all of Pan-STARRS; other Pan-STARRS subsystems use their own - internal databases to store engineering metadata and only the - necessary subset is transferred to the IPP Metadata Database. - -\item {\bf Astrometry \& Photometry Database (AP DB):} This component - stores and manipulates astronomical objects detected in various - images, as identified above, including individual measurements of - objects on the images, the summary information about those objects, - and reference object data. It also provides mechanisms for users to - query and manipulate the objects and detections. - -\item {\bf IPP Controller:} In order to achieve the required - processing throughput for the IPP analysis stages, it is necessary - to use distributed computing processes on a large number of - computers. The IPP Controller manages the collection of analysis - tasks performed on these machines. - -\item {\bf IPP Scheduler:} This component is a decision-making - mechanism which guides the operation of the IPP. It evaluates the - currently available collection of data, identifies the necessary - analysis, and assigns the analysis tasks to the IPP Controller. - -\end{itemize} - -The relationship between these software units is shown in -Figure~\ref{fig:overview}. This figure also shows the interactions -between the IPP and other Pan-STARRS systems. The architectural -components are discussed in detail in Section~\ref{sec:ArchComponents}. - -\begin{figure} -\begin{center} -\resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}} -\caption{ \label{fig:hardware} IPP Hardware Organization} -\end{center} -\end{figure} - -\subsection{IPP Hardware Organization} - -The IPP will utilize substantial computer resources, both in terms of -computational power and in terms of data storage and network -bandwidth. The IPP requires relatively large amounts of data storage -space, primarily for the image data. Image data is organized in two -categories. First, there is the per-OTA data -- data associated with -specific OTAs, including the raw images, the calibration images, and -temporary processed images at various stages. Second, there is the -data associated with the static sky imagery, which is in turn -organized into smaller sky-cell units. In addition to image data, -there are the somewhat smaller data entities of the Metadata Database -and AP Database. - -The computer hardware is organized into nodes which provide both data -storage and computational resources. The data storage nodes are -divided into three classes: those which deal with the per-OTA image -data, those that provide the storage for the static sky images, and -those that provide the storage for the other data systems, the -Metadata Database and the AP Database. In addition, the computational -tasks related to the individual images take place on the per-OTA -storage nodes and the processing of stacks of images takes place on -the static sky storage nodes. - -Figure~\ref{fig:hardware} presents the basic concept for the hardware -organization for the IPP. This diagram shows the two types of compute -nodes: (1) OTA-level processing and storage nodes and (2) Static Sky -processing and storage nodes. Also shown are two switches which -divide the network into OTA and Static-Sky portions. In such an -organization, the inter-switch communication must meet the throughput -needs between these network portions (though a single switch may also -be used if its backplane capacity is sufficient). The additional data -systems (Metadata Database and AP Database) are also shown. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{System Design : Architectural Components} -\label{sec:ArchComponents} - -\subsection{IPP Image Server} - -\subsubsection{Corresponding Requirements} - -The Image Server must meet the requirements specified in Section 3.4.1 -of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The specified design -is chosen to meet requirements 3.4.1.3, and 3.4.1.5. The other three -requirements (3.4.1.1, 3.4.1.2, and 3.4.1.4) depend on the volume and -capabilities of the hardware, and are addressed in -Section~\ref{sec:Hardware}. - -\subsubsection{Image Server Overview} - -The IPP Image Server is a repository for all images and other large -data files required by the IPP. Along with the storage hardware, it -provides tools for managing the distribution of these large data files -and for accessing the files. Data files stored by the IPP Image -Server include the raw images, the calibration images, intermediate -processing stage images as needed, final processed images, difference -images, image subsections, and any large non-imaging data files needed -by the IPP. The IPP Image Server must retain the files for as long as -they are needed by the IPP. - -The IPP Image Server is a parallel storage system. It stores data -across a collection of computer nodes, each with their own data -storage resources. Any single file is stored on only a single -computer and storage device. In order to achieve the data throughput -requirements, the IPP Image Server may distribute the images across -the processor nodes in an organized fashion, i.e., associating -specific machines with specific detectors. It is not the -responsibility of the IPP Image Server to determine which computer -should be associated with a specific data concept (Chip / region of -sky), but it must enable the association of a particular file with a -particular machine. - -There are three data concepts relevant to the IPP Image Server: -\begin{itemize} -\item {\bf Storage object:} This represents a single, unique data - entity in the Image Server. - -\item {\bf Instance:} A single copy of the storage object in the Image - Server. In general, a given storage object may have several instances - in the Image Server, normally on different computer nodes. - -\item {\bf File ID:} This is the identifier of a particular storage - object in the Image Server. The file ID is simply a unique string, - equivalent to the filename in a UNIX file system. -\end{itemize} - -The Image Server provides file pointers (in C), handles (in Perl or -Python), or file names corresponding to the instances of the storage -objects. The Image Server provides the data organization but does not -define a file system; it assumes the existence of an appropriate file -system which makes the files visible as local files. This -may be done over many machines with a network file system such as NFS -or GFS. - -The IPP Image Server provides the storage and access mechanisms, but -it does not include any logic or information about the data. The -Image Server does not, e.g., monitor the age of images and delete them -on some schedule. - -As shown in Figure~\ref{fig:ImageServer}, the IPP Image Server -consists of the following components: - -\begin{itemize} -\item Image Server storage hardware -\item Image Server database -\item Image Server daemon -\item Image Server client APIs -\item Image Server maintenance tools (not shown) -\end{itemize} - -\begin{figure} -\begin{center} -\resizebox{4.5in}{!}{\includegraphics{pics/ImageServer}} -\caption{The components of the IPP Image Server.} -\label{fig:ImageServer} -\end{center} -\end{figure} - -\subsubsection{IPP Image Server Client APIs} - -Clients interact with the IPP Image Server via a small number of C -APIs. Bindings are also provided for Perl and Python and UNIX shell -commands in some cases. The client commands are: - -\begin{itemize} -\item {\tt new object}: create a new storage object in the Image - Server. This function takes as input the file ID and returns a - C-style file pointer or a Perl file handle to the instance of the - storage object. The arguments to the function include an optional - node name on which the new storage object must be located. If this - target is not given, the Image Server places the new storage object - on an appropriate machine from the pool, though the details need to - be specified. - -\item {\tt open object}: open an instance of an existing storage - object, as identified by the file ID. This function may also - specify the node on which the object should be opened (if an - instance of the object is not stored on that node, the function - returns an error). On success, the function returns a file pointer. - -\item {\tt find object}: return a list of filenames in the UNIX name - space associated with the storage object identified by the given - file ID. Since there are in general multiple instances for a given - storage object, this function returns the collection of all - available instances. These may be freely opened by the client - server using the standard \code{fopen} functions. - -\item {\tt stat object}: returns status information about the - specified storage object, including the number of instances of the object. - -\item {\tt replicate object}:a new instance of the given storage - object. The target node may be optionally specified, otherwise an - appropriate node is selected. - -\item {\tt cull object}: removes one of the instances of the storage - object. The input parameters may optionally specify the target - machine to delete. - -\item {\tt delete object}: deletes all instances of the storage object - and sets the storage object status to {\tt deleted}. -\end{itemize} - -\subsubsection{IPP Image Server Daemon} - -The Image Server client requests are mediated via the Image Server -daemon. Communication between the clients and the server is via SOAP -implementing the commands above. The identity of the machine on which -Image Server daemon runs is part of the Image Server configuration -information. - -\subsubsection{IPP Image Server Database} - -The IPP Image Server daemon uses a database to store the information -about the data storage objects, their instances, and the available -hardware resources. A {\tt mysql} database engine is used to manage -the database table. The database tables defined for the Image Server -are listed in Table~\ref{tab:ImageServerTables}, and their contents are -listed in Appendix~\ref{sec:ImageServerTableContents}. This database -engine need not be the same one used for other IPP subsystems. -% -\begin{table}[ht] -\begin{center} -\caption{Image Server Database Tables\label{tab:ImageServerTables}} -\begin{tabular}{ll} -\hline -\hline -{\bf Table Name} & {\bf Description} \\ -\hline -\code{storage_object} & all storage objects known to Image Server \\ -\code{instance} & all instances of all storage objects \\ -\code{volume} & data storage devices known to Image Server \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\subsubsection{IPP Image Server Storage Hardware} - -The IPP Image Server manages data across a collection of computers and -possibly on multiple storage devices on those computer nodes. The -Image Server maintains a table of the available data volumes. The -Image Server tracks information about each volume such as the total -capacity, the current capacity, the association between computer and -data volume. - -\subsubsection{IPP Image Server Maintenance Tools} - -The IPP Image Server provides a collection of administration tools -which allow for maintenance. These are operations which may be -automatically scheduled by the IPP or which may be initiated by a -human via a command-shell interface. The maintenance functions -include migrating data between nodes to re-balance the available space -(this would only occur for instances which have not been placed on a -specific node by the client API). Other functions include checking -for file corruption, which involves sweeping all files on a data -volume and comparing the calculated file checksum to the currently -recorded value. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Metadata Database} -\label{sec:Metadata} - -\subsubsection{Corresponding Requirements} - -The Metadata Database must meet the requirements specified in Section -3.4.2 of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The specified -design is chosen to meet requirements 3.4.2.1, 3.4.2.2, 3.4.2.3, -3.4.2.4, 3.4.2.5. - -\subsubsection{Overview} - -The IPP Metadata Database acts as a repository for non-pixel data -needed by the IPP subsystems. This includes the image metadata, the -environmental data, system configuration data and system reference -data. The Metadata Database is required to save the non-ephemeral -data for the lifetime of the project for future reference and -additional analysis. The Metadata Database may be used in close -coupling with the analysis pipelines to store temporary data either -within or between stages of the analysis. In this scenario, the -analysis pipeline will interact directly with the database. However, -database latency may make this scenario impractical, in which case the -database may be used for long-term storage only. In this scenario, -the data produced by analysis pipelines which is destined for the -Metadata Database may be collected and inserted by a separate, -dedicated process. Metadata which is large in volume or poorly -structured may also be stored in an appropriate container file (FITS -Table, FITS Header, XML File) in the Image Server with the Metadata DB -providing pointers to these files. - -The IPP Metadata Database is a simple database system, consisting of a -number of simple tables without extensive inter-table links. The -\code{mysql} database engine will be used to drive the database. - -\begin{table}[hb] -\begin{center} -\caption{Metadata Database Tables\label{tab:MetadataDBTables}} -\begin{tabular}{ll} -\hline -\hline -{\bf Table Name} & {\bf Description} \\ -\hline -Weather & Details on the weather, including internal temperatures. \\ -SkyProbe Transparency & Analysis of SkyProbe B \& V data. \\ -SkyProbe Absorption & Analysis of SkyProbe A data. \\ -SkyProbe Emission & Analysis of SkyProbe E data. \\ -DIMM & Summary of DIMM data analysis. \\ -NIR & Summary statistics from NIR camera. \\ -Dome Status & The time history of the dome status. \\ -Telescope Status & The time history of the telescope status. \\ -Raw FPAs & Information about the raw FPA exposures. \\ -Pending Science Chips & Science images to be processed and status. \\ -Processed Science Chips & Science images which have been migrated to the processed state. \\ -Observation Group & Details about a group of associated observations. \\ -Observation Frame & Major frame information. \\ -Science Processing stats & Details on processed cells. \\ -Chip / Sky overlaps & List of overlaps between sky cells and detectors. \\ -Processed Sky-Cell stats & Details of the sky cell processing. \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Metadata Tables} - -Table~\ref{tab:MetadataDBTables} lists the Metadata tables identified to -date for the Metadata Database. The contents of these tables are -outlined in Appendix~\ref{sec:MetadataTableContents}, with examples for -the data entries and their data types in many cases. Additional -tables will be added as necessary as the data analysis scripts are -fleshed out in detail. The Metadata Database, with a flat data -organization, is flexible enough to add additional information as it -is identified. - -\subsubsection{Metadata Queries} - -The IPP provides simple queries to the Metadata Database tables using -auto-coded APIs. These queries return a single row or a collection of -rows based on the primary key. The format of the API is identical for -all Metadata tables. New tables and APIs can be added to the IPP -system by adding to the auto-code table description files. The -auto-code API includes read and write access permissions to be set -for each table independently. See Appendix~\ref{sec:AutocodeIO} for -further information. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{AP Database} - -\subsubsection{Corresponding Requirements} - -The AP Database must meet the requirements specified in Section 3.4.3 -of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The specified design -is chosen to meet requirements 3.4.3.1 and 3.4.3.2. In order to meet -the throughput requirements, the AP Database will be distributed -across 10 Nodes independent of the Image Server Nodes. An alternative -organization of the database which will be studied will have the AP -Database co-located with the Image Server Phase 4 Nodes. - -\subsubsection{Overview} - -The AP (Astrometry \& Photometry) Database is a CSCI which stores data -related to astronomical objects derived from various sources with a -variety of associations. The AP Database deals with two related -concepts: {\em objects} and {\em detections}. The {\em objects} are -descriptions of astronomical objects while the {\em detections} are -the specific measurements of those objects, typically measured from -astronomical images. A collection of {\em detections} may be used to -derive average quantities which describe a particular {\em object}. A -third class of measurement to be considered are those supplied by -external references. Such measurements may be treated as {\em -detections}, with the caveat that access to the raw measurements and -metadata are usually unavailable: the reported measurements and errors -must be accepted as they are reported. - -The AP Database stores the collections of detections which were -derived from specific images from any of the analysis stages. It -provides a mechanism to determine the image from which a specific -detection was derived, and in conjunction with the Image Server locate -the corresponding data file. The AP Database also makes it possible -to extract all detections derived from a specific image and to -determine quantities such as the pixel coordinates of the detection on -the image. - -The AP Database also has the capability to associate multiple -detections of a specific object. Several major classes of objects -will be present, each of which must be handled correctly. - -First, the most distant stars, compact galaxies, and QSOs will have -nearly fixed locations relative to other distant stars, with only -small deviations for individual measurements. The association between -multiple detections of such objects is made on the basis of their -coincident positions. The AP Database determines the average position -of the object and the deviations of the individual detections from -that average on the basis of the ensemble of individual detection. - -Second, solar system objects do not have a fixed location. Detections -of such objects are linked by their orbits, and depend on both the -position and the time of the image. The AP Database does not attempt -to make this link; this is the role of the MOPS system. However, it -has the ability to accept identifications made externally with -specified detections and to return the identifier of the moving object -associated with the specific detections. These associations also -include descriptive information such as the offset of the detection -from the predicted location of the detection based on the orbit. This -functionality is required to allow the AP Database to ignore known -moving object detections from other types of queries. - -Third, objects in the general vicinity of the solar system fall in -between these first two classes of objects. Their proper motion and -parallax response is significant enough ($>0.2$ arcsec in 1 year) that -they are not well-described by an average location and a collection of -offsets. These objects are described by a distance and a proper -motion vector. The AP Database provides the association between the -specific detections and an average object which includes finite -parallax and proper motion. - -Fourth, many detections, especially in their initial states, will not -be associated with a specific astronomical object of any of the above -classes and are treated as orphans. Most of these will be spurious -(not representing real objects), some will be from solar system -objects for which orbits are not yet determined, some will be from -faint stars near the detection limits, and some will be from -short-term transients which have only been detected once. The AP -Database maintains these detections until they have been associated -with one of the objects above. The AP Database provides mechanisms by -which individual detections may be migrated back and forth between the -orphan state and association with an astronomical object. - -For every object, and all orphaned detections, the AP Database also -provides the capability to determine the images containing the -location of the object but for which no detection was made. The -minimum set of information which must be carried for these -non-detections is the image and the associated object or orphan. - -The AP Database also stores the relationships between various -photometric systems and the evolution of that relationship. It -provides mechanisms to convert between the measured instrumental -magnitude of a detection with a specific filter, detector, and -telescope, and at a particular time and the implied magnitude in the -average Pan-STARRS photometry system, given a determined set of -calibrations. It also provides the capability to convert magnitudes -in one system to the magnitudes in another system; an example of such -a conversion is between the average Pan-STARRS filter systems and the -various reference systems appropriate for those filters. - -\begin{figure} -\begin{center} -\resizebox{4.5in}{!}{\includegraphics{pics/APDB}} -\caption{AP DB components} -\label{fig:APDBComponents} -\end{center} -\end{figure} - -The AP Database provides interfaces to extract lists of objects and -detections based on various query parameters. It provides the -capability to extract all detections associated with a specific -object, all non-detections of that object, all non-detections of an -orphan, and summary statistics from these collections. It will also -return all objects or detections within specified spatial regions -including regions bounded by great circles (RA,DEC; GLAT,GLON; -ELAT,ELON) and regions described by a location and a search radius. -It will also return the image parameters associated with a specific -detection including image coordinates of the detection, exposure time, -time and date of the detection, etc. - -As shown in Figure~\ref{fig:APDBComponents}, the IPP AP Database -consists of the following components: - -\begin{itemize} -\item AP Database database tables -\item AP Database database engine -\item AP Database servers -\item AP Database client APIs -\end{itemize} - -\subsubsection{AP Database Tables} - -Table~\ref{tab:APDBTables} lists the tables used by the AP Database. The -contents of these tables are outlined in -Appendix~\ref{sec:APDBTableContents}. Below, the use of these tables by -the AP Database software is discussed below. Several of the tables -are not just simple tables in the database but are instead table -groups divided into many subtables, each of which represents a portion -of the sky (a {\tt region}). These subtables may also be distributed -across different computers to distribute the processing load. - -\paragraph{Images Table Group} - -The {\tt Images} table group lists all of the images which provided -the data in the AP Database. These tables are subdivided by region on -the sky. In general, the images listed in this table correspond to -the Chips. This group of tables includes sufficient astrometric -parameters to represent the coordinates of the detections to a -sufficient accuracy. Parallel to the Images table is the Mosaic -table. This table is very similar to the Images table, but defines -the Mosaic which corresponds to a group of Images. The parameters -include the astrometric information needed to define the camera -distortion. - -\paragraph{Image Overlaps Table Group} - -The specific subtable of {\tt Images} which contains a given image is -the one which contains the center pixel of that image. An additional -table group, {\tt Image Overlaps} (with the same subtable organization -as the {\tt Images} subtables), lists images which overlap that -specific subtable. Thus, given a particular coordinate, in order to -find that images which overlap that coordinate, it is necessary to -search the images in the {\tt Images} subtable which includes that -coordinate, and all images in the {\tt ImageOverlaps} subtable for -that coordinate. - -\begin{table}[hb] -\begin{center} -\caption{AP Database Tables\label{tab:APDBTables}} -\begin{tabular}{ll} -\hline -\hline -{\bf Table Name} & {\bf Description} \\ -\hline -Images & The images that have objects in the DB. \\ -Image Overlaps & Image regions which are touched by specific images. \\ -Objects & The objects --- average properties of multiple detections of the same object. \\ -Average Magnitudes & Average photometry in multiple filters \\ -Solar System Objects & Identification of solar system objects \\ -Matched Detections & Detections of sources in an image identified with an Object. \\ -Orphaned Detections & Detections of sources in an image not identified with an Object. \\ -Non-detections & Non-detections of objects in an image. \\ -Regions & spatial distribution of tables \\ -Filters & Filters understood by the system. \\ -Photcodes & Transformations between different photometric systems \\ -Zero Points & History of Zero-point \& Airmass terms \\ -Distortion Models & History of Optical Distortion terms \\ -Database Hosts & computers used to store the tables \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\paragraph{Objects Table Group} - -The {\tt Objects} table group (also divided by region) stores the -average parameters for each astronomical object. Certain details of -this table have not yet been specified. In particular, objects with -significant parallax and/or proper motion may potentially be stored in -a distinct table. Solar system object identifications, to the extent -average properties are maintained in the AP Database, will certainly -be stored in a separate table. - -\paragraph{Average Magnitudes Table Group} - -A related table, also divided into the same regions, is the {\tt -Average Magnitudes} table. In this table, there are multiple rows per -object, one for each of the primary filters of interest for which -photometric averaging is performed. This organization makes the -number of primary (averaged) filters a configurable value. - -\paragraph{Matched Detections Table Group} - -The {\tt Matched Detections} table stores all of the measurements of -astronomical objects on specific images. This table includes all -detections associated with the average {\tt Objects}. As discussed -below, bright objects (above a configuration-specified signal-to-noise -level) are defined object even if only one detection has been found at -that position. Faint orphaned objects are not added to this list or -the list of objects. The different types of detections (P2, -P4$\Delta$, P4$\Sigma$) are distinguished by their photometry codes. -(This is only valid if the AP Database does not store different -quantities for these types of detections.) - -\paragraph{Orphaned Detections Table Group} - -The {\tt Orphaned Detections} table stores the detections which have -not been correlated with an existing object. This table is only -populated for objects below a configuration-specified signal-to-noise -limit (e.g., 5$\sigma$). Bright orphaned detections are assigned an -object and added to the {\tt Matched Detections} table. - -\paragraph{Non-detections Table Group} - -The {\tt Non-detections} table stores information about detection -failures for each object. If an image is added to the database which -overlaps an object but the object is not detected, an entry is made in -this table. In practice, this table may store only the most recent -non-detection and the total number, or a similar reduced set of -non-detection statistics. - -\paragraph{Regions Table} - -The {\tt Regions} table is used to subdivide the tables of images, -objects, and detections, etc, as discussed above. The AP Database -divides the sky into a hierarchy of regions (portions of the sky) each -of which is in turn subdivided into smaller portions. Since nearly -all interactions with the AP Database performed by the IPP are limited -in spatial coverage, subdividing the tables allows a specific -interaction to search only a small subset of the data. The table of -images is the smallest of the three; the table of detections is likely -to be the largest. As a result, the {\tt Images} table group will be -subdivided at a shallow hierarchical level, while the {\tt Objects} -and {\tt Detections} are subdivided on deeper (more finely sampled) -levels. The {\tt Regions} table defines the boundaries of the sky -regions and specifies if the region corresponds to an {\tt Images} -table, an {\tt Objects} table, and/or a {\tt Detections} table. It -also specifies which regions in the next level of the hierarchy are -contained by the region, and which parent region it belongs to. In -addition to improving the spatial access to the image, object, and -detection data, the {\tt Regions} table allows for multiple computers -to serve the database tables. The region file specifies the machine -which stores the specific table. Figure~\ref{fig:APDBRegions} -illustrates schematically the subdivision of the sky and the -association between different levels of the hierarchy with different -subtables. - -\begin{figure} -\begin{center} -\resizebox{6in}{!}{\includegraphics{pics/APDBRegions}} -\caption{AP DB Regions and Image / Object tables} -\label{fig:APDBRegions} -\end{center} -\end{figure} - -\paragraph{Other Reference Tables} - -The {\tt Filters} table identifies all of the physical filters -(specific pieces of glass) known to the system. A related table, {\tt -Photcodes}, defines relationships between photometry systems. A -photometry system may consist of a detector, telescope, and specific -filter, or it may be a derived photometry system. The {\tt Database -Machines} table identifies all of the computers available to the AP -Database. - -\subsubsection{AP Database servers} - -The AP Database functions on a group of computers, with portions of -the tables stored on separate machines, as described above. The -association between a machine and the corresponding table or part of -the sky is defined by the Region table. Each machine has a -corresponding AP Database server which runs on that machine to -interact with the tables available on that machine. Two possible -interaction models are considered. - -{\bf Option A:} A client chooses one of the machines and sends its -query or data to that machine. The server then uses the region table -to determine which machines contain the relevant portion of the sky. -Data to be added to the database is divided into corresponding region -chunks and sent to the appropriate servers. Queries are redirected to -the appropriate server(s). The original server may collect the -results and return them to the original client. - -{\bf Option B:} The client downloads the region table and performs the -division of the data into appropriate subsets. The subsets are then -sent to the corresponding servers by the client. - -The differences between these models is small. The first option may -make the code more testable, placing all of the logic in the servers -and making each server symmetric. The smaller tables (ie, Region, -Filters, etc) could either be downloaded from a single server or -replicated to all AP DB servers. For these reasons, Option A will be -used for the PS-1 IPP.. - -\subsubsection{AP Database engine} - -The backend database engine for the AP Database stores the tables and -provides them to the servers on demand. The AP Database will use a -\code{mysql} database engine for this function. - -\subsubsection{AP DB Client operations} - -The AP Database client interactions consist of a collection of basic -queries of the database, along with more complex operations to perform -particular tasks. The complex operations are listed below. - -\paragraph{Insert Image \& Detection Set (addstar)} - -One of the most basic operations needed by the AP Database is to -insert a collection of detections derived from a specific image, and -add the definition of that image to the database. This operation is -critical in terms of the processing throughput. After the detections -have been assigned to the appropriate regions, they are matched -against all objects in the {\tt Objects} table. Matches are performed -only on the basis of positional coincidence, using a matching radius -which may depend on the image astrometry errors, or may be a fixed -distance. Any matched detections are added to the {\tt Matched -Detections} table. Any unmatched detections brighter than the Faint -Detection cut-off are specified as a new {\tt Object} and also added -to the {\tt Matched Detections} table. Any faint unmatched detections -are added to the {\tt Orphaned Detections} table. This division is -important because it allows the automatic association of new -detections with existing bright objects while limiting the I/O volume -required to make the detections. In general, there will be many fewer -{\tt Objects} than {\tt Detections}, and there will be fewer bright -orphans than faint orphans. - -\paragraph{Insert Reference Objects (addrefs)} - -This operation is very similar to the previous one. A collection of -reference objects are added to the database as a collection of -detections. The reference photometry should in general be given its -own photometry code. The reference data is different from the image -detection set because the associated image information is not -included. Thus, no corresponding images are added to the database. - -\paragraph{Determine Relative Photometry in region (relphot)} - -This operation uses the overlaps of images and multiple observations -of the same objects to determine the relative photometry zero-points -for a collection of images. This is a task that wil be run much more -infrequently than the object insertion tasks. - -\paragraph{Determine Consistent Photometry Zero Points (uniphot)} - -This operation uses the time history of relative photometry zero -points for images and the spatial overlap information to determine a -best set of image zero points which have a specific time scale for the -atmospheric stability. - -\paragraph{Determine Distortion and Static Astrometry Model (mosastro)} - -This operation uses the reference and image detections to determine an -optical distortion model for the camera and static astrometry model -components. The astrometry model includes: (1) field distortion -introduced by the telescope optics, which is a smoothly-varying -function of the field position relative to the center of the telescope -boresite coordinates. (2) focal plane geometry, which includes the -chip positions and rotations in the focal relative to the boresite, -along with chip-dependent plate-scale modifications needed to -represent tilts or warps of the individual detectors relative to the -ideal flat focal plane. . - -\begin{table} -\begin{center} -\caption{AP Detection Classes \& Object Parameters\label{tab:APdetections}} -\begin{tabular}{lrrrr} -\hline -\hline -Object Parameter & P2 & P4S & P4D & SS \\ -\hline -PSF x,y, covar, $\alpha,\delta$ & + & + & + & + \\ -PSF mag, $\sigma_{\rm mag}$ & + & + & + & + \\ -star/gal sep & + & + & + & + \\ -$\sigma_x$, $\sigma_y$, $\theta$ & + & + & + & + \\ -local sky data & + & + & + & + \\ -Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ -S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ -W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ -exp. spaced aps., Poisson noise, variance & - & - & - & + \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Throughput} - -The AP Database design partly driven by the need to make the -detection-object associations quickly and to processes the incoming -detections at a sufficiently high rate to meet the throughput -requirements. For each upload of the object detections from a -complete FPA, the AP Database must match roughly $1.4 \times 10^{6}$ -detections from an FPA with roughly $6.4 \times 10^{6}$ objects, -including orphaned bright detections. This corresponds to roughly 640 -MB, if each object uses 100 bytes for its descriptive informations -(more than is currently specified in the Object table). With a -throughput of 100 MB/s for reads from a RAID, the AP Database can -perform the data read in a fraction of a second if the data is -distributed across 10 computers. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Controller} -\label{sec:Controller} - -\subsubsection{Corresponding Requirements} - -The Controller must meet the requirements specified in Section 3.4.4 -of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The design must meet -requirements 3.4.4.1 - 3.4.4.7. In particular, the Controller / Node -Agent architecture is chosen to control the I/O flow between the -Controller and the individual processes so that blocking on the I/O -from many remote processes does not saturate the Controller -processing. - -\subsubsection{Overview} - -\begin{figure} -\begin{center} -\resizebox{4.5in}{!}{\includegraphics{pics/Controller}} -\caption{Schematic illustration of the Controller components} -\label{fig:Controller} -\end{center} -\end{figure} - -The IPP uses a group of computers to store and process images and to -manipulate collections of detections. These computers perform any of -a large number of analysis stages or other processing tasks without -significant interprocess communication. It is necessary to have a -mechanism which initiates computing tasks on the different computers, -which monitors the tasks as they are executed, which handles the -output and the errors from these tasks, and which reacts to the -failure of any of the computing nodes. The system responsible for the -tasks in the IPP is the IPP Controller. - -The IPP Controller interacts with the collection of computers under -its management and with other subsystems in the IPP. The IPP -Controller receives a variety of inputs from other subsystems, -described below, and initiates actions such as adding a new process to -the queue of pending tasks. The IPP Controller also provides -information to other subsystems on demand about its processing history -and current state. Each physical computer may have multiple -processors; since the IPP Controller is managing processing tasks, it -treats each processor independently. It is up to the system -configuration if each computer needs to reserve one of its CPUs to -manage background tasks or if the IPP Controller should attempt to -send one task per CPU and let the operating system handle the I/O -load. The relationship between the different components of the -Controller is illustrated in Figure~\ref{fig:Controller} and discussed -below. - -\subsubsection{Nodes} - -The Controller maintains a table of available processing computers -(`Nodes') and tracks the status of these Nodes. Nodes managed by the -IPP Controller are allowed to be in one of several states, and the IPP -Controller must interact with it in an appropriate way for each of -those states. A Node may be {\tt alive}, {\tt dead} or {\tt off}. -If the Node is {\tt alive}, it responds to commands from the IPP -Controller and may be used for tasks subject to other constraints. If -it is {\tt dead}, the Node is not responsive and must not be used -for executing tasks. The IPP Controller must identify Nodes which -have died (not responding) and occasionally test them to see if they -are {\tt alive} again. Nodes which are {\tt off} are not -available for tasks and must not be tested. Nodes may be set to -the {\tt off} or {\tt dead} states by external subsystems; it is the -responsibility of the IPP Controller to return a Node to the {\tt -alive} state if possible. - -The IPP Controller must honor requests (normally from the users) to -change the mode of any computing node on demand between {\tt off} and -{\tt dead}. This would normally be done after a Node has been -rebooted and is released to the IPP Controller for its use. It must -also be able to change the list of allowed tasks as requested by -external commands. - -Two example scenarios illustrate the transition between these states, -and the basic concept of operations for the IPP Controller. First, -imagine a computer crashes. At this point the IPP Controller should -detect that the Node is no longer responsive and mark it as {\tt -dead}. It should occasionally try to re-establish communication with -the Node, potentially with longer and longer delays between attempts. -A human could be notified if the Node seems to remain {\tt dead} for a -very long time. In another scenario, a person needs to work on a -Node. They notify the IPP Controller that the machine is {\tt off}, -perhaps with a prior notification that the machine should be prepared -to go off. When work on the machine is complete, it should be placed -in the {\tt dead} state. Only when the person is done working and -testing the machine, and tells the IPP Controller that the machine is -now {\tt dead} can the IPP Controller attempt to re-start -communications and re-new processing operations on that Node. - -\subsubsection{Node Agents} - -When the Controller starts, it attempts to launch a Node Agent on each -of the available processing Nodes. Nodes which are not responsive are -marked as {\tt dead} so they may be re-tried. A Node Agent runs on -each of the individual nodes to execute the tasks as directed by the -Controller. The Node Agents communicate with the Controller via a -socket connection. - -A Node Agent (which is only running on a Node in the {\tt alive} -state) may be in one of four modes: {\tt idle}, {\tt busy}, {\tt -done}, {\tt crash}. A Node Agent which is {\tt busy} currently has a -task assigned to it which is executing. The IPP Controller may only -assign one task to a Node at a time. A Node Agent which is in the -{\tt idle} state may have a task assigned to it. When the Node Agent -detects that a tasks has finished, it changes to either the {\tt done} -or {\tt crash} states depending on the outcome of the process -execution. The IPP Controller must also respect a list of task -restrictions which may require specific tasks to run on specific CPUs -or exclude specific tasks from specific CPUs. - -A task being executed by the Node is run in the UNIX user space as a -forked process. The Node Agent must monitor the standard error and -standard output of the executing task and save them in separate -buffers. If the process exits or dies, the Node Agent must detect -this result and change state appropriately. The Node Agent must -respond to various commands from the Controller, as follows: - -\paragraph{Report status} - -The Node Agent returns its state ({\tt idle}, {\tt busy}, {\tt done}, -{\tt crash}) and the exit status of the current processing task, if -available. The reported exit state, if the process has completed -without crashing, is the UNIX exit state reported by the task: 0--256 -with 0 indicating a successful completion. - -\paragraph{Report stdout} - -Send and flush the current stdout buffer. The Node Agent will return -the complete contents of the stdout buffer via a buffered write and -flush the buffer when it is finished. The Node Agent will not accept -more data on the stdout buffer from the current processing task until -the send is complete and the buffer is flushed. The daemon must -accept all of the buffer output. - -\paragraph{Report stderr} - -Identical to `report stdout', but for stderr. - -\paragraph{Kill task } - -The Node Agent should send a kill signal (\code{KILL} or \code{TERM}) -to the current processing task. When the processing task has exited, -the Node Agent should set its state to {\tt crash}. - -\paragraph{Clear task} - -The Node Agent should set its state {\tt idle}. If a processing stage -is currently running, it should be killed (\code{KILL} or \code{TERM}) -before the task is cleared. - -\paragraph{Start processing stage} - -The Node Agent forks a specified command. The command should be a -standard UNIX command without command line redirection or -backgrounding. The task is run with the same user ID as the Node -Agent, which is also the same user ID as the Controller. - -\subsubsection{Tasks} - -The IPP Controller accepts tasks from other IPP subsystems. The task -requests include the specific command to be executed and are in the -form of a UNIX command which could be performed on any of the -computing nodes. Any input or output data in the commands must be a -valid resource regardless of the node on which the task is executed. -Input and output data resources must be unique where necessary to -avoid conflicts. It is the responsibility of the task to wait for -network lags (ie, NFS delays). The IPP Controller gives each task a -unique identifier, which is returned to the requesting entity. The -requestor may then use that ID to obtain status information on that -task or to send control signals to the specific task. - -Task requests may specify a desired node for the task execution. The -IPP Controller attempts to honor the request if the node is {\tt -alive}, but will execute it on another node if the requested one is -{\tt dead} or {\tt off}. Even if a node is {\tt alive}, the IPP -Controller will choose another node if the specified task is not -allowed on the requested node. In all other cases, the IPP Controller -waits until the currently executing processes, and processes with -higher priority, are completed before executing the specified task on -the requested node. - -Task requests may specify an urgency level. The IPP Controller -determines the priority of the task on the basis of both the urgency -and the age of the request. An executing task must be completed on a -CPU before any new task is started on that CPU, regardless of -priority. The urgency levels range from 0 to 2. Tasks with an -urgency of 1 are scheduled whenever they reach the top of the stack. -Tasks with an urgency of 2 are sent immediately to the top of the -stack. Tasks assigned a priority of 0 are maintained in the queue and -never executed. - -It may be useful for the Controller to distinguish between tasks -dominated by I/O and tasks dominated by data processing. It is -possible that one of each of these types of tasks may be sent to the -same node without significantly impacting the system performance. -Alternatively, it may be necessary to limit a single machine with 2 -CPUs to only one of each of these types of tasks (i.e., one processor -will be working on I/O while the other is working on processing). -Such details will be studied by the IfA IPP Team. - -The IPP Controller monitors the output streams from the executing -tasks and the exit status of the tasks. Each task is associated with -a log file, to which all output is written. The status, including the -exit status, of each task is maintained by the IPP Controller so that -other subsystems may determine if specific tasks have started or -completed. - -\subsubsection{Controller Interfaces} - -The IPP Controller must accept commands from other IPP subsystems. -These commands include those which govern the processing of specified -tasks, those which govern the behavior of specific computing nodes, -and those which request information from the IPP Controller. The IPP -Controller must be able to halt the execution of a specified task, -delete an unexecuted task from the task list, change the priority of -tasks, and change the requested nodes for tasks. The IPP Controller -must also be able to stop the current execution of a task and push it -to the end of the queue and also change its priority. - -The IPP Controller must respond to informational requests regarding the -collection of machines and their states as well as the collection of -tasks and their states. The IPP Controller must monitor the execution -times of the different tasks and provide summary statistics. Finally, -the IPP Controller must respond to three top-level commands: {\tt finish}, -{\tt stop} and {\tt abort}. When {\tt finish} is requested, no more -new tasks are accepted on the stack of task, and when all tasks in the -stack have completed, the IPP Controller must exit. When {\tt stop} is -requested, the currently executing tasks must be completed at which -point the IPP Controller must exit, but tasks remaining in the stack which -have not been started are flushed. When {\tt abort} is issued, the -IPP Controller immediately kills all executing tasks and exits. - -The IPP Controller and the IPP Image Server have related needs for -information from the combined storage-and-processing nodes regarding -which nodes are available. It is not yet clear if this information is -best stored in a single location (either IPP Controller or IPP Image -Server), which provides the information to other systems on demand, or -if both systems should maintain the information. Also, it may be -necessary to distinguish nodes which are available for processing from -those that are available to serve data as part of the IPP Image -Server. - -The Controller maintains three tables of processing jobs: pending -stages, active stages, and completed stages. The pending stages are -those which have not yet been performed. The active stages are those -currently being performed on one of the remote nodes. The completed -stages are those which have finished, either successfully or with an -error state. The Controller daemon monitors the collection of remote -clients and sends them new pending stages when they become free. - -The IPP Controller provides a mechanism for users (either other -programs or humans) to interact with it. The user interface provides -commands to check the current processing job queues, the tables of -successful and failed jobs, to stop or delete jobs, etc. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Scheduler} - -\subsubsection{Corresponding Requirements} - -The Scheduler must meet the requirements specified in Section 3.4.5 of -the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The design must meet -requirements 3.4.5.1 - 3.4.5.7. In particular, the Task / Test -division is chosen to prevent the Scheduler from blocking while an -analysis process is performed. Scheduling requirements will be met by -defining appropriate Test periods for the different Tasks. - -\subsubsection{Overview} - -The IPP is responsible for a variety of analysis jobs: processing of -the science images through several stages; routine assessment of the -detrend (instrumental calibration) images used in processing the -science images; construction of replacement detrend images when -needed; generation of astrometric and photometric reference catalogs -based on the collected dataset; and the performance of test analysis -programs. At any point, decisions need to be made about which of -these tasks should be performed, based on an analysis of the contents -of the metadata database, the requirements of the people monitoring -the IPP, and the near-term observing plans. The IPP Scheduler is the -mechanism that assesses these various inputs to guide the decisions -and initiate the actions. - -The IPP Scheduler acts as an interface between several components of -the IPP and also between the IPP and external agents such as OTIS and -the users who must monitor the behavior of the IPP. The IPP Scheduler -may be viewed as the central brain of the IPP. -Figure~\ref{fig:Scheduler} illustrates the design of the IPP -Scheduler. - -\subsubsection{Scheduler Tasks and Tests} - -The IPP Scheduler performs two types of actions. 'Tasks' are -long-running programs which are executed by the Controller. These are -not only background tasks, but are distributed computing tasks. -Examples of these include the science analysis tasks (eg, Phase 1, 2, -3, 4), the Calibration construction tasks, and data copy tasks (such -as copying images and metadata from the summit system). 'Tests' are -short-running programs which are used to decide which tasks should be -run. Tests should be designed to return immediately ($< 100 ms$) and -are not run in the background; the Scheduler will block until the test -is complete. The IPP Scheduler daemon, which runs continuously, -performs tests (eg, queries of the IPP Metadata Database, queries of -OTIS, checks of the IPP hardware status, etc). Based on these tests, -the daemon defines appropriate tasks and sends them to the Controller. -When tasks are completed, their results may be used by the Scheduler -to update the external systems (update the Metadata Database), or the -tasks themselves may send their results directly to the Metadata -Database or other subsystems. Based on the successful completion (or -not!) of the tasks, and the new state of entries in the Metadata -Database, the Scheduler can define new tasks. - -\begin{figure} -\begin{center} -\resizebox{6in}{!}{\includegraphics{pics/Scheduler}} -\caption{ \label{fig:Scheduler} IPP Scheduler} -\end{center} -\end{figure} - -The IPP Scheduler sends tasks to the IPP Controller for execution. -While the IPP Scheduler chooses the tasks to be performed, it is the -IPP Controller's responsibility to manage the specific tasks executing -on a given processing node. This division of responsibilities allows -the different functionalities of the IPP Scheduler and the IPP -Controller to be isolated and encapsulated. With this separation, the -IPP Controller does not information about the details of the tasks it -executes, while the IPP Scheduler does not need to monitor the -computer hardware. - -Communication between the IPP Scheduler and the IPP Controller is -bi-directional; the IPP Scheduler sends tasks to the IPP Controller, -while the IPP Controller informs the IPP Scheduler of the outcome of -those tasks. For the PS-1 IPP, the IPP Scheduler and the IPP -Controller are distinct, interacting software components. The -interface mechanisms are described in Section~\ref{sec:interfaces}. - -\subsubsection{Task Rules} - -The IPP Scheduler takes as input a collection of rules which define -the dependency of tasks on certain tests. The IPP Scheduler must -choose between several types of analysis tasks based on those rules -and on results of the tests. The timescale on which different tasks -(and their related tests) are executed may vary from 10s of seconds to -hours, days, or even as long as a week. The list of tasks which the -IPP Scheduler must decide between, and the relevant timescale, follow: -\begin{itemize} -\item moving data from the Summit pixel server ($\sim 30$ second timescales) -\item running the science analysis stages ($\sim 30$ second timescales) -\item testing the validity of the current detrend images ($\sim$ - nightly) -\item constructing new detrend images ($\sim$ weekly) -\end{itemize} -The scheduler may be viewed as a complex state machine. The goal is -to design the scheduler so that rules may be specified independently -from the engine which parses the rules to determine which specific jobs -to send to the controller. - -\subsubsection{User Interface} - -The IPP Scheduler shall possess a user interface which allows a human -operator, or other processes, to monitor the current state of the -Scheduler. Users have the option to specify that a particular task or -set of tasks is of higher or lower urgency (as defined in -Section~\ref{sec:Controller}) than the norm, or to schedule a -particular tasks on a different timescale from the basic rule. - -The IPP Scheduler defines the operating state of the IPP and shares -the same set of states: -\begin{itemize} -\item active state -\item interactive state -\item paused state -\end{itemize} -When the IPP Scheduler is in the {\em active state}, it performs the -most appropriate of all possible tasks at a particular time. When the -IPP Scheduler is in the {\em interactive state}, it performs only a -specific requested action regardless of the outcome of the decision -trees. In addition, in the interactive state, the IPP Scheduler must -only perform the requested actions and not attempt to perform the -other normally-required actions. The only exception to this exclusion -is that, in the interactive state, data is still copied from the -summit system. An additional IPP state is the {\em paused state}, -intended for tests or maintenance, in which case the IPP Scheduler -does not perform even the data copy tasks. Every task is performed on -demand by the user. A user command sets the IPP Scheduler in one of -these three states, {\em active}, {\em interactive}, and {\em paused}. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{System Design : Science Analysis Tasks and Stages} -\label{sec:AnalysisStages} - -This section describes the design of the science analysis stages which -perform the fundamental image analysis steps of the IPP. The IPP -science image processing stages perform analyses on the night-sky -science images to extract the science data from these images. These -consist of: -\begin{itemize} -\item Phase 1, the image processing preparation stage, -\item Phase 2, the image reduction stage -\item Phase 3, the exposure analysis stage -\item Phase 4, the image combination stage. -\end{itemize} -These analysis tasks must process the images in a timely manner so -that the incoming data stream will not overload the IPP Image Server. -The decision to execute a specific pipeline for a specific dataset is -made by the Scheduler, which sends the information to the Controller. -The Controller executes the pipeline for the data on an appropriate -machine and monitors the success or failure of the processing stage. - -The analysis stages are written as UNIX commands, which may be -executed by the IPP Controller, or may be executed individually by -hand. This option makes testing of the complete analysis system much -easier because the individual analysis stages may be tested -independently of each other and the IPP infrastructure. - -As part of this design model, the analysis stages have several methods -for accepting and returning the input and output data and for defining -optional choices in the analysis. All of the analysis stages load an -analysis recipe, which defines the details of that analysis. The -recipe includes the location of the data sources (from the metadata, -from the image headers, from other external files, or supplied -directly), which steps to employ, and how to assign optional -parameters. For example, in the discussion of the Phase 2 analysis -below, the recipe file may specify {\em if} a bias subtraction should -be applied, {\em where} to find the overscan region and {\em which} -bias image, {\em if any}, to apply. - -The recipe is loaded as part of the runtime configuration information -loaded when the analysis script starts. Four levels of runtime -configuration information are defined. The {\tt site} configuration -defines values specific to the particular installation of the -software. For example, the name of the machine which hosts the -Metadata Database or a default path for data files could be part of -the {\tt site} configuration. Multiple installations or versions of -the IPP software would need to have separate {\tt site} configuration -entries. For example, a version of the IPP installed at the IfA would -use a different computer for the Image Server from the live IPP -installation running on the Pan-STARRS cluster. The {\tt base} -configuration defines general data sources which may be needed by any -portion of the IPP. The list of known telescopes or filters might be -an example. The {\tt camera} configuration consists of information -which defines the parameters relevant to the cameras known by the IPP. -For example, the default layout of the detectors or the names of -specific header keyword values would be defined for each camera in a -camera-specific configuration collection. Finally, each analysis -script loads its own recipe. The location of this configuration -information may be a collection of configuration files available on -disk or some subset of the information may be stored in the Metadata -Database. The source of these configuration entries can be overridden -when the script is executed, and individual configuration values may -also be specified on the command line. Examples of the recipe and -other runtime configuration options are given in -Appendix~\ref{sec:RuntimeConfig}. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Phase 1: image processing preparation} - -The Phase 1 analysis stage is performed on each science exposure (each -complete FPA image) to calculate basic astrometric data needed by the -later stages. Phase 1 uses the static (pre-determined) telescope -distortion model and a table of nominal OTA positions and rotations, -combined with the guide star pixel and celestial coordinates, to -determine the correct telescope bore-sight, field rotation and -magnification. The guide star coordinates are loaded from the -Metadata database. These calculations are performed by comparing the -observed guide star detector coordinates with the known astrometric -positions of these same stars as reported by an external astrometric -reference. The accuracy of the resulting astrometric solution is -expected to be $\sim 1$ arcsec across the field, sufficient in later -stages to match the vast majority of astrometric reference stars with -their detections with minimal effort. - -In some circumstances, science images may have no guide stars. This -may occur in the Pan-STARRS system if the detectors are not run in OTA -mode, for example for short snapshot images. This may also be the -case if the IPP is being run on non-Pan-STARRS data. In such a -circumstance, the Phase 1 stage uses the provided boresight -coordinates, exposure time, and camera zero-point to predict the pixel -coordinates of known bright stars expected to be found on the -detectors. It then extracts a large box ($\sim$ 30 $\times$ -30\arcsec) around these locations and performs extremely basic object -detection to determine the detector coordinates of those bright stars -which are not saturated but which are significantly above the -background level. By targeting known locations in the image files, -only a small amount of data will have to be read. - -If the image has invalid coordinates or no detectable bright stars, -Phase 1 fails and reports a descriptive error. - -Given the above astrometric solution, the Phase 1 analysis stage -constructs a table of the overlaps between the science image to be -processed and the static sky images that must be constructed. This -table will be used to guide the processing of the static sky in Phase -4. The overlaps should be generously calculated so that small errors -in astrometry at Phase 1 will not cause any valid static sky / science -image pairs to be missed because of the astrometric error at this -phase. It is acceptable for a small number of invalid overlaps to be -identified as these will be excluded in Phase 4. Static Sky cells -which do not have sufficient science image overlap ($< 5\%$) need not -be processed because the few new measured pixels do not add -significantly to the Static Sky. - -\subsubsection{Examples} - -Examples of Phase 1 as called from the command line, with different -types of images: - -\begin{verbatim} -Phase1 -file filename.fits [FPA is single fits file] -Phase1 -list filename.list [FPA is collection of files] -Phase1 -imdb ID [FPA is single file in image server] -Phase1 -FPA ID [FPA identifier from metadata db] -\end{verbatim} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Phase 2 : image reduction} - -\subsubsection{Overview} - -Phase 2 processing within the Pan-STARRS image processing pipeline is -the detrend stage, where the images from the detector are processed to -remove instrumental signatures. This analysis is performed on -individual chips, which can be identified as the data entity which has -a single, continuous astrometric solution. - -Phase 2 consists of the following operations, some of which as noted -may be skipped by the recipe: -\begin{itemize} -\item Load science image -\item Identify appropriate detrend images -\item Load detrend images -\item Form OT kernel -\item Convolve detrend images with the OT kernel -\item Bias/dark/overscan subtraction -\item Mask bad pixels -\item Trim overscan -\item Non-linearity correction -\item Flat-field -\item Mask diffraction spikes and optical ghosts -\item Subtract sky -\item Find and photometer objects in the image -\item Identify CRs by morphology -\item Determine PSF model -\item Improve astrometry -\item Extract Bright object postage stamps -\end{itemize} -The steps are explained in detail below. - -\subsubsection{Load Images} - -The Phase 2 analysis must load the science image to be analyzed into -memory, as well as the corresponding metadata (either from the image -header and/or from the IPP Metadata Database). It must use the -metadata for the image, along with information from the processing -recipe, to determine the appropriate detrend images to be used for -this analysis. The Metadata Database stores the information necessary -to associate a specific science image with one of the registered -master detrend images for each type. These images are also loaded by -the Phase 2 analysis (note that the design of Phase 2 may perform the -actual loading of pixels in blocks or groups to minimize the memory -impact). - -\subsubsection{Form OT Kernel \& Convolve with Detrend Images} - -Science images which have been obtained with Orthogonal-Transfer -Guiding have had their pixel response smoothed by the image correction -motion. For these images, some of the detrend images need to be -convolved by the same OT kernel, so that they accurately represent the -effective pixel response. The detrend images which must be convolved -include: the flat-field image, the high-spatial-frequency fringe -images, and dark images, if they are used. The appropriate kernel for -each cell of an OTA must be determined from the guide star history, -extracted from the IPP Metadata Database or from the image header. If -the OT kernel is not available, but the image metadata notes that it -should be, the convolution is skipped, with a warning. - -The convolution method depends on the size and structure of the OT -kernel. If the kernel is small ($< 5x5$ pixels), direct convolution -may be employed. If the kernel is large, but may be decomposed using -Gaussians, then it may be convolved using a decomposition method. - -The module convolves each of the dark frame, flat-field, and the -fringe frame(s) by the OT convolution kernel. Specific flags in the -static bad pixel mask are also grown by the outline of the OT -convolution kernel (see Section~\ref{sec:masks}). - -\subsubsection{Bias Correction / Overscan Subtraction} - -The image bias must be subtracted. Since different detectors behave in -different ways, several options for modeling the bias are available. -The bias is measured from the image overscan region. The bias -subtraction method must be capable of subtracting a single constant -from the complete image, or to subtract a 1-D bias which varies as a -function along the overscan. The function used to represent the -overscan region may be a spline or a Chebychev polynomial derived from -the data values along the overscan. The values used to determine both -the single constant or the inputs to the spline and polynomial fits -are derived from groups of pixels on the basis of one of several -statistics, including the sample and robust mean, median, and modes. -In the case of a single constant, all of the overscan pixel values are -used in the calculation of this statistic. In the case of the 1-D -functional representation, the input values to the fit must represent -the coordinate along the overscan, with the statistic derived from the -pixels in the perpendicular direction at each location. -Sigma-clipping on the input data values must be an option. - -\subparagraph{Flag bad and saturated pixels} -\label{sec:masks} - -A static bad pixel mask is used to identify pixels which are known to -be bad in the camera. This mask is then processed with the science -image. Bad pixels which are charge traps are grown by the extent of -the OT convolution kernel. Bad pixels above a charge trap (i.e.\ bad -columns) must not be grown, since they were not affected by pixel -shifting, but only became bad at read-out. - -Pixels which are saturated in the A/D converter, or with a signal -level at which the response is excessively non-linear, must also be -masked, and this area must be grown by an additional pixel to mask -excess charge spillover. - -The bad pixel mask is carried with the science images. Different bits -are set to identify different reasons for masking the pixel. Flags -are required for at least each of the following pixel attributes: - -\begin{itemize} -\item The pixel is a charge trap; -\item The pixel is a bad column; -\item The pixel is saturated in the A/D converter; -\item The pixel is non-positive in the flat-field; -\item The pixel is part of a row that has excess noise; and -\item The pixel is determined to be a cosmic ray, based on its -morphology. -\end{itemize} - -\subsubsection{Trim} - -The image is trimmed to remove the non-imaging pixels, such as the -overscan and any pre-scan pixels, along with those pixels near the -edges that have been compromised due to OT operation. The definition -of the imaging area of the detector is determined from the camera -configuration data or from the metadata associated with the image, -with the choice a user-configurable option. - -The input science and mask frames are additionally trimmed by the -extent of the OT convolution kernel in each direction ($+x$, $-x$, -$+y$, $-y$). Within the PSLib image handling functions, the trim -function is a virtual operation which simply marks the boundaries of -the trimmed image but does not remove the corresponding memory. This -allows the later corrections to work with untrimmed correction images -and still apply the correct pixels. At the end of Phase 2, the only -the trimmed portions of the output images are written out to disk. - -\subsubsection{Non-Linearity Correction} - -If required, the science image (after bias correction) must be -corrected for the effects of non-linearity through a provided -polynomial fit to the pixel data values or a numeric lookup table as a -function of pixel flux. The choice to apply the correction must be -set by the analysis recipe. - -\subsubsection{Flat field Correction} - -The science image (after bias correction and non-linearity correction) -must be corrected for sensitivity variations as a function of -position, dividing by a flat-field image. The mask is also modified -for zero-valued pixels in the flat-field image. - -\subsubsection{Sky \& Fringe subtraction} - -After the science image has been flat-fielded, the flux contribution -of the sky (from both continuum emission and the line emission that -causes fringing) must be subtracted from the image. The subtraction -needs to remove background (technically, foreground) variations which -are not celestial but generated in the atmosphere or by more localized -scattering. This background should include the contribution from the -zodiacal light. This background subtraction does not address the -artifacts generated by bright stars: bleeding columns, ghosts, or -other localized reflection effects. This process also produces a -super-binned image of the background map which may be used as a -debugging diagnostic. - -This analysis step temporarily masks objects on the image using the -astrometric solution from the boresight and fits for the sky -background, consisting of a polynomial to model the continuum, and the -fringe frame(s) to model the fringes from sky emission lines. If the -concentration of objects in the image is too high to reliably fit the -sky background, the background solution from an exposure close in time -and airmass to the current object image is used. The output is the -sky-subtracted object image. - -\subsubsection{Detect and Measure objects} - -After the image have been processed by the preceding steps, the Phase -2 analysis performs a basic object detection analysis. Objects on the -flat-fielded object image are found, and general parameters are -measured. Object detection is performed at several stages by the IPP, -with different object parameters measured in each case. -Table~\ref{tab:APdetections} gives a list of the different detection -stages and the object parameters measured for those stages. For the -Phase 2 analysis, the object parameters are: the object centroid and -the position covariance matrix, the instrumental PSF magnitude and -error, local background level and error, a measurement of the -star-galaxy separation, and a measurement of the object shape -($\sigma_x, \sigma_y, \theta$). The detection threshold must be -configurable, and be a function of the average background flux or the -image noise map. Minimal object classification must be performed to -distinguish objects which are consistent with a single PSF, objects -which are inconsistent, and objects which are saturated. The -resulting collection of detected objects are saved in the AP Database -along with the relevant image metadata (\ie filter, exposure time, -etc). In addition, this process constructs a model of the -point-spread-function (PSF) as a function of position in the image. -This PSF model is saved as part of the image metadata. - -\subsubsection{Identify CRs by morphology} - -Charged particles in the detector frequently cause features which do -not have the morphology of astronomical objects. In a well-sampled -image, these may be easily identified by the sharpness of the image. -In a near critically-sampled image, these `cosmic rays' may be -indistinguishable from stellar objects. This analysis step makes -morphological identification of cosmic rays if the imaging data is -sufficiently well sampled. The identified cosmic rays are masked with -a configurable growth factor so that additional pixels beyond the -detected pixels in the feature are also masked. - -\subsubsection{Perform Astrometry} - -The detected objects are matched with known astrometric reference -objects, using reference object coordinates which have been adjusted -for proper motion. The matches are then used to improve the -astrometric parameters for the individual OTAs. The OTA astrometric -parameters which are determined may include terms up to 3rd order in -position, though the terms which are actually fitted are -user-configurable. The Cell astrometric parameters are not allowed to -vary at this stage. The fit must be robust, rejecting outlier matches -(either stars with poorly determined proper motion or spurious -matches). The resulting astrometric solution is consistent across the -OTA field to within 1.0 arcsec. - -\subsubsection{Perform Photometry} - -If possible (if a local photometry reference exists), the Phase 2 -analysis determines a photometry zero point for each image. To do -this, it extracts the appropriate reference objects (from the AP -Database) and matches the stars between the two samples. The -zero-point is derived on the basis of a static atmospheric absorption -model (eg, linear airmass slope). - -\subsubsection{Bright object postage stamps} - -The IPP must have the capability of extracting regions surrounding a -specified subset of objects from the flattened images. These postage -stamp images must be saved for additional use by client science -pipelines. The identification of these objects must be on the basis -of a set of rules applied to the object's magnitude and position. The -postage stamps are not restricted in shape to simple rectangles, but -may represent more complex regions. They are written to the Image -Server. The outputs are these postage stamps and pixel masks, which -are sent to the IPP Pixel Server. - -%\begin{figure} -%\begin{center} -%\resizebox{6in}{!}{\includegraphics{pics/phase2}} -%\caption{ \label{fig:phase2} Phase 2 dataflow - this diagram is old: update} -%\end{center} -%\end{figure} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Phase 3 : exposure analysis} - -The Phase 3 system operates on the combined Phase 2 results from an -FPA to determine improved solutions for the image calibrations and to -provide the parameters needed by Phase 4. The Phase 3 output is saved -by the Metadata Database, and consists largely of improved values of -the calibrations already determined by Phase 2. The analysis -performed by this pipeline consists of: - -\begin{itemize} -\item improved astrometric solution based on comparison between - objects in the images and the astrometric reference. -\item improved background model based on the full telescope field, or - fields. -\item photometric solution based on comparison to photometric - standards -\item FPA-wide PSF analysis -\end{itemize} - -In the Phase 2 analysis, the astrometric solutions were determined -independently for each chip. These solutions are limited by the -assumption of a static distortion and by the accuracy of the -astrometric reference. In the phase 3 analysis, the astrometric -solutions of the complete FPA images are improved by combining the -astrometry for all chips. The astrometry model consists of a -projection of the celestial coordinates to the telescope boresite -center, followed by a rotation to the average rotation of the FPA and -adjustment for the central plate scale. The free parameters in this -stage are the boresite coordinates ($R_o, D_o$), the field rotation -($\theta_o$) and the plate scale ($\rho_o$), and are fitted in Phase -1. These tangent plane coordinates are then distorted by the optical -distortion model, consisting of $N^{\rm th}$ order polynomials in two -dimensions. Finally, the focal plane coordinates are mapped to the -individual chip coordinates, including the chip position and rotation, -as well as possible higher order terms representing warping of the -individual detectors. A first pass at the chip positions is -calculated in Phase 2, while the complete set of parameters is fitted -as a whole during Phase 3. The fitting process is determined in a -robust and stable way by fitting the gradient of the distortion as a -function of field position, removing the degeneracy of the distortion -coefficients on the chip position parameters. - -In the Phase 2 analysis, the background is determined based only on -the available sky in a single chip. However, the background -structures are normally correlated on the scale of the FPA, so an -improved background solution can be determined by combining the -information from many chips. A high-order polynomial model of the -background may be used and subtracted from all chips. - -The Phase 3 photometric improvement is made by comparing the -photometered objects from Phase 2 with the corresponding objects in a -local reference catalog. This analysis may only be performed if a -local reference is available. Note that improved relative photometry -calculations may be performed in the absence of a reference catalog on -the basis of image overlaps in the AP Database {\em after} the -detections have been added to the Database. Such a relative -photometry analysis is outside the scope of Phase 3 and will likely be -performed as an independent analysis process. Given the presence of a -local photometry reference, the zero point variations across the field -may be measured, and possibly modeled. If the zero-point variations -are excessive, then the image is marked as non-photometric by the -analysis. - -In the Phase 4 analysis, the $N$ FPA images are optimally combined to -create a single image of the sky with bad-pixel and cosmic-ray -rejection. This combination requires the calculation of a set of PSF -kernels to convert each of the input images to a single, common PSF. -These PSF kernels are determined from the per-chip PSFs measured in -Phase 2. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Phase 4 : image combination} - -\subsubsection{Overview} - -Phase 4 is the image combination stage, in which multiple images of -the same portion of the sky are merged and confronted with the static -sky image. Phase 4 operates on the smallest data unit of the static -sky, the sky cell, along with the associated pixels from a collection -of images which have been processed through phases 1--3. The size and -exact representation of a static sky cell are yet to be determined. -The working concept is that the static sky cells contain roughly the -same number of pixels as an OTA (4k x 4k) and represent a portion of a -local tangent plane projection. In order to meet the image -degradation requirements, the pixel scale of the static sky is planned -to be 0.2\arcsec, somewhat smaller than the 0.3\arcsec\ raw image -pixel scale. - -For each sky cell, the corresponding pixels are extracted from the -exposures being processed and mapped to the projection of the sky -cell. The pixels from the multiple input processed images are combined -into a single, cleaned image. Outlier pixels may be optionally -rejected at this stage (optionally, since moving objects will be -rejected in images obtained over a wide range of times). This image -is then confronted with the static sky cell data to produce a -difference image. Residual objects in the difference image above a -threshold are detected and excised from the original cleaned image. -The remaining pixels are added to the existing static sky image. -Object detection must be performed on the difference ($P4\Delta$) and -cleaned ($P4\Sigma$) images. - -\subsubsection{Image Warping and Matching} - -The analysis first maps the detector images to the sky cell using the -specified linear transformations, then combines the images with strong -rejection criteria and uses the combined sky cell image to identify -artifacts in the original detector images. It is desirable that the -artifacts are masked in the detector plane (i.e.\ before mapping to -the sky cell) so that they are not smeared out by the mapping; -alternatively, the CR mask needs to be grown by an additional pixel. -The mapped and masked detector images are then optimally combined -using the specified weighting. Both sets of combinations use the -photometric calibration for the images to set the relative scales of -the input images. The limiting magnitude for the combined sky cell -image should also be estimated. -% - -\subsubsection{Static Sky Subtraction} - -The corresponding static sky image is subtracted from the combined -image stack. In this step, it is necessary to match the image kernel -between the input image and the static sky image. This will be done -by solving for a best-fit image kernel which minimizes the difference -image using a technique equivalent to the Allard-Lupton method. One -modification for the IPP is to represent the kernel as a combination -of independent pixels rather than represent the components of the -image difference kernel as a combination of Gaussians. This method -also automatically determines a photometric match between the static -sky image and the input science image. - -\subsubsection{Object Detection and Measurement} - -Objects in the difference image are detected and a specific set of -object parameters are measured from these detections. -Table~\ref{tab:APdetections} gives a list of the different detection -stages and the object parameters measured for those stages. For the -Phase 4 difference image (P4$\Delta$), the measured object parameters -consist of: the object centroid and the position covariance matrix, -the instrumental PSF magnitude and error, local background level and -error, a measurement of the star-galaxy separation, and a measurement -of the object shape ($\sigma_x, \sigma_y, \theta$). The detection -threshold must be configurable, and be a function of the average -background flux or the image noise map. Detections must be performed -for both positive and negative (static sky only) sources. Minimal -object classification must be performed to distinguish objects which -are consistent with a single PSF, objects which are inconsistent, and -objects which are saturated. The resulting collection of detected -objects are saved along with the relevant image metadata (\ie filter, -exposure time, etc). - -Objects in the cleaned, summed image are detected and a specific set -of object parameters are measured from these detections. -Table~\ref{tab:APdetections} gives a list of the different detection -stages and the object parameters measured for those stages. For the -Phase 4 summed image (P4$\Sigma$), the measured object parameters -consist of: the object centroid and the position covariance matrix, -the instrumental PSF magnitude and error, local background level and -error, a measurement of the star-galaxy separation, a measurement of -the object shape ($\sigma_x, \sigma_y, \theta$), the Petrosian radius, -magnitude, axis ratio, and angle; and the S\'ersic radius, magnitude, axis -ratio, angle, and parameter $\nu$. The detection threshold must be -configurable, and be a function of the average background flux or the -image noise map. Minimal object classification must be performed to -distinguish objects which are consistent with a single PSF, objects -which are inconsistent, and objects which are saturated. The -resulting collection of detected objects are saved along with the -relevant image metadata (\ie filter, exposure time, etc). In this -measurement, objects at known positions will also be measured even if -they have not been detected. - -Objects which are detected in both of the Phase 4$\Sigma$ and Phase -4$\Delta$ images are saved to the AP Database, along with the relevant -image metadata (\ie filter, exposure time, etc). In the process of -adding these objects to the database, the transients which are -correlated with previous detections of an object (and those which are -not) will automatically be determined. A subset of these transient -objects are sent, along with their associated metadata, to the MOPS -and other preferred science client pipelines. - -\subsubsection{Static Sky Update} - -It is essential that the static sky image (which may have been -painstakingly accumulated over many months) not be corrupted by adding -in transient sources, or data that is of suspect quality (due, e.g., -to an error upstream in the processing). - -Object analysis of the static sky images is {\em not} a part of the -Phase 4 analysis. This processing is envisioned to take place -relatively infrequently (perhaps weekly or even monthly) and is -scheduled as a separate analysis task, probably run during the day at -a time when the computing infrastructure is not under significant load. - -%\begin{figure} -%\begin{center} -%\resizebox{6in}{!}{\includegraphics{pics/phase4}} -%\caption{ \label{fig:phase4} Phase 4 dataflow} -%\end{center} -%\end{figure} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{System Design : Calibration Image Processing} - -The Calibration Analysis Stages construct calibrations from the -relevant input data. Some of these combine multiple raw input images -together, after processing, to create a high-quality high-signal -master calibration image. Some of the calibrations are used to -correct other calibrations. Each of the calibration stages must also -provide the tools to test the quality of the input data against -existing master calibration data and to test the consistency of -multiple measurements of the calibration. - -The Calibration analysis stages may be performed on whatever -timescales are appropriate and necessary to maintain the quality and -relevance of the calibration images. The specific calibration data -which must be constructed in the calibration analysis stages is listed -below. - -The IPP must generate basic calibration images using the raw bias, -dark, and flat-field (dome or twilight) images obtained by the -telescope as the input. The analysis of these images requires -relatively simple stacking of the input set of images. Outlier -rejection, both of complete input images as well as pixels within the -input stack, must be performed. In addition, each type of image -requires an appropriate normalization which may depend on the data -levels in other detectors in the input set. Each of these calibration -stages must be able to determine from the input stack if the relevant -calibration image needs to be updated and perform an initial test to -see which input images are consistent and valid. - -\subsection{Bias Images} - -Bias images may be needed to correct for structure in the bias. The -IPP must have the capability of constructing a master bias image from -a stack of raw bias frames. The input bias images, representing -offsets from the overscan level, are processed by subtracting the -overscan, including 1D structure if needed. - -The master bias frame construction uses outlier image and outlier -pixel rejection to construct a single high-quality bias frame. The -statistic used to determine pixel values from the input stack can be -set by the user to be one of the following: the sample mean, median, -and mode, robust mean, median, and mode, and the clipped mean and -median. Testing of the input images consists of constructing residual -images, in which the master bias is applied to the input images. -These images may be included or excluded from an additional iteration -of the stack on the basis of their pixel-to-pixel statistics. - -\subsection{Dark Images} - -Dark images may be needed to correct for structure in the dark -current. The IPP must have the capability of constructing a master -dark image from a stack of raw dark frames. The input dark images are -first corrected for the bias using whatever method is appropriate for -the science images. Master dark frames depend on their exposure time. -As such, the input dark frames must have a limited range of exposure -times, and the output dark frame includes the exposure time as part of -its associated metadata. - -The master dark frame construction uses outlier image and outlier -pixel rejection to construct a single high-quality dark frame. The -statistic used to determine pixel values from the input stack can be -set by the user to be one of the following: the sample mean, median, -and mode, robust mean, median, and mode, and the clipped mean and -median. Testing of the input images consists of constructing residual -images, in which the master dark image is applied to the input images. -These images may be included or excluded from an additional iteration -of the stack on the basis of their pixel-to-pixel statistics. A -collection of master dark frames with a range of exposure times are -used to determine the scaling of the dark frame as a function of -exposure time. - -\subsection{On-Off Dark Images for Light Leaks} - -A type of image which may be necessary for calibrations will be pairs -of images taken at night with the shutter closed with and without the -dome shutter closed. Such a pair of images can be used to determine -any light-leak in the camera which may contribute additional flux -across the mosaic. - -\subsection{Flat-Field Images} - -Master flat-field images must be constructed from a collection of -input flat-field images. The input flat-field images may be obtained -from any of the standard sources: the dome, the twilight sky, and the -night-time sky. The choice of flat-field input image must be -determined experimentally from observations during the commissioning -phase of the telescope. The IPP flat-field construction system must -be capable of handling any of these sources. - -An appropriate set of input images is selected on the basis of their -flux levels, time of observations, and the observing conditions. The -input flat-field images are processed (bias and dark corrected if -needed) and the resulting images are stacked. The master flat-field -construction uses image and pixel outlier rejection to construct a -single high-quality master flat-field frame. The statistic used to -determine pixel values from the input stack can be set by the user to -be one of the following: the sample mean, median, and mode, robust -mean, median, and mode, and the clipped mean and median. Testing of -the input images consists of constructing residual images, in which -the master flat-field image is applied to the input images. These -images may be included or excluded from an additional iteration of the -stack on the basis of their pixel-to-pixel statistics. - -\subsection{Mask Images} - -Preliminary bad-pixel mask images are generated on the basis of -comparison between raw flat-field images and a cleaned, stacked -master. The mask creation system accepts a collection of flat-field -images and identifies pixels which are consistently poorly flattened. -Pixels which are under-responsive are also identified as pixels to be -masked. - -\subsection{Sky \& Fringe Frames} - -Fringe-correction frames must be generated to remove the fringe -pattern caused by thin-film interference in the top layers of CCDs, -particularly in the redder passbands. Fringe correction frames may be -constructed on the basis of observations of the night-sky in the -appropriate filters or on the basis of dome fringe lamp observations. -The choice of the appropriate source will be determined experimentally -on the basis of data obtained during the commissioning phase. The IPP -must be capable of handing either source. The images are first -flattened to remove the pixel-to-pixel sensitivity variations of the -detector. The combination of multiple input fringe frames may not be -simply stacked since the amplitude of the fringe pattern varies -independently of other variations in the image. The amplitude of the -fringe pattern in the input frames is measured and the images scaled -to normalize the fringe amplitude to a consistent range (-1 to +1) for -all input images before they are combined with one of the standard -combination statistics (mean, median, mode, etc). The quality of the -input frames is tested by flattening the input image and applying the -master fringe-frame. The resulting residual image statistics are used -to select or exclude specific input images. - -\subsection{Shutter Correction Map} - -Shutter correction map images may be generated based on the timing -measurements of the shutter itself, or on the basis of dome-flat -images of decreasing exposure times down to the shortest available -exposures. - -\subsection{Low-k Sky Models} - -Large-scale background structure in images which is not caused by -thin-film interference must also be detected and corrected. Models of -this background structure may be a necessary input to the correction -procedure. The IPP must have the capability of generating image -models of the large-scale structure patterns observed with the -telescope - -\subsection{Flat-Field Correction Frame} - -Flat-field images, whether constructed from the dome, twilight, or -night-sky images, do not perfectly correct the detector response in a -consistent fashion across the full field of the camera. The IPP must -have the capability of generating flat-field photometric correction -frames on the basis of the measured photometry of objects which are -moved to a variety of locations on the detector in a sequence of -images. The flat-field correction frames analysis stage makes use of -targeted observations following a specified dither pattern, and -extracts the photometered objects from the AP Database to determine -the necessary photometric corrections. The resulting image is applied -to the master flat-field image. Testing of the correction is -performed by applying the correction to the basic master flat-field -image, applying that flat-field image to the dithered photometry -observations, and performing the object detections. Comparison of the -photometry of individual stars at different locations on the mosaic -will demonstrate the consistency of the flat-field image. - -\subsection{Non-Linearity Correction} - -The IPP must have the capability of constructing a correction for -non-linearity in the detectors. These frames are constructed from -exposures of a uniform source with a range of exposure times. The -non-linearity correction frames provide polynomial correction -coefficients or a lookup table describing the correction. There is -likely to be a single non-linear correction for each OTA detector, or -potentially for each Cell. The IPP must handle these two cases. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{System Design : Miscellaneous Tasks} - -This section discusses additional operations which are performed by -the IPP but which do not fall under the analysis of the science images -or the creation of the calibration images. - -\subsection{Retrieval} - -The retrieval stage simply retrieves images from an external source -(ordinarily OTIS at the Summit, but it could conceivably be some other -external source) and store it in the Image Server. - -\subsection{Static Sky Analysis} - -The IPP is responsible for performing object detection and analysis on -the static sky. This analysis is performed continuously (every day or -week) on those portions of the sky within 15\degree\ of the sun. In -this analysis, the object measurement is much more detailed than those -performed in the real-time analysis. The currently envisioned -parameters to be measured for every object are listed in -Table~\ref{tab:APdetections}. The parameters include the object centroid -and the position covariance matrix, the instrumental PSF magnitude and -error, local background level and error, a measurement of the -star-galaxy separation, a measurement of the object shape ($\sigma_x, -\sigma_y, \theta$), the Petrosian radius, magnitude, axis ratio, and -angle; the S\'ersic radius, magnitude, axis ratio, angle, and -parameter $\nu$, and a collection of annular aperture flux -measurements, all of which are also measured for the P4$\Sigma$ -images. In addition, the galaxy-shape parameters $Gamma_1 \& -\Gamma_2$, along with the complete `polarization' terms are measured, -as well as a collection of annular aperture flux and variance -measurements. Another unique feature of the static sky analysis is -that the object detection may be performed simultaneously on all -filters, in which case the locations and other parameters may be more -strongly constrained by simultaneously fitting between all bands. The -analysis to be performed may be substantially more complex than the -real-time analysis because of the relatively low data rate. Instead -of needing to process thousands of images per night ($\sim 350$Mpix -per second), it is only necessary to process the complete sky in a -year, or an average rate of $\sim$2 Mpix per second, or $< 1$\% of the -object analysis in the other analysis stages. - -\subsection{AstroRef: Astrometric Reference Catalog creation} - -\tbd{needs to be fleshed out substantially} - -This processing stage shall use many observations over a given time -period to fit a consistent global astrometric solution, resulting in a -high quality and internally-consistent astrometric catalog that may be -published. - -\subsection{PhotoRef: Photometric Reference Catalog creation} - -\tbd{needs to be fleshed out substantially} - -This processing stage shall use many observations over a given time -period to fit a consistent global photometric solution, resulting in a -high quality and internally-consistent photometric catalog that may be -published. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Software Hierarchy} - -In order to facilitate testing and development, and to encourage -flexibility, the IPP will be built in a layered fashion. The lowest -level functions will be written in C and collected together into a -Pan-STARRS library. These library functions will be used to write -more complex modules. The modules will be written in C but will make -use of SWIG to make their functionality available within other -languages. In particular, the modules can be tied together with a -program in C or through the use of a high-level language such as Perl, -Python, or Tcl employing the SWIG interfaces. For the high-level -functions in the operational system, the IPP will make use of Perl as -the scripting language to provide the required flow-control to tie the -modules together. - -This approach satisfies the requirement that complicated low-level -analysis steps run fast, while preserving flexibility for coding the -high-level wrappers for which the speed requirements are not so -stringent. - -\subsection{External Libraries} - -Pan-STARRS will employ several external libraries to save duplicating -functionality that is already available. These external libraries -will be wrapped by the Pan-STARRS Library, insulating the project from the -implementation details of the external libraries. Examples of the -external libraries are FFTW and SLALib. - -\subsection{Pan-STARRS Library} - -The Pan-STARRS Library will consist of C structures describing the -basic data types needed by the IPP and C functions which perform the -basic data manipulation operations. Note that a subset of the library -functions will be provided with SWIG interfaces as well to allow for -their use in the creation of the processing stages. Examples of the -Pan-STARRS Library are Fourier transforms and transforming between -pixel and celestial coordinates. The details of the Pan-STARRS -Library are specified in the document Pan-STARRS IPP PSLib -Supplementary Design Requirements Specification (PSDC-430-007), which -also addresses coding requirements detailed in the IPP PS-1 SRS -(PSDC-430-005), Section 3.3. - -\subsection{IPP Modules} - -The IPP analysis stages are broken down into modules which represent -specific functional operations. The modules will be written in C -using the Pan-STARRS Library functions and will be grouped into a -Pan-STARRS Module Library. The modules will be provided with SWIG -interfaces to all public APIs for their use in processing stages. -Examples of modules are overscan subtraction and image combination. -Some modules (e.g.\ find objects on an image) will be used by multiple -stages. The details of the Pan-STARRS Modules are specified in the -document Pan-STARRS IPP Modules Supplementary Design Requirements -Specification (PSDC-430-012), which also addresses coding requirements -detailed in the IPP PS-1 SRS (PSDC-430-005), Section 3.3. - -\subsection{IPP Stages} - -The major IPP processing tasks are organized into stages, which -consist of multiple modules. Each stage represents a collection of -complex operations performed on a single data entity. Each stage -therefore represents the maximum amount of effort which can be -performed in serial without interaction between parallel threads. The -stages will be written in Perl, linking the modules together. -Examples of stages are Phase 2 (detrend images) and Phase 4 (combine -images from multiple telescopes and search for transients). - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Interfaces} -\label{sec:interfaces} - -\subsection{Internal Interfaces} - -Internal interfaces consist of interactions between the analysis -scripts and the IPP Metadata Database, Image Server or AP Database. -There are also interfaces between the IPP Scheduler, Controller, and -the Metadata Database. - -The science and calibration image processing pipelines make requests -for images from the Image Server, metadata from the Metadata Database, -and push their results back onto the Image Server and Metadata -Database. The Scheduler specifies analysis tasks and sends them to -the Controller, and determines the next action based on the contents -of the Metadata Database. The various subsystems specify the API for -client / server interactions, and are discussed in their individual -section. Commands will be sent using either text-based commands, SOAP -or an equivalent protocol. The format of the exchanged data may be in -one of several forms discussed below. - -FITS Images will be used to transport images between the components of -the IPP. Non-standard FITS images representing triangular collections -of pixels may be used to store the static sky images. - -FITS Tables will be used to store and transport tabular data, -especially large queries from database subsystems. The Auto-coding -technique discussed in Appendix~\ref{sec:AutocodeIO} is used to define many -different table interactions. - -XML files will be used to store and transport data which is not -well-suited to the rectangular form of FITS Tables. Hierarchical data -concepts and variable-length structures fall in this class. Examples -include mosaic astrometry description information and configuration -information. - -SQL queries and C wrappers of SQL queries will be used as the direct -interface to the databases. - -Within IPP and Pan-STARRS in general, process-to-process communication -will be defined through auto-coded APIs which support a limited and -validated communication protocol. The APIs will be coded based on a -table which defines the allowed command set and the grammar to be -used. This mechanism will allow a single code block to define -inter-process communication methods for many Pan-STARRS subsystems, -including, within the IPP, the Scheduler-Controller communications. - -\subsection{External Interfaces} - -This subsection describes the interfaces between the IPP and other -Pan-STARRS systems and the external clients. The interfaces are -illustrated in Figure~\ref{fig:overview}. - -\subsubsection{OTIS} - -The IPP Scheduler may query OTIS for a list of new images and -metadata. The locations of those images in the Summit Pixel Server is -sent back as a table, and all metadata may be sent to the IPP as a -collection of FITS Tables. The IPP also may send quality assessment -information for each FPA and major frame by writing out FITS tables -and notifying OTIS of the presence of the new tables. - -\subsubsection{Camera} - -Images are pulled from the Summit Pixel Server, part of the Camera -team's purview. The locations of the files are sent by OTIS. IPP may -grab these via {\tt http} commands or via {\tt NFS} or another network -file exchange protocol. The IPP notifies OTIS (and Camera) when each -image has been received. - -\subsubsection{PSPS} - -Data will be sent to PSPS from the IPP as part of a daily or weekly -analysis process on the Static Sky. The data will be pushed from the -IPP to PSPS when they are available. The data to be transfered -include: -\begin{itemize} -\item Static Sky images - to be transferred as FITS images or - FITS triangular image regions. -\item Postage Stamps - to be transferred as FITS images. -\item Metadata tables - to be transferred as FITS tables -\item Detections \& Object associations - to be transferred as FITS tables. -\end{itemize} - -\subsubsection{MOPS} - -Data will be sent to MOPS from the IPP as part of the Phase 4 -analysis. The data will be pushed from the IPP to MOPS when they are -available. The data to be transfered include: -\begin{itemize} -\item Image Metadata tables - to be transferred as FITS tables -\item Orphaned Detections - to be transferred as FITS tables -\end{itemize} - -\subsubsection{Other Preferred Client Science Pipelines} - -These cannot be completely defined until the Clients are defined and -their requirements are specified. The expectation is that the data -products will be the same as for the MOPS. The data will be pushed -from the IPP to the Client Science Pipeline when they are available. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Computer Hardware} -\label{sec:Hardware} - -\subsection{PS-1 Cluster Design} - -The PS-1 IPP computer system is designed as a cluster of 'fat bricks': -computers with both processing power and large amounts of local disk -storage. These computers are large rack-mount boxes with space for -10s of disks (24 and 36 disk cases are available) and a motherboard -with two CPUs and two Gig-E ethernet ports. One set of machines is -specified for storage and processing of the individual OTAs up through -Phase 2 (the `OTA nodes'), another set of machines are specified for -storage of the Static Sky and processing of data from Phase 3 and -Phase 4 (the `Sky nodes'). Other machines will be necessary to -support the Metadata DB and the AP DB. - -The IPP PS-1 SRS (PSDC-430-005) specifies the processing throughput -requirements for the IPP. Benchmark tests of the IPP processing -algorithms have been used to drive the design needed to achieve the -throughput requirements. The details of this study are presented in -the IPP Computational Challenge (PSDC-400-006), summarized here. The -analysis measures the processing time (excluding I/O) for both Phase 2 -and Phase 4 on an Intel Pentium 4 processor, and expresses the -processing time in GHz-seconds, under the assumption that a machine -with the same architecture and twice the processor speed will perform -the same analysis in half the time. This is probably a valid -assumption within a limited range on hardware using the same -architecture. Independent tests show that 32-bit Pentium processors -perform somewhat slower (up to a factor of 2) than equivalently rated -64 bit Opteron processors. This discrepancy makes the measured -numbers somewhat conservative, and compensates for the simplified -analysis performed. The benchmarks show that the Phase 2 analysis -takes 12000 GHz-seconds for a complete major frame (4 FPAs) while the -Phase 4 analysis takes 7800 GHz-seconds for the same major frame. - -The total data I/O required for each processing node, both locally to -disk and across the network to other machines, has also been measured. -These numbers in turn depend on whether the data is optimally stored -on the OTA nodes (raw images matched to their calibration images) or -if the data are randomized across the storage nodes. There are also -differences in the analysis for the number of bits per pixel and the -number of calibration images used in the processing. For PS-1, the -`minimal' data set is appropriate, resulting in a total Phase 2 I/O of -21 GBs per major frame and a total Phase 4 I/O of 36 GBs. The -randomized numbers are used as a conservative estimate, under the -assumption the network, not local disk access, is the dominant I/O -bottleneck. - -The analysis assumes each CPU (rated at 2.2 GHz) is associated with -one RAID array (maximum throughput 110 MB/sec) and one network -controller (maximum throughput 70 MB/s). In this case, given the CPU -load and I/O throughput above, Phase 2 will require a total of 190 -seconds of I/O and 5500 seconds of processing distributed across the -cluster. Likewise, the Phase 4 analysis will require a total of 330 -sec of I/O and 3500 seconds of processing. Given the 160 seconds -available per major frame, these numbers imply a total of 63 -processors are needed to keep up with the processing and I/O load. - -The other major driver on the IPP PS-1 cluster is the data storage -requirements. It is necessary to store the raw images from the entire -AP Survey, the MOPS Verification Program (MVP) and the IPP -Verification Program (IVP), and to have storage enough to represent -the Static Sky by the end of the two year mission. These storage -requirements as a function of time are shown in -Figure~\ref{fig:StorageProfile}. Based on the PS-1 Design Reference -Mission (PSDC-230-001), by the end of the second year, the total -storage requirements for raw images and the Static Sky will be 850 TB, -along with and an additional 55 TB needed for the AP DB storage - -To meet these requirements, the IPP cluster is designed to use fat -bricks which will be capable of holding 24 disks each. The 5U / 24 -disk rack mount computer cases are one of the highest density -solutions currently available. A 4U / 36 disk box is also available -and will be considered. The disk purchases will be staggered in three -waves. Before PS-1 goes on the sky, the first 1/3 of the disks (600 -disks total) will be purchased. Since the lead time for disks is -fairly short, the purchase will be made only when other portions of -Pan-STARRS are clearly on a timeline to success. After 9 months -(tentatively 2006 September), the next 1/3 of the disks will -purchased, and the remaining disks 9 months after that (tentatively -2007 June). Using conservative estimates of the available disk sizes -at these purchase dates (400 GB, 600 GB, and 900 GB), and allocating 1 -of 12 disks to the RAID and 10\% of the volume to file system and -binary Gigabyte overheads, the disk purchases outlined above result in -a total volume after the last purchase of 950 TB. This meets the -requirements with 10\% spare excess. The disk volume profile is also -shown in Figure~\ref{fig:StorageProfile} and shows that the disk space -will be available in the time it is required. - -The total number of computers to be purchased is 80. This provides -the 1800 disk slots and more than enough processors to meet the -processing requirements. This also leaves 5 live spare machines. - -There are two details which are not included in the analysis above: -compression and replication. Compression of the older raw data will -reduce the volume requirements by a factor of roughly two. However, -replication of the data across the network is necessary to ensure the -data against catastrophic failures on a single machine. Replication -doubles the total data space needed. These two factors will tend to -cancel each other, and are ignored in the calculations above. - -The IPP PS-1 clusters will have the following allocations of computers -from this cluster: -\begin{itemize} -\item Phase 2 Nodes: 32 -\item Phase 4 Nodes: 30 -\item AP Database: 10 -\item Metadata Database: 1 -\item Image Server Database: 1 -\item Controller / Scheduler: 1 -\end{itemize} -This distribution meets the projections for computational power for -each of these data systems, and leaves 5 computers as live spares for -redundancy. - -\subsection{PS-1 Cluster Expected Reliability} - -With 80 computers and 1920 disks, component failures are inevitable. -The cluster design and management must be chosen to minimize their -impact on operations and data integrity. - -There are several factors which reduce the cluster's exposure to -hardware failures. First, the use of RAID controllers and RAID-5 -striping of the data will protect the data on a single RAID set -against the failure of a single disk in the array. Second, -duplication of data across the cluster will protect against -catastrophic failures of the array (loss of two disks, loss of the -array controller card). Finally, the flexibility of the distributed -computing plan minimizes the impact the loss of individual machines -has on operations by making changes in the data and processing -assignments on the cluster a trivial matter. - -The components which are most likely to fail in the experience of our -team are, in order: hard drives, RAID controllers, ram, power -supplies, and other components. The hard drive and RAID controller -failure rates are by far the dominant concerns as they potentially -affects the data integrity. - -Most sources (REFS: UCSD article, Samsung White Paper) currently imply -hard disk failure rates (MTBF) in the range 400,000 hours and 500,000 -hours. These are used as an upper limit, with the more historically -conservative value of 100,000 hours used instead. With 1920 disk, -this MTBF implies a failure of one disk every 2.2 days. Since the -disks are in a RAID which reports the disk failures immediately and -drops the array into degraded mode, these failures will not have a -huge impact on the operations, and recovery time is only 10s of -minutes. This failure rate implies that the maintenance plan must -include checks for hard disk failures on a daily basis, and should -make use of email notification and early warning information (ie, -SMART messages). - -A catastrophic failure for the array would require two of the 12 disks -to fail before the first failed disk is replaced. Assuming that -maintenance is poor and it is possible for a disk to take 1 week to -be replaced, the probability of a catastrophe is 1.8\% each time the -first disk fails. Combined with the disk failure rate, RAID -catastrophes are expected 6 times over the 2 year operation of PS-1. -These numbers can be used as a guideline for the level of support -needed to avoid these RAID failures. Note that these 6 failures -should not cause loss of data since the data is duplicated across the -cluster, but they require over 1 day for recovery (as the entire array -must be replicated across the network). - -A detailed IPP computer cluster commissioning and maintenance plan is -specified in the document `Pan-STARRS PS-1 IPP Cluster Support' -(PSDC-430-014). - -\begin{figure} -\begin{center} -\resizebox{6in}{!}{\includegraphics[angle=-90]{pics/ps1_ipp_storage.ps}} -\caption{ \label{fig:StorageProfile} Storage Profile} -\end{center} -\end{figure} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\clearpage -\appendix -\section{Image Server Database Table Contents} -\label{sec:ImageServerTableContents} - -Tables~\ref{tab:ImageServerTables:SO} - \ref{tab:ImageServerTables:VOL} list -the basic contents of the Image Server database tables. - -\begin{table}[bh] -\begin{center} -\caption{Storage Object Table Contents\label{tab:ImageServerTables:SO}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ -\hline -\code{so_id} & integer & internal storage object identifier \\ -\code{ext_id} & string & external storage object identifier (file ID) \\ -\code{comment} & string & user description of object \\ -\code{epoch} & date/time & last date of access \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Instance Table Contents\label{tab:ImageServerTables:INT}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ -\hline -\code{ins_id} & integer & internal instance identifier \\ -\code{so_id} & integer & key to storage object table \\ -\code{uri} & string & location in hardware collection \\ -\code{sha1sum} & string & checksum information \\ -\code{assigned_location} & boolean & is location user-specified? \\ -\code{epoch} & date/time & last date of access \\ -\code{atime} & date/time & last date of access \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Volume Table Contents\label{tab:ImageServerTables:VOL}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype} & {\bf Description} \\ -\hline -\code{vol_id} & integer & internal volume identifier \\ -\code{uri} & string & node name \\ -\hline -\end{tabular} -\end{center} -\end{table} -\clearpage - -\section{Metadata Database Table Contents} -\label{sec:MetadataTableContents} - -Tables~\ref{tab:WeatherTable} -- \ref{tab:overlaps} list the basic contents of -each of the Metadata Database tables listed in Section~\ref{sec:Metadata}. - -\begin{table}[bh] -\begin{center} -\caption{Weather Table: some sample weather points\label{tab:WeatherTable}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Time & date/time & The time the weather information was measured. \\ -Temperature 01 & float & The external temperature \\ -Temperature 02 & float & The temperature at top of the dome \\ -Temperature 03 & float & The temperature on the primary mirror \\ -Humidity & float & The relative humidity. \\ -Pressure & float & The (external) atmospheric pressure. \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{SkyProbe Transparency Table (sample entries)\label{tab:SkyprobeBVTable}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Time & date/time & The time the SkyProbe image was taken. \\ -Filter & string & Filter used for SkyProbe image. \\ -Transparency & float & The derived transparency. \\ -Number of stars & int & The number of stars used to measure the transparency. \\ -Astrometry & coords & The astrometry used on the SkyProbe image. \\ -Exposure time & float & The exposure time of the SkyProbe image. \\ -Sky brightness & float & The measured sky (surface) brightness, counts / second \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Skyprobe Line Absorption Table (sample entries)\label{tab:SkyprobeATable}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Time & date/time & The time the LRProbe observation was taken. \\ -Disperser ID & string & ID of the dispersing element \\ -Atm Component 1 & float & The strength of the 1st atmospheric component. \\ -Atm Component 2 & float & The strength of the 2nd atmospheric component. \\ -Atm Component 3 & float & The strength of the 3rd atmospheric component. \\ -Disperser ID & string & ID of the dispersing element \\ -Number of stars & int & Number of stars used to measure the absorptions. \\ -Astrometry & coords & The astrometry used on the LRProbe image. \\ -Exposure time & float & The exposure time of the LRProbe image. \\ -Sky brightness & float & The measured sky (surface) brightness, in physical units. \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Skyprobe Line Emission Table (sample entries)\label{tab:SkyprobeETable}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Time & date/time & The time the LRProbe observation was taken. \\ -Disperser ID & string & ID of the dispersing element \\ -Atm Component 1 & float & The strength of the 1st atmospheric component. \\ -Atm Component 2 & float & The strength of the 2nd atmospheric component. \\ -Atm Component 3 & float & The strength of the 3rd atmospheric component. \\ -Continuum & float & The strength of the continuum emission. \\ -Disperser ID & string & ID of the dispersing element \\ -Exposure time & float & The exposure time of the LRProbe image. \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{DIMM Measurements Table\label{tab:DimmTable}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Time & date/time & The time the DIMM observation was taken. \\ -$\sigma_x$ & float & Raw dispersion in $x$. \\ -$\sigma_y$ & float & Raw dispersion in $y$. \\ -FWHM & float & Dervied seeing full width at half maximum. \\ -RA & float & The coordinates of the measured star. \\ -DEC & float & The coordinates of the measured star. \\ -Exposure time & float & The exposure time of the DIMM observation. \\ -Telescope ID & string & source of the DIMM data \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Near IR Wide-field Camera Results Table\label{tab:NIR-Table}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Time & date/time & The time the NIR observation was taken. \\ -Sky brightness & float & The sky (surface) brightness in the NIR observation. \\ -Sky variance & float & The variance in the sky (surface) brightness. \\ -Astrometry & coords & The astrometry used on the NIR image. \\ -FOV X & float & field width \\ -FOV Y & float & field height \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Dome Status Table\label{tab:DomeStatusTable}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Time & date/time & The time for which the dome status is valid. \\ -Azimuth & float & The azimuth of the dome. \\ -Open status & boolean & Whether the dome is open or not. \\ -Lights status & boolean & Whether lights are on in the dome or not. \\ -Track status & boolean & Whether dome is tracking telescope or not. \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Telescope Status\label{tab:TelescopeStatusTable}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Time & date/time & The time for which the telescope status is valid. \\ -Guide status & enum & The status of the guiding. \\ -Altitude & float & The telescope altitude. \\ -Azimuth & float & The telescope azimuth. \\ -RA & float & The telescope Right Ascension (ICRS $\approx$ J2000). \\ -Dec & float & The telescope Declination (ICRS $\approx$ J2000).\\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Raw FPA Images\label{tab:RawFPAs}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -ID & string & FPA image ID \\ -RA & float & Coordinates of the boresight (i.e. telescope pointing). \\ -DEC & float & Coordinates of the boresight (i.e. telescope pointing). \\ -Filter & string & Filter used for the exposure. \\ -Image Type & enum & image exposure type \\ -Exposure time & float & Exposure time for the image. \\ -Airmass & float & Airmass at which the image was taken. \\ -ObsFrame ID & int & Observation frame identification number, ties FPAs into major frame \\ -ObsGroup ID & int & Observation group identification number, ties FPAs into observing group \\ -Observer & string & The name of the observer, or the version of the telescope scheduler software. \\ -Program & string & The observing program being executed. \\ -Nchips readout & int & Number of detector chips read out \\ -Camera & string & Identification of camera source \\ -Telescope & string & Telescope used for observation \\ -Astrometry & coords & The astrometry used for the FPA. \\ -Chip Metadata & string & metadata resource file \\ -Cell Metadata & string & metadata resource file \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Pending Science Chips\label{tab:PendingChips}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -FPA ID & string & FPA image ID \\ -Chip ID & string & Chip identification number. \\ -Proc Status & enum & Current Processing Status. \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Processed Science Chips\label{tab:ProcessedChips}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -FPA ID & string & FPA Image ID \\ -Chip ID & string & Chip identification number. \\ -Status & enum & Current Processing Status. \\ -Residual Stats & float & quality statistics. \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Observation Group Information\label{tab:OBSGroup}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -ObsGroup ID & string & Identification number for the observation group. \\ -Number of images & string & Number of images in the observation group. \\ -Type & string & Type of observation. \\ -Status & string & Status of the observation group. \\ -etc & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Observation Frame Information\label{tab:OBSFrame}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -ObsFrame ID & string & Identification number for the observation frame. \\ -Number of images & string & Number of images in the observation group. \\ -Type & string & Type of observation. \\ -Status & string & Status of the observation group. \\ -etc & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Science Processing Stats\label{tab:PSStats}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Chip ID & string & The chip identification number. \\ -State & string & The state of the processing. \\ -ObsFrame ID & string & The major frame the chip belongs to. \\ -ObsGroup ID & string & The observation group the chip belongs to. \\ -P1 astrom & string & The Phase 1 astrometry results file. \\ -P2 astrom & string & The Phase 2 astrometry results file. \\ -P3 astrom & string & The Phase 3 astrometry results file. \\ -N guide stars & string & Number of guide stars used for the exposure. \\ -Astrometry stats & string & Summary statistics for astrometry (number of stars, $sigma_x$, $sigma_y$) \\ -Astrom catalog & string & The reference catalog that was used for the astrometry. \\ -Bias method & string & Method used to correct the bias. \\ -Bias stats & string & Summary statistics for bias \\ -Flat-field image & string & The flat-field image that was applied. \\ -Kernel data & & A description of the OT kernel. \\ -Flat-field stats & & Summary statistics for flat-field (sigma of sky). \\ -Mask image & string & The mask image that was applied. \\ -Mask method & string & The algorithm used to mask the bad pixels. \\ -Fringe images & string & The fringe model images that were used. \\ -Fringe stats & & Summary statistics for fringes (fringe amplitude, sky sigma) \\ -Object stats & & Summary statistics for object detection (number of objects, depth, other input parameters). \\ -Photometry data & & photometry information: magnitude zero point and other corrections. \\ -Photometry stats & & Summary statistics for the photometry (number of stars, $sigma_m$) \\ -Photom catalog & string & The reference catalog that was used for the photometry. \\ -PSF stats & & Summary statistics of the PSF. \\ -Software ver & string & Versions of each of the modules used in the processing. \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Chip / Sky overlaps\label{tab:overlaps}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Chip ID & string & The identification number of the chip. \\ -Sky Cell ID & string & The identification number of the sky cell. \\ -State & string & Processing state of overlap \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Processed Sky-Cell stats\label{tab:ProcessedSky}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Input Chips & string & Identification numbers of the chips used to produce the sky cell. \\ -PSF adjustments & string & Adjustments to the PSF. \\ -CR rejection stats & string & Statistics from the CR rejection (number of CRs, distribution, limiting flux). \\ -Image comb params & string & Parameters used for the image combination. \\ -Diff image params & string & Parameters used for the image differencing. \\ -Average weight & string & The weight of the reference image \\ -P4D object stats & string & Summary statistics of the object detection \\ -P4S object stats & string & Summary statistics of the object detection \\ -Software versions & string & Software versions of modules used in the sky cell processing. \\ -Processing stats & string & Summary statistics of the processing (CPU time, etc). \\ -\hline -\end{tabular} -\end{center} -\end{table} -\clearpage - -\section{AP Database Table Contents} -\label{sec:APDBTableContents} - -\begin{table}[bh] -\begin{center} -\caption{Images\label{tab:images}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Image ID & & \\ -time/date & & \\ -Exposure Time & & \\ -Nstars & & \\ -NX & & \\ -NY & & \\ -photcode & & \\ -Mcal & & \\ -Mcal error & & \\ -Mcal chisq & & \\ -Airmass & & \\ -Astrometry & & \\ -PSF & & \\ -flags & & \\ -Camera & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Image Overlaps\label{tab:ImageOverlaps}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Image ID & & \\ -Region Table & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Objects\label{tab:Objects}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -ID & & \\ -$\alpha$ & & \\ -$\delta$ & & \\ -$\mu_{\alpha}$ & & \\ -$\mu_{\delta}$ & & \\ -$\sigma_{\alpha}$ & & \\ -$\sigma_{\delta}$ & & \\ -$\chi^2$ position & & \\ -$N_{\rm det}$ & & \\ -$N_{\rm miss}$ & & \\ -flags & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Average Magnitudes\label{tab:AveMags}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -object ID & & \\ -$M_{\rm int}$ & & \\ -$M_{\rm ext}$ & & \\ -$\chi^2_{\rm mag}$& & \\ -$\sigma_{\rm mag}$& & \\ -photcode & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Solar System Objects\label{tab:SSObjs}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -SSO ID & & \\ -$N_{\rm det}$ & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Matched Detections\label{tab:Detections}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -$\alpha$ & & \\ -$\delta$ & & \\ -$\sigma_{\alpha}$ & & \\ -$\sigma_{\delta}$ & & \\ -$M_{\rm inst}$ & & \\ -$M_{\rm cal}$ & & \\ -$\sigma_{\rm mag}$& & \\ -photcode & & \\ -type & & \\ -flags & & \\ -time/date & & \\ -airmass & & \\ -$\sigma_{x}$ & & \\ -$\sigma_{y}$ & & \\ -$\theta$ & & \\ -object ID & & \\ -exptime & & \\ -sky & & \\ -$\sigma_{\rm sky}$& & \\ -etc & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Orphaned Detections\label{tab:Orphans}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -$\alpha$ & & \\ -$\delta$ & & \\ -$\sigma_{\alpha}$ & & \\ -$\sigma_{\delta}$ & & \\ -$M_{\rm inst}$ & & \\ -$M_{\rm cal}$ & & \\ -$\sigma_{\rm mag}$& & \\ -photcode & & \\ -type & & \\ -flags & & \\ -time/date & & \\ -airmass & & \\ -$\sigma_{x}$ & & \\ -$\sigma_{y}$ & & \\ -$\theta$ & & \\ -exptime & & \\ -sky & & \\ -$\sigma_{\rm sky}$& & \\ -etc & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Non-detections\label{tab:NonDetects}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -object ID & & \\ -$N_{\rm non-det}$ & & \\ -last time/date & & \\ -last mag & & \\ -faintest time/date & & \\ -faintest mag & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Regions\label{tab:Regions}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -$\alpha_0$ & & \\ -$\alpha_1$ & & \\ -$\delta_0$ & & \\ -$\delta_1$ & & \\ -Region ID & & \\ -Parent ID & & \\ -Nchildren & & \\ -Images & & \\ -Objects & & \\ -Detections & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Filters\label{tab:Filters}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Filter ID & & \\ -Filter name & & \\ -Photcode & & \\ -$\lambda_0$ & & \\ -$\delta_\lambda$ & & \\ -$\epsilon$ & & \\ -transmission curve& & \\ -time/date & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Photcodes\label{tab:Photcodes}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Photcode & & \\ -Telescope & & \\ -Camera & & \\ -Detector & & \\ -Filter & & \\ -Nominal ZP & & \\ -airmass terms & & \\ -color terms & & \\ -Target & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Zero Point History\label{tab:Zpts}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Photcode & & \\ -start Time/date & & \\ -end Time/date & & \\ -Zero Points & & \\ -airmass & & \\ -color & & \\ -error & & \\ -N measurements & & \\ -N stars & & \\ -photom ref set & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Distortion History\label{tab:Distortions}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -Camera & & \\ -Telescope & & \\ -distortion terms & & \\ -time/date & & \\ -residuals / error & & \\ -N stars & & \\ -N images & & \\ -astrom ref set & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\begin{table}[bh] -\begin{center} -\caption{Database Hosts\label{tab:APDBHosts}} -\begin{tabular}{lll} -\hline -\hline -{\bf Column Name} & {\bf Datatype } & {\bf Description} \\ -\hline -machine name & & \\ -machine ID & & \\ -\hline -\end{tabular} -\end{center} -\end{table} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\section{Software Runtime Configuration Issues} -\label{sec:RuntimeConfig} - -The IPP Software requires extensive runtime configuration information. -This includes default parameters for analysis to be performed, -descriptions of how a particular analysis is performed, locations of -data sources, and so forth. The IPP may store this information in the -Metadata Database or in configuration files available to the user. -Both methods are implemented in the current design. In either method, -the necessary parameters are identical. This section discusses the -contents of specific portions of the runtime configuration. - -\subsection{Camera Definition Information} - -Every camera which may be analysed by the IPP has differences in how -the data is represented. The IPP is built with the flexibility to -handle data from many different cameras, not just the Pan-STARRS -Gigapix cameras. This is partly to allow testing of the analysis -system on data from other telescopes, such as MegaPrime on CFHT and -Suprime on Subaru, but also to allow us to adapt to changes in the -design of the Gigapix cameras themselves. It also means the IPP -software may be used by astronomers for other analysis projects beyond -the IPP. - -Most cameras provide extensive descriptive information in the FITS -image headers when the images are read out. Typically, the location -and orientations of the individual detectors are defined by keywords -such as DATASEC and DETSEC. Other variations on these words are used -for cameras which place the pixels from multiple amplifiers in the -same FITS data segment. Other parameters, such as astrometric -information or exposure times, are stored in headers as well. It is -possible to use these header keywords to guide the analysis software, -but there are two difficulties. - -First, it is very common for different keywords to be used by -different cameras, sometimes even the same camera may use different -keywords for the same information at different times (major readout -software upgrades, for example, can be accompanied by keyword -revisions). In addition, within Pan-STARRS and the IPP, it is -necessary to have the capability to refer to the Metadata database as -the authoratative sources of some of these entries rather than the -image headers. Given this circumstance, it is at least necessary to -define the appropriate source for a given data concept appropriate to -data from a specific camera. - -The second problem arises when actually performing an analysis. In -many circumstances, the software needs to know what data to expect -even when an appropriate camera image is not available. This is -particularly true for a camera which is composed of multiple chips and -multiple amplifiers. It is a frequent circumstance than some subset -of the chips or amplifiers will either be unavailable or are invalid -for one reason or another. It is important for the software to have a -guide for what data should be available from a perfect readout of the -given camera so decisions can be made how to handle data which is not -complete. This is also important to validate that a particular -dataset, which appears to be from a known camera, actually corresponds -to that camera and has all of the necessary information where -expected. - -In order to facilitate the operation of the IPP with a variety of -cameras, and to allow the software the flexibility to change the -camera defintion dynamically, the IPP includes a collection of -software runtime configuration information which defines a given -camera. This information is represented below in the form of the -PSLib Metadata Config file, but may be stored in the Metadata Database -or in an alternate format as appropriate. - -The a single camera is represented as a Focal Plane Array (FPA), -divided into Chips, divided into Cells. For a single FPA, all imaging -data is stored in a FITS file or a collection of FITS files. Software -needs to know where in a given file or set of files to find a -particular Cell, what Cells to expect, what chips to expect, and the -relationships between those entities, etc. - -A single camera configuration file (or dataset) represents the -description of a complete FPA. In the configuration file, any -parameters which are specific to the complete FPA are placed on their -own lines. These include the definition of the keywords or database -locations. An incomplete example is given below. - -\begin{verbatim} -NCELL S32 NN -NCHIP S32 NN -EXPTIME-SRC STR HD:EXPTIME # need to specify PHU vs EXTNAME -EXPTIME-KEY STR EXPTIME -DATE-KEY STR DATE-OBS -DATE-FMT STR YYYY/MM/DD - -TYPE CELL FILENAME EXTNAME CHIP DATASEC BIASSEC -CELL.nn CELL @ROOT@CELL AMP00 CHIP.00 CF:[0,0:0,0] HD:BIASSEC -CELL.01 CELL @ID/@ID@CELL.fits AMP01 CHIP.00 DB:??? -\end{verbatim} - -\subsection{Analysis Recipe Information} - -In order to maintain flexibility in the analysis details, the IPP uses -recipes to define how a particular analysis is implemented. Each -major analysis script (eg, Phase 2) has its own recipe configuration -information, which may be stored in the Metadata Database or in the -form of the PSLib Metadata Config file. This configuration -information includes all of the user configurable parameters. Many of -these may specify a specific value, or they may specify lookup methods -(database locations, or header locations). The specifies of each -depends on the context. Below is an example recipe file for the bias -subtraction portion of Phase 2, giving several alternative options for -certain entries. Note that, for example, the overscan subtraction may -be specified as using a particular region given in the recipe file, or -on the basis of a particular header keyword. - -\begin{verbatim} -# BIAS: -BIAS.IMAGE STR NONE -BIAS.IMAGE STR FILE:bias.fits -BIAS.IMAGE STR DB:BEST -BIAS.IMAGE STR DB:CLOSE - -BIAS.OVERSCAN STR HD:BIASSEC -BIAS.OVERSCAN STR CF:[0,16:0,2048] -BIAS.OVERSCAN STR NONE - -BIAS.OVERSCAN.STATS STR MEDIAN -BIAS.OVERSCAN.STATS STR MEAN - -BIAS.OVERSCAN.FIT STR SPLINE -BIAS.OVERSCAN.FIT.NPTS S32 5 - -BIAS.OVERSCAN.FIT STR POLYNOMIAL -BIAS.OVERSCAN.FIT.ORDER S32 3 -BIAS.OVERSCAN.FIT.NBIN S32 5 -\end{verbatim} - -\section{I/O Code Autogeneration} -\label{sec:AutocodeIO} - -The IPP includes a number of data collections which have multiple -representations. A software tool will be used to automatically -generate code to provide I/O APIs to read and write these data and to -define the data structures used to carry them within a program. -Within the IPP, examples of these different data entities include -database tables (ie, in the Metadata Database), FITS Tables (to -exchange bulk data), and XML (to exchange more complete datasets). - -I/O API Autocode template (example.def): -\begin{verbatim} -Name Example -Table EXAMPLE -EXTNAME EXAMPLE - -KEY XVALUE - -# name format unit comment -XVALUE F32 pixels "x coordinate" -BINNING S32 fraction "binning factor" -NAME STR[32] string "description of entry" -\end{verbatim} - -Running autocode on such a file would generate an output header and C -files \code{example.h, example.c} with the following structure and APIs: - -\begin{verbatim} -typedef struct { - psF32 XVALUE; // x coordinate - psS32 BINNING; // binning factor - char NAME[32]; // description of entry -} Example; - -psMetadata *psFITSTableInitExample (); -psExample *psFITSTableLoadExample (char *filename, int *Nrows); -bool psFITSTableSaveExample (char *filename); - -psMetadata *psDatabaseTableInitExample (); -psExample *psDatabaseTableLoadExample (char *filename, int *Nrows); -bool psDatabaseTableSaveExample (char *filename); -psExample *psDatabaseTableLoadExampleRow (char *filename, psF32 XVALUE); -\end{verbatim} - -%\bibliographystyle{plain} -%\bibliography{panstarrs} - -\input{glossary.tex} - -\end{document}