Index: /trunk/doc/design/glossary.tex =================================================================== --- /trunk/doc/design/glossary.tex (revision 2545) +++ /trunk/doc/design/glossary.tex (revision 2545) @@ -0,0 +1,113 @@ +{\Large \bf Acronyms} + +{\footnotesize +\begin{tabular}{|p{50pt}|p{400pt}|} \hline +CAN & Controller Area Network \\ \hline +CCD & Charge Coupled Device \\ \hline +CFHT & Canada-France-Hawaii Telescope \\ \hline +DC & Data Collection - Database or other data storage container. \\ \hline +DML & Device Meta-Language \\ \hline +DMT & Dark Matter Telescope \\ \hline +FOM & Figure of Merit \\ \hline +FOV & Field of View \\ \hline +FWHM & Full-Width at Half-Maximum \\ \hline +GPC & Giga-Pixel Camera \\ \hline +GS & Guide Star \\ \hline +GUI & Graphical User Interface \\ \hline +IAU & International Astronomical Union \\ \hline +IOD & Initial Orbit Determination \\ \hline +IPP & Image Processing Pipeline \\ \hline +KBO & Kuiper Belt Object \\ \hline +LAN & Local Area Network \\ \hline +LSN & Local Solar Neighborhood \\ \hline +LSS & Large-Scale Structure \\ \hline +LSST & Large Synoptic Survey Telescope \\ \hline +MBA & Main Belt Asteroid \\ \hline +MDS & Medium-Deep Survey \\ \hline +MOID & Minimum Orbital Intersection Distance - The minimum distance between two orbits. \\ \hline +MOPS & Moving Object Processing System \\ \hline +MPC & Minor Planet Center (of the IAU) \\ \hline +NEO & Near Earth Object - An asteroid or comet with perihelion < 1.3AU. \\ \hline +OBS & Observation Seqencer \\ \hline +ODA & OTIS Data Archive \\ \hline +OOF & OTIS Observe File \\ \hline +OOT & OTIS Observation Tool \\ \hline +OTA & Orthogonal Transfer Array \\ \hline +OTF & Optical Transfer Function \\ \hline +OTIS & Observatory and Telescope System \\ \hline +OWS & OTIS Weather Server \\ \hline +PHO & Potentially Hazardous Object \\ \hline +PSDC & Pan-STARRS Document Control \\ \hline +PSF & Point Spread Function \\ \hline +PSPS & Published Science Products System \\ \hline +PTS & Pan-STARRS Telescope Scheduler \\ \hline +Pan-STARRS & Panoramic Survey Telescope and Rapid Response System \\ \hline +RFI & Radio Frequency Interference \\ \hline +SCD & System Concept Definition \\ \hline +SDSS & Sloan Digital Sky Survey \\ \hline +SGS & Science Goals Statement \\ \hline +SNe & Supernovae \\ \hline +SRS & Software Requirements Specification \\ \hline +SSS & Solar System Survey \\ \hline +TAC & Time Allocation Committee \\ \hline +TBD & To Be Determined \\ \hline +TBR & To Be Reviewed \\ \hline +TCS & Telescope Control System \\ \hline +TLA & Three Letter Acronym \\ \hline +\end{tabular}} + +\newpage +{\Large \bf Acronyms (con't)} + +{\footnotesize +\begin{tabular}{|p{50pt}|p{400pt}|} \hline +TLR & Top Level Requirements \\ \hline +TNO & Trans-Neptunian Object \\ \hline +TTI & Transient Time Interval \\ \hline +UDS & Ultra-Deep Survey \\ \hline +UDS & Ultra-Deep Survey \\ \hline +UET & Unit Exposure Time \\ \hline +WFS & Wavefront Sensor \\ \hline +WFSS & Wavefront Sensor Star \\ \hline +WL & Weak Lensing \\ \hline +- & - \\ \hline +RA & right ascension \\ \hline +DEC & Declination \\ \hline +GLAT & Galactic Latitude \\ \hline +GLON & Galactic Longitude \\ \hline +ELAT & Ecliptic Latitude \\ \hline +ELON & Ecliptic Longitude \\ \hline +\end{tabular}} + +{\Large \bf Glossary} + +{\footnotesize +\begin{tabular}{|p{50pt}|p{400pt}|} \hline +Subaru & National Astronomical Observatory of Japan's 8.3m telescope \\ \hline +cadence & \\ \hline +Phase 1 & IPP image processing preparation stage \\ \hline +Phase 2 & IPP image reduction stage \\ \hline +Phase 3 & IPP exposure analysis stage \\ \hline +Phase 4 & IPP image combination stage \\ \hline +Detection & Identification of a source (real or not) in an image \\ \hline +Observation & In MOPS, a detection that corresponds to a real Solar System object \\ \hline +Designation & The identifying label assigned to newly identified Solar System objects \\ \hline +Orbit Identification & The identification of two separately determined orbits as representing the same object. \\ \hline +Attribution & The identification of a detection with a known orbit. \\ \hline +Linkage & The identification of sets of detections that allow an orbit determination for a Solar System object \\ \hline +Autonomous & operates at night without human intervention + for a minimum of three nights out of seven. + Daytime summit calibration and maintainance carried out + four consecutive days a week (Monday-Thursday). \\ \hline +Robotic & operates at night without human intervention + for a minimum of three nights out of seven. + Only four days summit maintainence per week necessary, + human intervention in calibration not required. \\ \hline +Remote & operates without human intervention at the summit at night. \\ \hline +Transient Time Interval & Time interval between two successive images of the same footprint in order to distinguish between stationary and non-stationary transient detections. \\ \hline +Trans-Neptunian Object & An asteroid or comet that spends most of its time outside Neptune's orbit. Include classical \\ \hline +Sweet Spots & \\ \hline +Potentially Hazardous Object & An asteroid or comet with MOID<0.05AU with Earth's orbit \\ \hline +Observing Efficiency & ratio of shutter open time to total time in a night excluding weather loss \\ \hline +\end{tabular} +} Index: /trunk/doc/design/ippSCDdraft.tex =================================================================== --- /trunk/doc/design/ippSCDdraft.tex (revision 2545) +++ /trunk/doc/design/ippSCDdraft.tex (revision 2545) @@ -0,0 +1,2854 @@ +% -*- latex -*- +%%% $Id: ippSCDdraft.tex,v 1.1 2004-11-30 23:16:57 eugene Exp $ + +\providecommand{\setflag}{\newif \ifwhole \wholefalse} +\setflag +\ifwhole\else +\documentclass[panstarrs,spec]{panstarrs} +\setcounter{secnumdepth}{5} +\begin{document} +\section{Pan-STARRS IPP SCD Standalone Section} +\newcommand\PSeten{$\gtrsim 40$\,m$^2$\,deg$^2$} +\newcommand\PSiqdeg{$<$ 27\%} +\newcommand\FRM[2]{\parbox[t]{#1}{\raggedright #2}} +\newcommand\FRA{100pt} +\newcommand\FRB{260pt} +\newcommand\FRC{40pt} +\newcommand\FRD{80pt} +\newcommand\FRN[1]{\FRM{50pt}{#1}} +\newcommand\FRS[1]{\FRM{190pt}{#1}} +\newcommand\grizy{$grizy$} +\fi + +% \section{Image Processing Pipeline} + +\subsection{Subsystem Overview} +\label{IPP} + +The Pan-STARRS Image Processing Pipeline (IPP) performs the image +processing and data analysis tasks needed to enable the scientific use +of the images obtained by the Pan-STARRS telescopes. The primary +goals of the IPP are to process the science images from the Pan-STARRS +telescopes and make the results available to other systems within +Pan-STARRS. It also is responsible for combining all of the science +images in a given filter into a single representation of the +non-variable component of the night sky called the ``Static Sky''. To +achieve these goals, the IPP also performs other analysis functions to +generate the calibrations needed in the science image processing and +to occasionally use the derived data to generate improved astrometric +and photometric reference catalogs. It also provides the +infrastructure needed to store the incoming data and the resulting +data products. + +The IPP inherits lessons learned, and in some cases code and prototype +code, from several other astronomy image analysis systems, including +Imcat (Kaiser), the Sloan Digital Sky Survey (REF), the Elixir system +(Magnier \& Cuillandre), and Vista (Tonry). Imcat and Vista have a +large number of robust image processing functions. SDSS has +demonstrated a working analysis pipeline and large-scale database +system for a dedicated project. The Elixir system has demonstrated an +automatic image processing system and an object database system for +operational usage. + +The users of the IPP output are all systems internal to the Pan-STARRS +project. They consist of the Transient Science Client, which will +receive the detections of transient objects on short time-scales; the +Moving Object Processing System (MOPS), which will receive the +detections of non-stationary transient objects on day-to-week +timescales; and the Published Science Products Subsystem (PSPS), which +will receive all data products of interest to the outside world, and +will act as the long-term archive and publishing clearinghouse. + +An important operational choice for the IPP is the decision not to +attempt to save all raw data. Once the IPP is running in a standard +operational mode, data will be processed by the pipeline and deleted +when it is no longer needed. Raw images will only be saved for a +short period to allow tests and quality assurance, and potentially to +allow systems which study transient phenomena to return to recent data +for closer inspection. In general, during normal operations, raw +science images will be deleted after $\sim$1 month. + +The primary IPP hardware system on which the software operates will +not be located at the summit. Instead, because of thermal, power, and +space constraints, the hardware will likely be located in a facility +off the mountain. A subset of processing tasks may eventually be +assigned to machines at the summit if justified by the savings in data +transfer time and cost. + +\subsection{Subsystem Top-level Requirements} + +The IPP has the following top-level requirements derived from the +system requirements above (Section~\ref{sys:TLR}): + + +\begin{enumerate} +\item For images obtained in photometric weather with normal detector + characteristics and providing appropriate flat-field images and + correction data have been obtained, the IPP shall produce reduced + science images for each full camera exposure with relative + photometric zero-point scatter less than 1\% ($1 \sigma$) across the + full field. + \label{TLR:1} + +\item For images of reference fields calibrated for the IPP filter set + and obtained in photometric weather with normal detector + characteristics and providing appropriate flat-field images and + correction data have been obtained, the IPP shall determine and + track zero-points for these exposures with a 1$\sigma$ accuracy of + 1\%. + \label{TLR:2} + +\item For images obtained under normal seeing conditions and optical + distortion, the IPP shall produce reduced science images for each + full camera exposure with an astrometric calibration providing $< + 30$ milliarcsecond scatter (1$\sigma$) for sequential images of the + same location. + \label{TLR:4} + +\item For images obtained under normal seeing conditions and optical + distortion, the IPP shall produce reduced science images for each + full camera exposure with an astrometric calibration providing $< + 100$ milliarcsecond scatter (1$\sigma$) relative to the ICRS + reference system. + \label{TLR:3} + +\item In photometric weather and under moon conditions listed in + Table~\ref{moonconditions}, the IPP shall produce reduced science + images for each full camera exposure which have background + variations of less than 1\% in regions free of large ($> 30$ pixels + diameter) astronomical structures. + \label{TLR:5} + +\item In photometric weather, the IPP shall produce reduced science + images for each full camera exposure which have background + deviations from the static sky in the same filter of less than 1\% + for the median in large ($> 30$ pixels diameter) + regions. + \label{TLR:5a} + +\item The IPP shall merge all $g$ filter science images into a static sky image. + \label{TLR:6} + +\item The IPP shall merge all $r$ filter science images into a static sky image. + \label{TLR:7} + +\item The IPP shall merge all $i$ filter science images into a static sky image. + \label{TLR:8} + +\item The IPP shall merge all $z$ filter science images into a static sky image. + \label{TLR:9} + +\item The IPP shall merge all $y$ filter science images into a static sky image. + \label{TLR:10} + +\item The IPP shall merge all $w$ filter science images into a static sky image. + \label{TLR:11} + +\item The IPP shall detect and classify objects on the individual processed science + images. + \label{TLR:12} + +\item The IPP shall detect and classify objects on the stacked groups + of science images. + \label{TLR:13} + +\item The IPP shall detect and classify objects on the static sky + image. + \label{TLR:14} + +\item The IPP shall detect transients with significance $>3\sigma$ in + the individual science images relative to the static sky + image. + \label{TLR:15} + +\item The IPP shall degrade the stacked image by no more than \tbr{10 + milliarcseconds (FWHM added in quadrature)} over the theoretical + limit for the stack under infinite + sampling. + \label{TLR:16} + +\item The IPP shall perform the processing of science images to the + level of transient detection and static sky inclusion at a rate such + that exposures taken at an \tbr{average cadence of 40 seconds} do + not accumulate in the processing buffer (average throughput + requirement). + \label{TLR:17} + +\item The IPP shall limit the false alarm rate (FAR) to less than 5\% + for transient detections $> 5\sigma$ sent to the preferred client + science pipelines.\footnote{note difference with PS-4: 1\%} + \label{TLR:18} + +\item The IPP shall perform transient detection to a completeness of + 99\% at the completeness for transient detections with a significant + $> 5\sigma$. + +\item The IPP shall publish the static sky images to the Pan-STARRS + Published Science Products Subsystem (PSPS) at a rate so the full + sky is transmitted once per year. + \label{TLR:19} + +\item The IPP shall publish the detected objects to the Pan-STARRS + Published Science Products Subsystem (PSPS) at a rate such that the + objects from the full sky are transmitted once per + year. + \label{TLR:20} + +\item The IPP shall send the IPP metadata and received OTIS metadata + to the Pan-STARRS Published Science Products Subsystem (PSPS) + weekly. + \label{TLR:21} + +\item The IPP shall provide access to preferred Pan-STARRS science clients to the + detected transient objects within \tbr{5 minutes}. + \label{TLR:22} + +\item The IPP shall provide sufficent storage volume for raw images from the AP and + IVP Surveys and the \grizy\ Static Sky.\footnote{note difference with + PS-4: 1 month of raw images} + \label{TLR:23} + +\item The IPP shall provide sufficient storage volume for all detections from the + AP, IVP, and MVP Surveys.\footnote{note difference with PS-4: 1 year + of detections} + \label{TLR:24} + +\item The IPP shall provide sufficient storage volume for 2 years of + metadata.\footnote{note difference with PS-4: 10 years of + metadata} + \label{TLR:25} +\end{enumerate} + +\subsection{Subsystem Top Level Description} + +We now discuss three aspects of the IPP and their relationships: the +IPP Analysis Stages, the Architectural Components which make up the +IPP, and the Hardware System which provides the basic computer +resources needed by the IPP. We discuss these aspects of the IPP in +general terms first, and in later sections, address the conceptual +design issues for each of the three aspects of the IPP. + +\subsubsection{Analysis Tasks and Stages} + +Specific programs are required to perform the processing steps listed +above. These can be divided into well-defined analysis stages, each +of which operates on a particular unit of data, such as a single OTA +image or a collection of astronomical objects. Analysis tasks +representing the different analysis stages are performed on the IPP +computer cluster. Note the distinction between the generic analysis +{\em stage} and a specific analysis {\em task}. An analysis stage +represents a type of analysis which is performed, such as the basic +image calibration and object detection analysis. An analysis task is +a particular realization of an analysis stage, e.g., the analysis of +OTA number 61 from exposure 654321 to produce a specific set of output +data products. The analysis stages are discussed in detail in +Section~\ref{IPP:AnalysisStages}. + +\subsubsection{Architectural Components} + +In order to achieve the required functionality, the IPP provides an +infrastructure within which the Analysis Stages above are exectuted. +We have divided the IPP software infrastructure into a number of +clearly-defined architectural software units, listed as follows: + +\begin{enumerate} + +\item {\bf Image Server:} This component is a large data store for all + images used by the IPP, including the raw images from the telescope, + the master calibration images, the reference static-sky images, and + any temporary image data products produced by the IPP. The Image + Server accepts the incoming data and stores it until it is no longer + needed by other portions of the IPP. The Image Server is not + restricted to imaging data: it is capable of storing any large data + files which are not well-suited for inclusion in a more structured + relational database and for which access needs to be widely + available beyond the individual process which created the file. + +\item {\bf Astrometry \& Photometry Database (AP DB):} This component + stores and manipulates astronomical objects detected in various + images, as identified above, including individual measurements of + objects on the images, the summary information about those objects, + and reference object data. It also provides mechanisms for users to + query and manipulate the objects and detections. + +\item {\bf Metadata Database:} This component stores the data which is + not directly related to images or astronomical objects, but which is + needed to perform the IPP analyses. The metadata may include the + summary weather information for each night, or details about the + filters, camera, telescopes, etc. + +\item {\bf IPP Controller:} In order to perform the analysis stages + required by the IPP, it is necessary to use distributed computing + processes on a large number of computers. The IPP Controller + manages the collection of analysis tasks performed on these + machines. + +\item {\bf IPP Scheduler:} This component is a decision-making + mechanism which guides the operation of the IPP. It evaluates the + currently available collection of data, identifies the necessary + analysis, and assigns the analysis tasks to the IPP Controller. + +\end{enumerate} + +The relationship between these software units is shown in +Figure~\ref{overview}. This figure also shows the interactions +between the IPP and other Pan-STARRS systems. The architectural +components are discussed in detail in +Section~\ref{IPP:ArchComponents}. + +\begin{figure} +\begin{center} +\resizebox{6in}{!}{\includegraphics{pics/IPPoverview}} +\caption{ \label{overview} IPP System Overview} +\end{center} +\end{figure} + +\subsubsection{IPP Hardware Organization} + +\begin{figure} +\begin{center} +\resizebox{4.5in}{!}{\includegraphics{pics/IPPhardware}} +\caption{ \label{hardware} IPP Hardware Organization} +\end{center} +\end{figure} + +The IPP needs substantial computer resources, both in terms of +computational power and in terms of data storage and network +bandwidth. The IPP requires relatively large amounts of data storage +space, primarily for the image data. Image data is organized in two +categories. First, there is the per-OTA data -- data associated with +specific OTAs, including the raw images, the calibration images, and +temporary processed images at various stages. Second, there is the +data associated with the static sky imagery, which is in turn +organized into smaller sky-cell units. In addition to image data, +there are the somewhat smaller data entities of the Metadata Database +and AP Database. + +The computer hardware is organized into nodes which provide both data +storage and computational resources. The data storage nodes are +divided into three classes: those which deal with the per-OTA image +data, those that provide the storage for the static sky images, and +those that provide the storage for the other data systems, the +Metadata Database and the AP Database. In addition, the computational +tasks related to Phase 2 take place on the per-OTA storage nodes and +the Phase 4 computation takes place on the static sky storage nodes. + +Figure~\ref{hardware} shows our basic concept for the hardware +organization for the IPP. This diagram shows the two types of compute +nodes: OTA-level processing and storage nodes (dominated by Phase 2) +and static sky processing and storage nodes (mostly Phase 4). Also +shown are two switches which divide the network into OTA and +Static-Sky portions. In such an organization, the interswitch +communication must meet the throughput needs between these network +portions. The additional data systems (Metadata Database and AP +Database) are also shown. + +\subsubsection{Data Products} + +The IPP data products are listed in Table~\ref{SYS:IPPProducts} and +are discussed in detail in the sections below. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsection{Subsystem Tasks and Functions} +\label{IPP:Tasks} + +In order to achieve the top-level requirements listed above, the IPP +performs the following tasks: + +\begin{enumerate} + +\item Accept raw images from OTIS. + +\item Accept metadata from OTIS. + +\item Produce high-quality calibration images from the raw calibration + images. + +\item Pre-process the science images with the high-quality master + calibration images. This analysis is called ``Phase 2'' and the + resulting images are called ``Phase 2'' or P2 images. + +\item Perform an image-wide improvement to the astrometric and + photometric calibrations and the sky background subtraction (``Phase + 3''). + +\item Merge multiple pre-processed science images -- from multiple + telescopes or from sequential, dithered exposures -- into single, + cleaned, stacked images. This analysis, and the next task, + constitute ``Phase 4''. The resulting images are called ``Phase 4 + Summed'' or P4$\Sigma$ images. + +\item Subtract a static sky image from the cleaned, stacked images to + produce an image of only the transient events. The resulting images + are called ``Phase 4 Difference'' or P4$\Delta$ images. + +\item Excise the significant transients and outliers from the + pre-processed science images and merge the cleaned images into the + static sky image. + +\item Detect objects on the four types of images: pre-processed + images, the stacked image, the difference image, and the static sky + image. + +\item Determine astrometry of the detected objects relative to an + astrometric reference. + +\item Determine photometry of the detected objects relative to a + photometric reference. + +\item Provide the tools and data sources needed to construct a + high-quality astrometric reference catalog from the extracted + objects. + +\item Provide the tools and data sources needed to construct a + high-quality photometric reference catalog from the extracted + objects. + +\item Publish the static sky images to the Pan-STARRS PSPS after data +validation. + +\item Publish the detected objects to the Pan-STARRS PSPS after data +validation. + +\item Provide access to MOPS to the single-occurrence detections of +transient objects on short time scales. + +\item Provide access to MOPS to the metadata specific to the images in +which single-occurrence detections were found. + +\item Provide access to othe preferred Pan-STARRS science clients to + the measurements of the detected transient objects on short time + scales. + +\item Store the raw images for a period of time, depending on the + survey source of the data. During normal operations, raw data is + stored for \tbr{1 month}. + +\item Store the detected objects for a period of time, depending on + the type of detection. Transients from the P4$\Delta$ images may be + excised after \tbr{6 months}. + +\end{enumerate} + +\subsection{Operational Scenarios} + +In the normal operational state, the IPP continuously accepts data +from the summit (to the Image Server) and schedules new analysis tasks +for operation. At the start of a night, it will need to choose +between performing analysis on science images and analysis on +calibration images. It will also feed data to the MOPS and other +preferred science clients (such as the Transient Science Client) in +real time. For the MOPS, this data feed occurs at the end of each +night. Other clients may require the data to be delivered more +rapidly. The IPP has a top-level requirement to be capable of +delivering the transients to clients which need them within 5 minutes +of the image being taken. + +As a continuous process, the IPP will perform detailed object analysis +on the static sky images. The logical time to schedule this analysis +is to process those portions of the sky which are within $\sim 15$ +degrees of the sun (in RA), since these are regions which are +guaranteed to be unobserved and unchanging for at least 2 months. + +Some of the IPP computational and data resources will also be used to +construct the astrometric and photometric calibration catalogs. The +processing tasks involved in this effort will be assigned to the IPP +cluster along with the standard, nightly processing tasks. These +analysis tasks will be initiated by a human. + +On a longer term timescale (possibly once a month when the system is +running in an operational mode), some of the data products will be +pushed to the PSPS. In some cases (static sky images, most of the +metadata), this publication will involve the simple copying of the +data structures to an identical system on PSPS hardware. In other +cases (object detections), data will be sent to the PSPS which will be +processed by the PSPS for incorporation in PSPS databases. For the +static sky publication, it is again logical to send those portions of +the static sky which are within $\sim 15$ degrees of the sun in RA. + +The start-up process for the IPP as a whole may be viewed as a +start-up procedure for each of the architectural systems +independently. The three data storage systems (Image Server, AP +Database, and Metadata Database) need to check the existence and +validity of the hardware and data it manages. The IPP Controller +needs to determine which computers are available before allowing +processing. The IPP Scheduler will need to check that all of the +other systems are sufficiently operational before attempting to make +decisions and submit jobs to the IPP Controller. + +\subsection{Conceptual Design} + +\subsubsection{Architectural Components} +\label{IPP:ArchComponents} + +\paragraph{Image Server} +\label{IPP:ImageServer} + +The IPP Image Server is a large data store for all images used by the +IPP. The Image Server stores all of the images needed by the IPP for +the length of time they are required; total data volume is specified +in detail in the separate report, `The Pan-STARRS Image Processing +Pipeline Computational Challange' (PSDC-4xx-xx); the total data volume +for PS-4 is approximately 1000 TB. + +The IPP Image Server stores science and calibration images as FITS +files on disk Images of the Static Sky are stored as a variant of +FITS, using a data representation yet to be determined to minimize the +total data volume and pixel overlap and to minimize the losses from +warping of the images. The optimal representation of the Static Sky +is a topic for study by the IfA IPP Team. Two parameters which +critically affect the total data volume requirements of the Image +Server are 1) the pixel scale (arcseconds per pixel) and 2) the byte +representation of the data. Current working numbers for these are +0.2\arcsec and 4 bytes per pixel (2 for the signal and 2 for the noise +map). + +The IPP Image Server distributes images across a cluster of machines. +Multiple copies of each image may be requested for redundancy or for +improved throughput. The image (or one of the copies) may be placed +on a specified machine. The specific machine on which a particular +copy of an image resides may be determined via the user interface +methods. + +The IPP Image Server maintains a record of all images currently +available in the repository. The Image Server is only responsible for +tracking the location of the images, not for tracking metadata +information such as the image summary statistics or the state of the +image processing for the image. These aspects are included in the +Metadata Database discussed below. + +The IPP Image Server interfaces with other subsystems of the IPP. It +provides a mechanism by which other IPP subsystems may identify the +image location (the computer on which it resides). It also provides a +mechanism to serve a specified image to an IPP or Pan-STARRS subsystem +on request. It also provides maintainence mechanisms for deletion, +relocation, and duplication of images in the Image Server as +necessary. + +The IPP Image Server is not limited to image data. Any large file +would be an appropriate object to store in the Image Server. Raw +images from the telescope are stored as separate OTA images, with +multiple Cell images per file, as well as video sequences from the +guide stars in the form of MEF extensions. + +%% IPP Image Server T & F + +Image Server tasks and functions: + +\begin{itemize} + +\item The IPP Image Server stores images on a distributed collection + of computer disks. Individual instances of a file are only required + to be stored on a single machine (striping across computers is not a + requirement). + +\item The IPP Image Server attempts to store an image on a specific + machine if requested by the user. + +\item If such a request cannot be honored (ie, the machine is down), + the IPP Image Server selects an appropriate machine and notifies the + requesting agent of the new location. + +\item The IPP Image Server stores multiple copies of each image upon + request, the number of copies specified independently for each file + by the user. + +\item The IPP Image Server maintains a record of all image copies + currently available in the repository. This record includes at + least the image name, location (which machine), the image size, and + the state of the image (available, locked, + deleted). + +\item The IPP Image Server locks images in the repository on request. + Both read (shared) and write (exclusive) locks are provided. A read + lock prevents write access to the file; a write lock prevents both + read and write access. Access prevention may be advisory rather + than rigorously enforced. + +\item The IPP Image Server return the image location (the computer or + computers on which it resides) upon request. + +\item The IPP Image Server provides a specified image upon request. + +\item The IPP Image Server deletes images in the repository on + request. +\end{itemize} + +\paragraph{AP Database} +\label{IPP:APDB} + +The purpose of the AP Database is: +\begin{itemize} +\item to enable the photometric calibration of images +\item to enable the astrometric calibration of images +\item to enable the construction of flat-field correction frames +\item to enable the construction of a photometric calibration catalog +\item to enable the construction of an astrometric calibration catalog +\item to monitor the system photometry calibration parameters +\item to monitor the system astrometry calibration parameters +\item to perform the identification of single-occurance transients +\end{itemize} + +The AP (Astrometry \& Photometry) Database is a mechanism to store +data related to astronomical objects derived from various sources with +a variety of associations. The AP Database deals with two related +concepts: {\em objects} and {\em detections}. The objects are +descriptions of astronomical objects while the detections are the +specific measurements of those objects, typically measured from +astronomical images. A collection of {\em detections} may be used to +derive average quantities which describe a particular {\em object}. A +third class of object information which must also be considered are +those supplied by external references. These may be treated as {\em +detections}, with the caveat that access to the raw measurements and +metadata are usually unavailable; the reported measurements and errors +must be accepted as they are reported. + +The AP Database stores the collections of detections which were +derived from specific images from any of the analysis stages. It +provides a mechanism to determine and (in conjunction with the Image +Server) locate the image from which a specific detection was derived. +The AP Database also makes it possible to extract all detections +derived from a specific image and to determine quantities such as the +coordinates of the detection in pixel coordinates on the image. + +The AP Database also has the capability to associate multiple +detections of a specific object. Several major classes of objects +will be present, each of which must be handled correctly. + +First, the most distant stars, compact galaxies, and QSOs will have +nearly fixed locations relative to other nearby stars, with only small +deviations for individual measurements. The association between +multiple detections of such objects is made on the basis of their +coincident positions. The AP Database determines the average position +of the object and the deviations of the individual detections from +that average on the basis of the ensemble of individual detection. + +Second, solar system objects do not have a fixed location. Detections +of such objects are linked by their orbits, and depend on both the +position and the time of the image. The AP Database does not attempt +to make this link, which is the role of the MOPS system. However, it +has the ability to accept identifications made externally with +specified detections and to return the identifier of the moving object +associated with the specific detections. These associations also +include descriptive information such as the offset of the detection +from the predicted location of the detection based on the orbit. This +functionality is required to allow the AP Database to ignore known +moving object detections from other types of queries. + +Third, stars in the general vicinity of the solar system fall in +between these first two classes of objects. Their proper motion and +parallax response is significant enough ($>1$ arcsec in 1 year) that +they are not well-described by an average location and a collection of +offsets. These objects are described by a distance and a proper +motion vector. The AP Database provides the association between the +specific detections and an average object which includes finite +parallax and proper motion. + +Fourth, many detections, especially in their initial states, will not +be associated with a specific astronomical object of any of the above +classes and are treated as orphans. Most of these will be spurious +(not represent real objects), some will be from solar system objects +for which orbits are not yet determined, some will be from faint stars +near the detection limits, some will be from short-term transients +which have only been detected once. The AP Database maintains these +detections until they have been associated with one of the objects +above. The AP Database provides mechanisms by which individual +detections may be migrated back and forth between the orphan state and +association with an astronomical object. + +For every object, and all orphaned detections, the AP Database also +provides the capability to determine the images which observed the +location of the object but for which no detection was made. The +minimum set of information which must be carried for these +non-detections is the image and the associated object or orphan. + +The AP Database also stores the relationships between various +photometric systems and, in some cases, the evolution of that +relationship. It provides mechanisms to convert between the measured +instrumental magnitude of a detection with a specific filter, +detector, and telescope, and at a particular time and the implied +magnitude in the average Pan-STARRS photometry system, given a +determined set of calibrations. It also provides the capability to +convert magnitudes in one system to the magnitudes in another system; +an example of such a conversion is between the average Pan-STARRS +filter systems and the various reference systems appropriate for those +filters. + +The AP Database provides interfaces to extract lists of objects and +detections based on various query parameters. It provides the +capability to extract all detections associated with a specific +object, all non-detections of that object, all non-detections of an +orphan, and summary statistics from these collections. It will also +return all objects or detections within specified spatial regions +including regions bounded by great circles (RA,DEC; GLAT,GLON; +ELAT,ELON) and regions described by a location and a search radius. +It will also return the image parameters associated with a specific +detection including image coordinates of the detection, exposure time, +time and date of the detection, etc. + +\begin{table} +\begin{center} +\caption{AP Detection Classes \& Object Parameters\label{APdetections}} +\begin{tabular}{lrrrr} +\hline +\hline +Object Parameter & P2 & P4S & P4D & SS \\ +\hline +PSF x,y, covar, $\alpha,\delta$ & + & + & + & + \\ +PSF mag, $\sigma_{\rm mag}$ & + & + & + & + \\ +star/gal sep & + & + & + & + \\ +$\sigma_x$, $\sigma_y$, $\theta$ & + & + & + & + \\ +local sky data & + & + & + & + \\ +Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ +S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ +W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ +exp. spaced aps., Poisson noise, variance & - & - & - & + \\ +\hline +\end{tabular} +\end{center} +\end{table} + +The tasks and functions of the AP Database include: + +\begin{itemize} +\item The AP Database accepts and stores individual detections and + collections of detections along with information about the image + which provided the detections. + +\item Detections are saved as one of several detection classes (P2, + P4$\Sigma$, P4$\Delta$, SS) and the AP Database stores the + appropriate parameters, listed in Table~\ref{APdetections}, for each + class. + +\item The AP Database identifies the image which provided the + detection, or in the case of external references, an identifier + specific to the reference source. + +\item The AP Database groups detections into objects on the basis of + positional coincidence and measures average parameters of those + objects. + +\item The AP Database stores parallax and proper motion parameters for + a subset of the average objects. + +\item The AP Database stores image and filter calibration information + necessary to convert between instrumental magnitudes and calibrated + magnitudes in standard systems. + +\item The AP Database performs at least the follow queries, with + constraints on the output based on at least time ranges, magnitude + limits, error limits: + + \begin{itemize} + \item given $(RA,DEC)$ and a Radius, return all objects and/or + detections in the region. + + \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all objects and/or + detections in the region. + + \item given $(RA,DEC)$, return closest object. + + \item given object ID, return all detections. + + \item given detection, return source image data. + + \item given detection, return object. + + \item given $(RA,DEC)$, return all images overlapping coordinate. + + \item given $(RA,DEC)$ and a Radius, return all images overlapping region. + + \item given $(RA,DEC)_0$ to $(RA,DEC)_1$, return all images overlapping region. + + \item given detection instrumental magnitude, return derived + magnitudes based on calibration information. + + \item given a collection of detections in a filter, determine the + object average magnitude in that filter. + + \item given a collection of objects and detections, determine the + individual image zero-points. + + \item given a region, return all possible combinations of the object + or detection magnitudes $(M_1 - M_2)$. + + \item given a list of $(RA,DEC)$ entries, return all nearest objects. + + \item given a filter, telescope, or detector, return all calibration + terms and history. + + \item given a detection, return all non-detections from images which + overlapped the detection coordinates. + \end{itemize} + +\item The AP Database shall accept detection IDs of moving objects and + label the detections with the identified object. +\end{itemize} + +\paragraph{Metadata Database} +\label{IPP:MetadataDB} + +The IPP requires a Metadata Database to store and provide access to +metadata of various types and from various sources. Metadata in the +context of the IPP represents all data which is not included in the +two data stores discussed above (Images and Detection/Objects). +Metadata is generated at the telescope and during the various analysis +stages + +The Metadata Database stores and provides metadata for all raw images, +for processed images, for the calibration images (both raw and +master), for the extracted object lists. Metadata describing the +environmental conditions at the telescope must also be stored and +provided as needed. The Metadata Database stores descriptive +information about the raw images received from the summit and the +current state of the data processing. The Metadata Database also +stores descriptive information for each of the static sky images +currently available. + +If analysis results are exchanged via the Metadata Database, it must +provide access to the queried data on timescales of $<2$ seconds to +avoid slowing down the analysis systems. + +The IPP also requires a Configuration Database to store and provide +access to information about the IPP itself. Examples of data in the +configuration database include the default parameters for the various +analysis programs, the description of the computing environment, the +process status information, etc. Software configuration parameters +are stored in and extracted from the Metadata Database. +User-configurable software parameters are also stored in and extracted +from the Metadata Database. + +%% Metadata DB T & F + +The Metadata Database tasks and functions: + +\begin{itemize} +\item The Metadata Database stores the classes of data listed in + Table~\ref{metadata}. Thus, the Metadata Database stores and serves + metadata for all raw images, for processed images, for the + calibration images (both raw and master), for the extracted object + lists. Metadata describing the environmental conditions at the + telescope is also stored and provided as needed. + +\item The Metadata Database responds to simple queries which return + the data in the categories listed in Table~\ref{metadata} based on + the primary data key and with basic constraints of time ranges and + other simple conditional constraints. + +\item The Metadata database stores the configuration information with + restricted access so that only specific people may change the + information (eg, science parameters available to the science team; + software configuration parameters available to the system + maintainers). +\end{itemize} + +\paragraph{IPP Controller} + +The IPP uses a group of computers to store and process images and to +manipulate collections of detections. These computers perform any of +a large number of analysis stages or other processing tasks without +significant interprocess communication. It is necessary to have a +mechanism which initiates computing tasks on the different computers, +which monitors the tasks as they are executed, which handles the +output and the errors from these tasks, and which reacts to the +failure of any of the computing nodes. The system responsible for the +tasks in the IPP is the IPP Controller. + +The IPP Controller interacts with the collection of computers under +its management and with other subsystems in the IPP. The IPP +Controller receives a variety of inputs from other subsystems, +described below, and initiates actions such as adding a new process to +its queue. The IPP Controller also provides information to other +subsystems on demand about its processing history and current state. +Each physical computer may have multiple processors; since the IPP +Controller is managing processing tasks, it treats each processor +independently. It is up to the system configuration if each computer +needs to reserve one of its CPUs to manage background tasks or if the +IPP Controller should attempt to send one task per CPU and let the +kernel handle the I/O load. + +Computers managed by the IPP Controller are allowed to be in one of +several states, and the IPP Controller must interact with it in an +appropriate way for each of those states. A computer may be {\tt +alive}, {\tt dead} or {\tt off}. If the computer is {\tt alive}, it +responds to commands from the IPP Controller and may be used for tasks +subject to other constraints. If it is {\tt dead}, the computer is +not responsive and must not be used for executing tasks. The IPP +Controller must identify computers which have died and occasionally +test them to see if they are {\tt alive} again. Computers which are +{\tt off} are not available for tests and must not be tested. +Computers may be set to the {\tt off} or {\tt dead} states by external +subsystems; it is the responsibility of the IPP Controller to return a +computer to the {\tt alive} state if possible. An example scenario: a +computer crashes. At this point the IPP Controller should detect that +the computer is no longer responsive and mark it {\tt dead}. It +should occasionally try to re-establish communication with the +computer, potentially with longer and longer delays between attempts. +A human could be notified if the computer seems to remain {\tt dead} +for a very long time. In another circumstance, a person needs to work +on a computer. They should have the ability to notify the IPP +Controller that the machine is off, perhaps with a prior notification +that the machine should be prepared to go off. Only when the person +is done working and testing the machine, and tells the IPP Controller +that the machine is now {\tt dead} can the IPP Controller attempt to +re-start communications and processing on that computer. + +CPUs on computers which are in the {\tt alive} state may be in one of +two modes: {\tt busy} and {\tt free}. A CPU which is {\tt busy} +currently has a task assigned to it. The IPP Controller may only +assign one task to one CPU at a time. A CPU which is in the {\tt +free} state may have tasks assigned to it. The IPP Controller must +also respect a list of task restrictions which may require specific +tasks to run on specific CPUs or exclude specific tasks from specific +CPUs. + +The IPP Controller accepts tasks from other IPP subsystems. The task +requests include the specific command to be executed and are in the +form of a UNIX command which could be performed on any of the +computing nodes. Any input or output data structures in the commands +must be a valid resource regardless of the node on which the task is +executed. Input and output data resources must be unique where +necessary to avoid conflicts. The IPP Controller gives each task a +unique identifier, which is returned to the requesting agent. The +agent may then use that ID to obtain status information on that task +or to send control signals to the specific task. + +Task requests may specify a desired node for the task execution. The +IPP Controller attempts to honor the request if the node is {\tt +alive}, but will execute it on another node if the requested one is +{\tt dead} or {\tt off}. Even if a node is {\tt alive}, the IPP +Controller will choose another node if the specified task is not +allowed on the requested node. In all other cases, the IPP Controller +waits until the currently executing processes, and processes with +higher priority, are completed before executing the specified task on +the requested node. + +Task requests may specify an urgency level. The IPP Controller +determines the priority of the task on the basis of both the priority +and the age of the request. An executing task must be completed on a +CPU before any new task is started on that CPU, regardless of +priority. Tasks may be assigned a priority of 0 in which case they +are maintained in the queue and never executed. + +The IPP Controller monitors the output streams from the executing +tasks and the exit status of the tasks. Each task is associated with +a log file, to which all output is written. The status, including the +exit status, of each task is maintained by the IPP Controller so that +other subsystems may determine if specific tasks have started or +completed. + +The IPP Controller must accept commands from other IPP subsystems. +These commands include those which govern the processing of specified +tasks, those which govern the behavior of specific computing nodes, +and those which request information from the IPP Controller. The IPP +Controller must be able to halt the execution of a specified task, +delete an unexecuted task from the task list, change the priority of +tasks, and change the requested nodes for tasks. The IPP Controller +must also be able to stop the current execution of a task and push it +to the end of the queue and also change its priority. + +The IPP Controller must honor requests (normally from the users) to +change the mode of any computing node on demand between {\tt off} and +{\tt dead}. This would normally be done after a computer has been +rebooted and is release to the IPP Controller for its use. It must +also be able to change the list of allowed tasks as requested by +external commands. + +The IPP Controller must respond to informational requests regarding the +collection of machines and their states as well as the collection of +tasks and their states. The IPP Controller must monitor the execution +times of the different tasks and provide summary statistics. Finally, +the IPP Controller must respond to three top-level commands: {\tt finish}, +{\tt stop} and {\tt abort}. When {\tt finish} is requested, no more +new tasks are accepted on the stack of task, and when all tasks in the +stack have completed, the IPP Controller must exit. When {\tt stop} is +requested, the currently executing tasks must be completed at which +point the IPP Controller must exit, but tasks remaining in the stack which +have not been started are flushed. When {\tt abort} is issued, the +IPP Controller immediately kills all executing tasks and exits. + +The IPP Controller and the IPP Image Server have related needs for +information from the combined storage-and-processing nodes regarding +which nodes are available. It is not yet clear if this information is +best stored in a single location (either IPP Controller or IPP Image +Server), which provides the information to other systems on demand, or +if both systems should maintain the information. Also, it may be +necessary to distinguish nodes which are available for processing from +those that are available to serve data as part of the IPP Image +Server. + +It may be useful for the Controller to distinguish between tasks +dominated by I/O and tasks dominated by data processing. It is +possible that one of each of these types of tasks may be sent to the +same node without significantly impacting the system performance. +Alternatively, it may be necessary to limit a single machine with 2 +CPUs to only one of each of these types of tasks (i.e., one processor +will be working on I/O while the other is working on processing). +Such details will be studied by the IfA IPP Team. + +%% IPP Controller T & F + + IPP Controller tasks and functions: + +\begin{itemize} + +\item On startup, the IPP Controller attempts to establish + communication with all of its computers and set their state to be + {\tt alive} or {\tt dead} based on the success of the + connection. + +\item The IPP Controller detects computers which crash or stop + responding and set their state to {\tt dead}. + +\item The IPP Controller attempts to re-establish communication with + {\tt dead} computers. + +\item The IPP Controller accepts tasks from external users and + systems, which may specify a desired CPU (node) and priority in + addition to the task command. + +\item The IPP Controller attempts to run pending tasks on the desired + node, if available (not {\tt dead} or {\tt off}). + +\item If the node is unavailable, the IPP Controller attempts to run + the task on another node. + +\item If the node is available, the IPP Controller attempts to run a + given task only if no higher-priority tasks are available and no + task is currently being executed. + +\item The IPP Controller monitors the output from the task and writes + it to an associated log destination. + +\item The IPP Controller monitors the execution status of each task + currently executing on a node and performs the following actions: + + \begin{itemize} + \item identify the task as successful if it has a valid exit status. + \item identify the task as unsuccessful if it has an error exit status. + \item identify the task as unattempted if the computer crashed. + \end{itemize} + +\item The IPP Controller accepts and performs the following external + commands: + \begin{itemize} + \item add a task to the pending task list. + \item delete a specific task from the pending task list. + \item return the current status of a specific task. + \item return a list of all pending and non-pending tasks. + \item set a specified computer state to {\tt off} or {\tt dead}. + \item restrict a specified CPU to a class of tasks. + \item halt execution of a specified task. + \item set the IPP Controller state to {\tt finish}, {\tt abort}, or {\tt stop}. + \end{itemize} +\end{itemize} + +\paragraph{IPP Scheduler} +\label{IPP:IPPscheduler} + +The IPP is responsible for a variety of analysis tasks: processing of +the science images through several stages; routine assessment of the +detrend (instrumental calibration) images used in processing the +science images; construction of replacement detrend images when +needed; generation of astrometric and photometric reference catalogs +based on the collected dataset; and the performance of test analysis +programs. At any point, decisions need to be made about which of +these tasks should be performed, based on an analysis of the contents +of the metadata database, the requirements of the people monitoring +the IPP, and the near-term observing plans. The IPP Scheduler is the +mechanism that assesses these various inputs to guide the decisions +and initiate the actions. + +The IPP Scheduler acts as an intermediary between several components +of the IPP and also between the IPP and external agents such as OTIS +and the users who must monitor the behavior of the IPP. + +The IPP Scheduler sends commands to the IPP Controller for execution. +While the IPP Scheduler chooses the tasks to be performed, it is the +IPP Controller's responsibility to manage the specific tasks executing +on a given processing node. Examples of these tasks include the +process of copying or moving data from the Summit data systems to the +IPP Image Server; image processing analysis stages performed on the +science images by the appropriate processing nodes; and the analysis +of the data in the AP Database. This division of responsibilites +allows us to isolate and encapsulate the functionality of the IPP +Scheduler and the IPP Controller. With this separation, the IPP +Controller does not need to have any information about the details of +the tasks which it executes, while the IPP Scheduler does not need to +have detailed information about the available computer hardware. + +Communication between the IPP Scheduler and the IPP Controller is +bi-directional; the IPP Scheduler sends tasks to the IPP Controller, +while the IPP Controller informs the IPP Scheduler of the outcome of +those tasks. It is not specified whether the IPP Scheduler and IPP +Controller are components of a single software system or interacting +but distinct software components. + +The IPP Scheduler takes as input the current list of pending images, +both science and calibration, and a description of the current +observing plan or strategy on some time-scale. The IPP Scheduler also +takes input from humans who manage the IPP. + +The IPP Scheduler must choose between several types of analysis tasks +based on the contents of those lists and on the requirements of the +users. The list of tasks which the IPP Scheduler must decide between +includes: +\begin{itemize} +\item moving data from the Summit pixel server ($\sim 30$ second timescales) +\item running the science analysis stages ($\sim 30$ second timescales) +\item testing the validity of the current detrend images ($\sim$ + nightly) +\item constructing new detrend images ($\sim$ weekly) +\item updating and improving the photometric and astrometric reference + catalogs ($\sim$ yearly). +\end{itemize} + +The IPP Scheduler chooses between tasks which are relevant on several +different time-scales. The time-scales range from 2 times per minute +to once or twice a year, as noted in the list above. The IPP +Scheduler must also make use of user input in managing such choices. +Users have the option to specify that a particular task or set of +tasks is of higher or lower priority than the norm. + +The IPP Scheduler maintains a set of rules that define the dependency +of one type of analysis stage on other analysis products. For +example, the nightly science image processing depends on the existence +of valid detrend images. The IPP Scheduler must be able to recognize +the dependency and initiate the required analysis needed to perform +other analysis tasks. The IPP Scheduler must have the ability to +decide between postponing an analysis task until the required data are +available or initiating the task using a lower-quality or less +appropriate substitute. For example, in normal circumstances, a +science image must not be processed until the corresponding detrend +frame has been produced. However, if such a frame is unlikely to +appear soon, and the pressure to process the science image is +sufficiently high, then the frame could be processed with an older +detrend frame of known lower quality. The IPP Scheduler must have the +ability to choose the best, if not ideal, reference data for a +particular circumstance. Note that these rules are defined for the +IPP Scheduler in an abstract way as a relationship between analysis +stages, rather than as a specific rule relating one task to another +task. + +The IPP Scheduler defines the operating state of the IPP. When the +IPP is in the {\em automatic state}, the IPP Scheduler performs the +most appropriate of all possible tasks at a particular time. When the +IPP is in the {\em interactive state}, the IPP Scheduler performs only +the requested action regardless of the outcome of the decision trees. +In addition, in the interactive state, the IPP Scheduler must only +perform the requested actions and not attempt to perform the other +normally-required actions. The only exception to this exclusion is +that, in the interactive state, data is still copied from the summit +system. An additional IPP state is the {\em paused state}, intended +for tests or maintenance, in which case the IPP Scheduler does not +perform even the data copy tasks. Every task is performed on demand +by the user. The user command sets the IPP Scheduler in one of these +three states, {\em automatic}, {\em interactive}, and {\em paused}. + +%% IPP Scheduler T & F + +The IPP Scheduler tasks and functions: + +\begin{itemize} +\item The IPP Scheduler sends the analysis tasks which it initiates to + the IPP Controller. + +\item All analysis tasks sent by the IPP Scheduler include a complete + UNIX command with necessary arguments, the priority of the task, and + optionally the desired processing node. + +\item When the IPP Scheduler is placed in the {\em paused state}, it + only initiates User-requested tasks. + +\item When the IPP Scheduler is placed in the {\em interactive state}, + it initiates User-requested tasks as well as data transfer tasks. + +\item When the IPP Scheduler is placed in the {\em automatic state}, + it initiates the most appropriate task based on the inputs and + dependency rules. + +\item The IPP Scheduler sends the exit status of the analysis tasks to + the appropriate destination as defined by the task dependency table. +\end{itemize} + +\subsubsection{Analysis Stages} +\label{IPP:AnalysisStages} + +\begin{figure} +\begin{center} +\resizebox{6.4in}{!}{\includegraphics{pics/IPPstages}} +\caption{ \label{stages} IPP Analysis Stages} +\end{center} +\end{figure} + +\paragraph{Overview} + +We now consider the collection of analysis tasks which must be +performed by the IPP. These tasks represent the core of the required +IPP functionality; the architectural components discussed above can be +viewed as primarily supporting infrastructure to enable the analysis +tasks to be executed on the appropriate data and to store the results. +The tasks are divided into well-defined analysis stages, each of which +operates on a particular unit of data, such as a single OTA image or a +collection of astronomical objets. + +Depending on the analysis stage, the basic data unit may be individual +images, collections of images, or derived data products such as a +collection of detections of astronomical objects. Because of the +granularity of these data units, a large number of analysis tasks +representing the same analysis stage many be performed in parallel. +This is particularly true because the analysis tasks from any +particular stage do not depend on the results of another analysis task +from that stage. For example, the intial analysis of a chip from one +image does not depend on the results from another chip. The analysis +stages are divided into three categories, and further subdivided as +follows: + +\begin{enumerate} + \item {\bf Science Image Analysis} is performed on the night-sky + science images to extract the science data from these images. The + science image analysis is divided into 4 analysis stages, which we + call Phases 1 - 4: + + \begin{itemize} + \item {\bf Phase 1:} The image processing preparation stage, in + which basic astrometric analysis of the complete FPA image is + performed. + + \item {\bf Phase 2:} The image reduction stage, in which the + individual detector images (OTAs) are processed as much as possible + without reference to other chips in the same FPA image or other + exposures. + + \item {\bf Phase 3:} The exposure analysis stage, in which the + results of the multiple detectors are combined to improve the + calibrations for the complete FPA images. + + \item {\bf Phase 4:} The image combination stage, in which several + different exposures of the same part of the sky are combined to + produce high-quality difference and summed images. + \end{itemize} + + \item {\bf Calibration Image Analysis} is required to generate the + calibration images used in the science image analysis. There are + many different types of calibration images which must be produced, + some of which are derived from other calibration images. The type of + calibration image is generated by its own analysis stage. These + include the construction of simple bias, dark, and flat-field stacked + images, the generation of flat-field correction frames on the basis + of stellar photometry, and the construction of sky-model and + fringe-model images. + + \item {\bf Reference Catalog Creation} is required by the IPP to + generate improved astrometric and photometric reference catalogs on + the basis of Pan-STARRS observations. + +\end{enumerate} + +Figure~\ref{stages} shows the flow of data between the various IPP +software subsystems and the different analysis stages. The thick +lines represent the flow of pixel data, the thin lines represent the +flow of metadata and object data, and the grey lines represent the +flow of commands. The hatched systems represent external Pan-STARRS +systems (OTIS, the Sky Server, the PSPS Object Database, the +Moving/Transient Object Pipeline, and other Client Science Pipelines. + +The individual analysis stages are implemented as UNIX command-line +programs. Each command performs the action of the stage on a single +quantum of data. These analysis stages are built of lower-level +C-functions which may be wrapped in a higher-level programming +language such as Perl or Python. + +As discussed above (section~\ref{IPP:IPPscheduler}), the decision to +execute a specific analysis stage for a specific dataset is made by +the IPP Scheduler, which sends the infomation to the IPP Controller. +The IPP Controller executes the analysis tasks constructed by the IPP +Scheduler for that stage on an appropriate machine and monitors the +success or failure of the job. + +An important design decision which must be addressed by the IfA IPP +Team is the question of how the analysis stages determine +configuration and reference data needed by the analysis. In one +scenario, it is always the responsibility of the low-level function to +perform the necessary query of the reference databases. In an +alternative scenario, it is the responsibility of the scheduler to +extract that information and send it as part of the processing +command. + +\paragraph{Science Image Analysis} + +The Science Image analysis stages together represent the primary data +analysis performed by the IPP. The science image analysis which is +performed by the IPP consists of: + +\begin{itemize} +\item detrending the images to remove the instrumental signature + +\item astrometric and photometric calibration of the individual images + +\item merging a collection of several images of the same portion of +the sky obtained over a short period of time to remove image defects +and gaps + +\item subtracting the appropriate reference static-sky image + +\item cleaning the image of any transients + +\item adding the cleaned image to the static sky + +\item object detection of images at specific stages +\end{itemize} + +We have divided the analysis steps into four analysis stages, which we +call Phases 1 - 4. Each of these analysis stages deals with a single +data unit. + +\paragraph{Phase 1 : image processing preparation} + +The Phase 1 analysis stage is performed on each science exposure (each +complete FPA image) to calculate basic astrometric data needed by the +later stages. Phase 1 uses the static (pre-determined) telescope +distortion model and a table of nominal OTA positions and rotations, +combined with the guide star pixel and celestial coordinates, to +determine the correct telescope bore-sight, field rotation and +magnification. The guide star coordinates are loaded from the +Metadata database. These calculations are performed by comparing the +observed guide star detector coodinates with the known astrometic +positions of these same stars as reported by an external astrometric +reference. The accuracy of the resulting astrometric solution is +expected to be \tbr{1 arcsec} across the field, sufficient in later +stages to match the vast majority of astrometric reference stars with +their detections with minimal effort. + +In some circumstances, science images may have no guide stars. This +may occur in the Pan-STARRS system if the detectors are not run in OTA +mode, for example for short snapshot images. This may also be the +case if the IPP is being run on non-Pan-STARRS data. In such a +circumstance, the Phase 1 stage uses the provided boresight +coordinates, exposure time, and camera zero-point to predict the pixel +coordinates of known bright stars expected to be found on the +detectors. It then extracts a large box ($\sim$ 30 $\times$ +30\arcsec) around these locations and performs extremely basic object +detection to determine the detector coordinates of those bright stars +which are not saturated but which are significantly above the +background level. By targetting known locations in the image files, +only a small amount of data will have to be read. + +If the image has invalid coordinates or no detectable bright stars, +Phase 1 fails and reports a descriptive error. + +Given the above astrometric solution, the Phase 1 analysis stage +constructs a table of the overlaps between the science image to be +processed and the static sky images that must be constructed. This +table will be used to guide the processing of the static sky in Phase +4. The overlaps should be generously calculated so that small errors +in astrometry at Phase 1 will not cause any valid static sky / science +image pairs to be missed because of the astrometric error at this +phase. It is acceptable for a small number of invalid overlaps to be +identified as these will be excluded in Phase 4. Static Sky cells +which do not have sufficient science image overlap \tbr{$< 5\%$} need +not be processed because the few new measured pixels do not add +significantly to the Static Sky. + +Phase 1 is the image processing preparation stage. The analysis is +performed on a complete FPA. At the end of this analysis, the FPA is +ready to be analysed in detail in Phase 2. The Phase 1 tasks and +functions are: + +\begin{itemize} + +\item Extract FPA guide stars to determine astrometry across the full FPA + +\item If no guide stars are available, phase 1 must measure the pixel + coordinates of known bright stars expected in the field from the + image data. + +\item The total number of stars and size of the bright-star + acquisition box shall be a user-configurable parameter in the range + 20 - 250. + +\item Calculate the Image cell / Sky cell overlaps for each image. + Sky cells which do not have sufficient science image overlap $< 5\%$ + are excluded from the overlap table. + +\end{itemize} + +\paragraph{Phase 2 : image reduction} +\label{IPP:Phase2} + +The Phase~2 analysis is the detrend stage, in which the images from +the detector are processed to remove instrumental signatures. In +addition, basic object detection is performed along with improved +astrometric and photometric calibration. In each step of the analysis +process, an image mask and noise map must be constructed and updated +when appropriate. The following operations may be applied during the +Phase~2 processing: + +\begin{enumerate} +\item Convolve detrend images with the OT kernel, if appropriate +\item Flag bad and saturated pixels +\item Bias correction via overscan subtraction +\item Dark +\item Trim object image to remove overscan and edges corrupted by OT +\item Correct for non-linearity +\item Cross-talk +\item Flat-field correction +\item Sky \& Fringe subtraction +\item Identify CRs +\item Find objects in the image +\item Model the PSF variations across the image +\item Make postage stamps of bright objects. +\end{enumerate} + +Of the calibration steps, some may be skipped if they do not +contribute to an improved image. The decision to apply or skip a +particular step is determined by the Phase 2 recipe, which may specify +exposure time or flux limit cutoffs for some of the steps. + +\subparagraph{Convolve detrend images with the OT kernel} + +Certain detrend images are convolved by the OT kernel, so that they +accurately represent the detrend images appropriate for the object +images, which have been shifted using OT. The detrend images which +must be convolved include: the flat-field and the +high-spatial-frequency fringe images. The appropriate kernel for each +cell of an OTA must be determined from the guide star history, +extracted from the IPP Metadata Database\footnote{or image header}. +If the OT kernel is not available, the convolution is skipped, with a +warning. + +\subparagraph{Flag bad and saturated pixels} + +A static bad pixel mask is used to identify pixels which are known to +be bad in the camera. This mask is then processed with the science +image. Bad pixels which are charge traps are grown by the extent of +the OT convolution kernel. Bad pixels above a charge trap (i.e.\ bad +colums) must not be grown, since they were not affected by pixel +shifting, but only became bad at read-out. + +Pixels which are saturated in the A/D converter, or with a signal +level at which the response is excessively non-linear, must also be +masked, and this area must be grown by an additional pixel to mask +excess charge spillover. + +The bad pixel mask must be carried with the science images. Different +bits must be set to identify different reasons for masking the pixel. + +\subparagraph{Bias correction via overscan subtraction} + +The image bias must be subtracted. Since different detectors behave in +different ways, several options for modelling the bias are available. +The bias is measured from the image overscan region. The bias +subtraction method must be capable of subtracting a single constant +from the complete image, or to subtract a 1-D bias which varies as a +function along the overscan. The function used to represent the +overscan region may be a spline or a chebychev polynomial derived from +the data values along the overscan. The values used to determine both +the single constant or the inputs to the spline and polynomial fits +are derived from groups of pixels on the basis of one of several +statistics, including the sample and robust mean, median, and modes. +In the case of a single constant, all of the overscan pixel values are +used in the calculation of this statistic. In the case of the 1-D +functional representation, the input values to the fit must represent +the coordinate along the overscan, with the statistic derived from the +pixels in the perpendicular direction at each location. Sigma-clipping +on the input data values must be an option. + +\subparagraph{Trim object image} + +The image is trimmed to remove the non-imaging pixels, such as the +overscan and any pre-scan pixels, along with those pixels near the +edges that have been compromised due to OT operation. The definition +of the imaging area of the detector is determined from the camera +configuration data or from the metadata associated with the image, +with the choice a user-configurable option. + +\subparagraph{Correct for non-linearity} + +If required, the object image (after bias correction) must be +corrected for the effects of non-linearity through a provided +polynomial fit to the pixel data values or a numeric lookup table as a +function of pixel flux. The choice to apply the correction must be +set by the user. + +\subparagraph{Flat-field correction} + +The object image (after bias correction and non-linearity correction) +must be corrected for sensitivity variations as a function of +position, dividing by a flat-field image. + +\subparagraph{Sky \& Fringe subtraction} + +After the science image has been flat-fielded, the flux contribution +of the sky (from both continuum emission and the line emission that +causes fringing) must be subtracted from the image. The subtraction +needs to remove background (technically, foreground) variations which +are not celestial but generated in the atmosphere or by more localized +scattering. This background should include the contribution from the +zodiacal light. This background subtraction does not address the +artifacts generated by bright stars: bleeding columns, ghosts, or +other localized reflection effects. This process also produces a +superbinned image of the background map which may be used as a +debugging diagnostic. + +\subparagraph{Identify `cosmic rays'} + +Charged particles in the detector frequently cause features which do +not have the morphology of astronomical objects. In a well-sampled +image, these may be easily identified by the sharpness of the image. +In a near critically-sampled image, these `cosmic rays' may be +indistinguishable from stellar objects. The IPP must have the +capability of making the morphological identification of cosmic rays +if the imaging data is sufficiently well sampled. The identified +cosmic rays should be masked with a configurable growth factor so that +additional pixels beyond the detected pixels in the feature are also +masked. + +\subparagraph{Find objects in the image} + +After the image have been processed by the preceeding steps, the Phase +2 analysis performs a basic object detection analysis. Objects on the +flat-fielded object image are found, and general parameters are +measured. Object detection is performed at several stages by the IPP, +with different object parameters measured in each case. +Table~\ref{APdetections} gives a list of the different detection +stages and the object parameters measured for those stages. For the +Phase 2 analysis, the object parameters are: the object centroid and +the position covarience matrix, the instrumental PSF magnitude and +error, local background level and error, a measurement of the +star-galaxy separation, and a measurement of the object shape +($\sigma_x, \sigma_y, \theta$). The detection threshold must be +configurable, and be a function of the average background flux or the +image noise map. Minimal object classification must be performed to +distinguish objects which are consistent with a single PSF, objects +which are inconsistent, and objects which are saturated. The +resulting collection of detected objects are saved in the AP Database +along with the relevant image metadata (\ie filter, exposure time, +etc). In addition, this process constructs a model of the +point-spread-function (PSF) as a function of position in the image. +This PSF model is saved as part of the image metadata. + +\subparagraph{Astrometry} + +The astrometric parameters determined in Phase 1 have an accuracy of 1 +arcsec, sufficient to allow easy association between the newly +detected objects and many reference objects in the image. In Phase 2, +the detected objects are matched with known astrometric reference +objects, using reference object coordinates which have been adjusted +for proper motion. The matches are then used to improve the +astrometric parameters for the individual OTAs. The OTA astrometric +parameters which are determined may include terms up to 3rd order in +position, though the terms which are actually fitted are +user-configurable. The Cell astrometric parameters are not allowed to +vary at this stage. The fit must be robust, rejecting outlier matches +(either stars with poorly determined proper motion or spurious +matches). The resulting astrometric solution is consistent across the +OTA field to within \tbr{0.2 arcsec}. + +\subparagraph{Postage Stamps} + +The IPP must have the capability of extracting regions surrounding a +specified subset of objects from the flattened images. These postage +stamp images must be saved for additional use by client science +pipelines. The identification of these objects must be on the basis +of a set of rules applied to the object's magnitude and position. The +postage stamps are not restricted in shape to simple rectangles, but +may represent more complex regions. They are written the Image +Server. + +%% + +Phase 2 is the detrend stage, in which each detector is separately +processed to remove instrumental signatures. The result of Phase 2 is +an image with high-quality astrometric and photometric calibrations, a +collection of objects detected in the image and characterized in a +rudimentary way (star / non-stellar), and a measurement of the PSF +across the detector. + +The tasks and functions of Phase 2 are as follows: + +\begin{itemize} + +\item Convolve the flat-field and high-spatial-frequency fringe images + with the OT kernel. + +\item Mask ghosts of bright stars which introduce residual feature + more significant than 1\% of the background. + +\item Bias subtract the image. + +\item Correct each chip independently for non-linearity. + +\item Flat-field correct the image. + +\item Subtract a fit to the detector-dependent fringing pattern. + +\item Subtract a fit to the low-spatial frequency sky background. + +\item Identify `cosmic rays' on the basis of morphology. + +\item Perform (positive) object detection on the processed images, + down to a user-configured threshold, likely to be $\sim 20\sigma$. + The detection threshold may optionally be a function of the average + background flux or the local noise level. + +\item Measure the following object parameters: + + \begin{itemize} + \item object centroid and position errors. + \item an extended object position ($x_g, y_g$). + \item instrumental PSF magnitude and error. + \item local background level and error. + \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) of the object + and their covariance matrix. + \end{itemize} + +\item Perform minimal object classification to distinguish objects + which are consistent with a single PSF, objects which are + inconsistently large, objects which are inconsistently small, and + objects which are saturated. + +\item Match the detected objects with known astrometric reference + objects, including proper-motion compensation. + +\item Fit the reference and detected object coordinates to determine + astrometric parameters for the individual OTAs, including + polynomials of the coordinates up to 3rd order (user-specified + parameter). The Cell astrometric parameters are not allowed to vary + in the fit, which uses outlier rejection to determine a robust + solution. + +\item Extract subrasters (`postage stamps') surrounding a + user-specified list of coordinates from the flattened + images, to be saved in the IPP Image Server. + +\item measure the PSF variation as a function of detector position. + +\end{itemize} + +\paragraph{Phase 3 : exposure analysis} + +The Phase 3 analysis stage works with the results from a complete FPA +obtained during Phase 2 to improve the photometric and astrometric +calibrations. + +Phase 3 uses the objects detected in Phase 2, matched with an +appropriate reference catalog, to determine the image photometric zero +point and zero-point variations across the field. If zero-point +variations are significant, the zero-point variations are modeled with +a Chebychev polynomial correction of order 3 or less. The complete +FPA image must be categorized as photometric or not on the basis of +the zero-point consistency, comparisons between the zero-point of the +image and recent longer-term (week or month long) measurements of the +zero-point, and the external indicators of photometricity. In +addition, statistics of the transparency are measured and saved as +part of the related Metadata. + +Phase 3 also uses the objects detected in Phase 2, matched with an +appropriate reference catalog, to determine improvements to the +astrometric solutions. The improved solution is determined by fitting +a new distortion model appropriate to this image. The resulting +astrometric accuracy is limited by the astrometric reference +catalog. (see Table~\ref{AstroRefs} below). + +Phase 3 also uses the individual measurements of the background and +the superbinned background maps to generate an improved background map +over the entire FPA. The large-scale background correction is +determined on the basis that the background should be smoothly varying +between different chips (OTAs). + +Phase 3 also uses the individual chip models of the PSF variations to +model the global PSF variations across the field. There will be +discontinuities at the chip boundaries due to charge diffusion and +chip displacements along the optical axis, but there will also be an +overlying trend due to the local coherence of atmospheric seeing +variations. + +%%% + +The Phase 3 analysis uses the objects detected in Phase 2 and external +reference catalogs to determine improved photometric and astrometric +calibrations for the FPA as a whole, and to improve the measurement of +the PSF and sky variations across the field. The Phase 3 tasks and +functions are as follows: + +\begin{itemize} + +\item Phase 3 uses the objects detected in Phase 2, matched with a + user-specified reference photometry catalog, to determine the image + photometric zero point and zero-point variations across the field. + +\item If zero-point variations are significant ($> 0.01$ mag + peak-to-peak), the zero-point variations are modeled with a + polynomial correction of order 3 or less. + +\item The photometric nature of the FPA image is categorized on the + basis of the zero-point consistency, the transparency compared with + recent long-term measurements in the filter, and the external + indicators of photometricity. + +\item Phase 3 uses the objects detected in Phase 2, matched with an + appropriate astrometric reference catalog, to improve the distortion + model used for the image. The resulting astrometric accuracy is + consistent across the field to 30 mas, and is limited by the + astrometric reference catalog, (eg, 100 - 250 mas for + USNO-B1.0). + +\item The Phase 3 analysis modifies the background correction of Phase + 2 based on the full-field statistics to achieve an accuracy of 1\% + of the background. + +\end{itemize} + +\paragraph{Phase 4 : image combination} +\label{IPP:Phase4} + +Phase 4 is the image combination stage, in which multiple images of +the same portion of the sky are merged and confronted with the static +sky image. Phase 4 operates on the smallest data unit of the static +sky, the sky cell, along with the associated pixels from a collection +of images which have been processed through phases 1--3. The size and +exact representation of a static sky cell are yet to be determined. +The working concept is that the static sky cells contain roughly the +same number of pixels as an OTA (4k x 4k) and represent a portion of a +local tangent plane projection. As mentioned above +(Section~\ref{IPP:ImageServer}), the pixel scale of the static sky is +planned to be 0.2\arcsec, somewhat smaller than the 0.3\arcsec\ raw +image pixel scale. + +For each sky cell, the corresponding pixels are extracted from the +exposures being processed and mapped to the projection of the sky +cell. The pixels from the multiple input processed images are combined +into a single, cleaned image. Outlier pixels may be optionally +rejected at this stage (optionally, since moving objects will be +rejected in images obtained over a wide range of times). This image +is then confronted with the static sky cell data to produce a +difference image. Residual objects in the difference image above a +threshold are detected and excised from the original cleaned image. +The remaining pixels are added to the existing static sky image. +Object detection must be performed on the difference and cleaned +images. + +Objects in the difference image are detected and a specific set of +object parameters are measured from these detections. +Table~\ref{APdetections} gives a list of the different detection +stages and the object parameters measured for those stages. For the +Phase 4 difference image (P4$\Delta$), the measured object parameters +consist of: the object centroid and the position covarience matrix, +the instrumental PSF magnitude and error, local background level and +error, a measurement of the star-galaxy separation, and a measurement +of the object shape ($\sigma_x, \sigma_y, \theta$). The detection +threshold must be configurable, and be a function of the average +background flux or the image noise map. Minimal object classification +must be performed to distinguish objects which are consistent with a +single PSF, objects which are inconsistent, and objects which are +saturated. The resulting collection of detected objects are saved +along with the relevant image metadata (\ie filter, exposure time, +etc). + +Objects in the cleaned, summed image are detected and a specific set +of object parameters are measured from these detections. +Table~\ref{APdetections} gives a list of the different detection +stages and the object parameters measured for those stages. For the +Phase 4 summed image (P4$\Sigma$), the measured object parameters +consist of: the object centroid and the position covarience matrix, +the instrumental PSF magnitude and error, local background level and +error, a measurement of the star-galaxy separation, a measurement of +the object shape ($\sigma_x, \sigma_y, \theta$), the Petrosian radius, +magnitude, axis ratio, and angle; and the S\'ersic radius, magnitude, axis +ratio, angle, and parameter $\nu$. The detection threshold must be +configurable, and be a function of the average background flux or the +image noise map. Minimal object classification must be performed to +distinguish objects which are consistent with a single PSF, objects +which are inconsistent, and objects which are saturated. The +resulting collection of detected objects are saved along with the +relevant image metadata (\ie filter, exposure time, etc). In this +measurement, objects at known positions will also be measured even if +they have not been detected. + +Objects which are detected in both of the Phase 4$\Sigma$ and Phase +4$\Delta$ images are saved to the AP Database, along with the relevant +image metadata (\ie filter, exposure time, etc). In the process of +adding these objects to the database, the transients which are +correlated with previous detections of an object (and those which are +not) will automatically be determined. An independent process will +query the AP Database for such transient objects of interest which are +to be sent, along with their associated metadata, to the MOPS and +other science client pipelines. This step must be performed at least +once per night. + +It is essential that the static sky image (which may have been +painstakingly accumulated over many months) not be corrupted by adding +in transient sources, or data that is of suspect quality (due, e.g., +to an error upstream in the processing). + +Object analysis of the static sky images is {\em not} a part of the +Phase 4 analysis. This processing is envisioned to take place +relatively infrequently (perhaps weekly or even monthly) and is +scheduled as a separate analysis task, probably run during the day at +a time when the computing infrastructure is not under significant load. + +%% + +Phase 4 is the image combination stage, in which multiple images of +the same portion of the sky are merged and confronted with the static +sky image. The Phase 4 tasks and functions are as follows: + +\begin{itemize} + +\item The Phase 4 analysis determines the corresponding set of image + pixels for a given sky cell. + +\item These pixels are extracted from the input images, using the + astrometric information for each OTA and Cell to determine the exact + overlaps. + +\item The Phase 4 analysis skips any sky cells with fewer than 5\% of + their pixels overlapping the input images. + +\item Pixels which have been extracted from the input images are + geometrically warped to match the corresponding pixels in the sky + image. This transformation is based on the measured astrometric + solution for the input images relative to the reference catalog used + to generate the static sky image. The warping may use a + locally-linear astrometric solution to speed the processing. + +\item Phase 4 determines the appropriate photometry scaling factors + needed to combine the images photometrically. + +\item When multiple images are combined, the group of input pixels + which contribute to an output pixel are examined and pixels from the + group of images which are inconsistent with the ensemble (by an + amount defined by the user-configurable parameters) are identified + and flagged, though this outlier rejection shall be performed + optionally. + +\item The resulting collection of pixels is used to construct a single + output image, cleaned of the outliers. + +\item The cleaned, combined image is PSF matched with the static sky + image. + +\item The static sky image is subtracted from the stacked, cleaned + image, resulting in the difference image (P4$\Delta$ image) + +\item The Phase 4 analysis performs object detection on the difference + images. All objects in the difference image above a user-configured + signficance threshold are detected, including both positive and + negative flux objects. The detection threshold may optionally be a + function of the average background flux or the local noise + level. The likely significance threshold is $\sim 3\sigma$. + +\item P4$\Delta$ objects have the following object parameters + measured: + \begin{itemize} + \item object centroid and position errors. + \item instrumental PSF magnitude and error. + \item local background level and error. + \item streak L, $\phi$, $\sigma_L$, $\sigma_\phi$. + \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their covariance matrix. + \end{itemize} + +\item Minimal object classification is performed to distinguish + objects which are consistent with a single PSF, objects which are + inconsistent, and objects which are saturated. + +\item The pixels belonging to variable sources are masked in the + input image. + +\item A new, cleaned image is constructed from the masked input images + (P4$\Sigma$ image) + +\item Object detection is performed on the cleaned, summed image to a + user-configured significance threshold ($\sim 7\sigma$). Only + positive flux object are considered. The detection threshold may + optionally be a function of the average background flux or the local + noise level. + +\item P4$\Sigma$ objects have the following object parameters + measured: + \begin{itemize} + \item object centroid and position errors. + \item an extended object position ($x_g, y_g$). + \item instrumental PSF magnitude and error. + \item local background level and error. + \item second moments ($\sigma_{\rm min}, \sigma_{maj}$) and their + covariance matrix. + \item the Petrosian radius, magnitude, axis ratio, and angle. + \item the S\'ersic radius, magnitude, axis ratio, angle, and parameter $\nu$. + \end{itemize} + +\item Minimal object classification is performed to distinguish + objects which are consistent with a single PSF, objects which are + inconsistent, and objects which are saturated. + +\item Before the image is added to the static sky, it must pass Q/A + tests: + \begin{itemize} + \item the measured photometry scatter for the image must be less + than \tbr{1\%}. + + \item the measured astrometry scatter for the image must be less + than \tbr{30 mas}. + \end{itemize} + +\item The final, cleaned input image is added to the static sky so + that an incrementally-deeper static sky image may be + made. +\end{itemize} + +\paragraph{Static Sky Analysis} +\label{IPP:StaticSky} + +The IPP is responsible for performing object detection and analysis on +the static sky. This analysis is performed continuously (every day or +week) on those portions of the sky within 15\degree\ of the sun. In +this analysis, the object measurement is much more detailed than those +performed in the real-time analysis. The currently envisioned +parameters to be measured for every object are listed in +Table~\ref{APdetections}. The parameters include the object centroid +and the position covarience matrix, the instrumental PSF magnitude and +error, local background level and error, a measurement of the +star-galaxy separation, a measurement of the object shape ($\sigma_x, +\sigma_y, \theta$), the Petrosian radius, magnitude, axis ratio, and +angle; the S\'ersic radius, magnitude, axis ratio, angle, and +parameter $\nu$, and a collection of annular aperture flux +measurements, all of which are also measured for the P4$\Sigma$ +images. In addition, the galaxy-shape parameters $Gamma_1 \& +\Gamma_2$, along with the complete `polarization' terms are measured, +as well as a collection of annular aperture flux and variance +measurements. Another unique feature of the static sky analysis is +that the object detection may be performed simultaneously on all +filters, in which case the locations and other parameters may be more +strongly constrained by simultaneously fitting between all bands. The +analysis to be performed may be substantially more complex than the +real-time analysis because of the relatively low data rate. Instead +of needed to process thousands of images per night ($\sim 350$Mpix per +second), it is only necessary to process the complete sky in a year, +or an average rate of $\sim$2 Mpix per second, or $< 1$\% of the +object analysis in the other analysis stages. + +\paragraph{Calibration Stages} +\label{IPP:mkcal} + +The Calibration Analysis Stages construct calibrations from the +relevant input data. Some of these combine multiple raw input images +together, after processing, to create a high-quality high-signal +master calibration image. Some of the calibrations are used to +correct other calibrations. Each of the calibration stages must also +provide the tools to test the quality of the input data against +existing master calibration data and to test the consistency of +multiple measurements of the calibration. + +The Calibration analysis stages may be performed on whatever +timescales are appropriate and necessary to maintain the quality and +relevance of the calibration images. Below, we list the specific +calibration data which must be constructed in the calibration analysis +stages. + +The IPP must generate basic calibration images using the raw bias, +dark, and flat-field (dome or twilight) images obtained by the +telescope as the input. The analysis of these images requires +relatively simple stacking of the input set of images. Outlier +rejection, both of complete input images as well as pixels within the +input stack, must be performed. In addition, each type of image +requires an appropriate normalization which may depend on the data +levels in other detectors in the input set. Each of these calibration +stages must be able to determine from the input stack if the relevant +calibration image needs to be updated and perform an initial test to +see which input images are consistent and valid. + +\subparagraph{Bias Images} + +Bias images may be needed to correct for structure in the bias. The +IPP must have the capability of constructing a master bias image from +a stack of raw bias frames. The input bias images, representing +offsets from the overscan level, are processed by subtracting the +overscan, including 1D structure if needed. + +The master bias frame construction uses outlier image and outlier +pixel rejection to construct a single high-quality bias frame. The +statistic used to determine pixel values from the input stack can be +set by the user to be one of the following: the sample mean, median, +and mode, robust mean, median, and mode, and the clipped mean and +median. Testing of the input images consists of constructing residual +images, in which the master bias is applied to the input images. +These images may be included or excluded from an additional iteration +of the stack on the basis of their pixel-to-pixel statistics. + +\subparagraph{Dark Images} + +Dark images may be needed to correct for structure in the dark +current. The IPP must have the capability of constructing a master +dark image from a stack of raw dark frames. The input dark images are +first corrected for the bias using whatever method is appropriate for +the science images. Master dark frames depend on their exposure time. +As such, the input dark frames must have a limited range of exposure +times, and the output dark frame includes the exposure time as part of +its associated metadata. + +The master dark frame construction uses outlier image and outlier +pixel rejection to construct a single high-quality dark frame. The +statistic used to determine pixel values from the input stack can be +set by the user to be one of the following: the sample mean, median, +and mode, robust mean, median, and mode, and the clipped mean and +median. Testing of the input images consists of constructing residual +images, in which the master dark image is applied to the input images. +These images may be included or excluded from an additional iteration +of the stack on the basis of their pixel-to-pixel statistics. A +collection of master dark frames with a range of exposure times are +used to determine the scaling of the dark frame as a function of +exposure time. + +\subparagraph{On-Off Dark Images for Light Leaks} + +A type of image which may be necessary for calibrations will be pairs +of images taken at night with the shutter closed with and without the +dome shutter closed. Such a pair of images can be used to determine +any light-leak in the camera which may contribute additional flux +across the mosaic. + +\subparagraph{Flat-Field Images} + +Master flat-field images must be constructed from a collection of +input flat-field images. The input flat-field images may be obtained +from any of the standard sources: the dome, the twilight sky, and the +night-time sky. The choice of flat-field input image must be +determined experimentally from observations during the commissioning +phase of the telescope. The IPP flat-field construction system must +be capable of handling any of these sources. + +An appropriate set of input images is selected on the basis of their +flux levels, time of observations, and the observing conditions. The +input flat-field images are processed (bias and dark corrected if +needed) and the resulting images are stacked. The master flat-field +construction uses image and pixel outlier rejection to construct a +single high-quality master flat-field frame. The statistic used to +determine pixel values from the input stack can be set by the user to +be one of the following: the sample mean, median, and mode, robust +mean, median, and mode, and the clipped mean and median. Testing of +the input images consists of constructing residual images, in which +the master flat-field image is applied to the input images. These +images may be included or excluded from an additional iteration of the +stack on the basis of their pixel-to-pixel statistics. + +\subparagraph{Mask Images} + +Preliminary bad-pixel mask images are generated on the basis of +comparison between raw flat-field images and a cleaned, stacked +master. The mask creation system accepts a collection of flat-field +images and identifies pixels which are consistently poorly flattened. +Pixels which are under-responsive are also identified as pixels to be +masked. + +\subparagraph{Sky \& Fringe Frames} + +Fringe-correction frames must be generated to remove the fringe +pattern caused by thin-film interference in the top layers of CCDs, +particularly in the redder passbands. Fringe correction frames may be +constructed on the basis of observations of the night-sky in the +appropriate filters or on the basis of dome fringe lamp observations. +The choice of the appropriate source will be determined experimentally +on the basis of data obtained during the commissioning phase. The IPP +must be capable of handing either source. The images are first +flattened to remove the pixel-to-pixel sensitivity variations of the +detector. The combination of multiple input fringe frames may not be +simply stacked since the amplitude of the fringe pattern varies +independently of other variations in the image. The amplitude of the +fringe pattern in the input frames is measured and the images scaled +to normalize the fringe amplitude to a consistent range (-1 to +1) for +all input images before they are combined with one of the standard +combination statistics (mean, median, mode, etc). The quality of the +input frames is tested by flattening the input image and applying the +master fringe-frame. The resulting residual image statistics are used +to select or exclude specific input images. + +\subparagraph{Shutter Correction Map} + +Shutter correction map images may be generated based on the timing +measurements of the shutter itself, or on the basis of dome-flat +images of decreasing exposure times down to the shortest available +exposures. + +\subparagraph{Low-k Sky Models} + +Large-scale background structure in images which is not caused by +thin-film interference must also be detected and corrected. Models of +this background structure may be a necessary input to the correction +proceedure. The IPP must have the capability of generating image +models of the large-scale structure patterns observed with the +telescope + +\subparagraph{Flat-Field Correction Frame} + +Flat-field images, whether constructed from the dome, twilight, or +night-sky images, do not perfectly correct the detector response in a +consistent fashion across the full field of the camera. The IPP must +have the capability of generating flat-field photometric correction +frames on the basis of the measured photometry of objects which are +moved to a variety of locations on the detector in a sequence of +images. The flat-field correction frames analysis stage makes use of +targetted observations following a specified dither pattern, and +extracts the photometered objects from the AP Database to determine +the necessary photometric corrections. The resulting image is applied +to the master flat-field image. Testing of the correction is +performed by applying the correction to the basic master flat-field +image, applying that flat-field image to the dithered photometry +observations, and performing the object detections. Comparion of the +photometry of individual stars at different locations on the mosaic +will demonstrate the consistency of the flat-field image. + +\subparagraph{Non-Linearity Correction} + +The IPP must have the capability of constructing a correction for +non-linearity in the detectors. These frames are constructed from +exposures of a uniform source with a range of exposure times. The +non-linearity correction frames provide polynomial correction +coefficients or a lookup table describing the correction. There is +likely to be a single non-linear correction for each OTA detector, or +potentially for each Cell. The IPP must handle these two cases. + +\paragraph{Reference Catalog Creation} + +One of the primary goals inital goals of Pan-STARRS is the creation of +photometric and astrometric reference catalogs for the general +community and for additional Pan-STARRS calibration. This +internally-generated reference catalog is necessary to achieve the +photometry and astrometry goals set for the project. The generation +of these catalogs is inherently a research project, and will require +human control and intervention. The IPP will be required to provide +the data access, manipulation and visualization tools needed to +construct these reference catalogs and to assess their quality. In +this section, we discuss the tools needed for this effort. + +\subsubsection{Bias Image Creation} + +The Bias calibration stage constructs a master bias image from a +collection of raw bias images. The tasks and functions include: + +\begin{itemize} + +\item The Bias calibration stage corrects the input images based on + the overscan region, determined from either the header or from + metadata. + +\item The Bias calibration stage combines the input images using the + statistic specified by the user, selected from one of the following: + sample mean, median, and mode, robust mean, median, and mode, and + the clipped mean and median. + +\item The Bias calibration stage construct residual images, in which + the master bias is applied to the input images. + +\item Outlier residual images, those for which the residual bias and + variance in the bias image are excessive, are excluded from the + input image stack and the bias image reconstructed. +\end{itemize} + +\subsubsection{Dark Image Creation} + +The Dark calibration stage shall construct a master dark image from a + collection of raw dark images. The tasks and functions include: + +\begin{itemize} + +\item The Dark calibration stage raises an error if the input images + have exposure times which deviate by more than 2\%. + +\item The Dark calibration stage corrects the input dark images for + the bias. + +\item The Dark calibration stage combines the input images using the + statistic specified by the user, selected from one of the following: + sample mean, median, and mode, robust mean, median, and mode, and + the clipped mean and median. + +\item The Dark calibration stage constructs residual images, in which + the master dark is applied to the input images. + +\item Outlier residual images, those for which the residual level and + variance are excessive, are excluded from the input image stack and + the dark image reconstructed. +\end{itemize} + +\subsubsection{Flat-field Image Creation} + +The Flat-field calibration stage constructs a master flat-field image +from a collection of raw flat-field images. The tasks and functions +include: + +\begin{itemize} + +\item The Flat-field calibration stage accepts a group of images from + one of the following flat-field sources: dome, twilight, + night-sky. + +\item The flat-field calibration stage raises an error if the + input images in a single stack used more than one of the above + flat-field sources or multiple filters. + +\item The Flat-field calibration stage corrects the input flat-field + images for the bias and dark. + +\item The Flat-field calibration stage combines the input images using + the statistic specified by the user, selected from one of the + following: sample mean, median, and mode, robust mean, median, and + mode, and the clipped mean and median. + +\item The Flat-field calibration stage constructs residual images, in + which the master flat-field is applied to the input images. + +\item Outlier residual images, those for which the residual level and + variance are excessive ($> 0.1$\%, or 1.02 times the Poisson limit + of the flat-field image), are excluded from the input image stack + and the flat-field image reconstructed. +\end{itemize} + +\subsubsection{Mask Image Creation} + +The Mask calibration stage constructs a bad-pixel mask from a stack of +raw flat-field images and a master flat-field image. The tasks and +functions include: + +\begin{itemize} + +\item The Mask calibration stage masks the pixels which are + inconsistent in the input flats by more than 1\%, given sufficient + signal-to-noise in the input flats. + +\item The Mask calibration stage mask the pixels which are + consistently low or high in the input flats by more than a factor of + 3 beyond the typical pixel. + +\item The Mask calibration stage masks the pixels identified in a + table of bad pixels generated externally to the calibration stage. + +\item The Mask calibration stage uses multiple bit values to identify + the different types of masked pixels. + +\item The Mask calibration stage raises an error if the input images + generate too many bad pixels in the mask. +\end{itemize} + +\subsubsection{Fringe-frame Creation} + +The Fringe-frame Creation calibration stage constructs a master fringe +frame from a stack of raw night-time sky images or from a stack of +dome fringe frames. The tasks and functions include: + +\begin{itemize} + +\item The Fringe-frame Creation calibration stage raises an error if + the input stack consists is images generated with more than one type + of fringe source or with multiple filters. + +\item The Fringe-frame Creation calibration stage flattens the input + images to remove the pixel-to-pixel sensitivity variations of the + detector. + +\item The Fringe-frame Creation calibration stage measures the fringe + amplitude on the input fringe images. + +\item The Fringe-frame Creation calibration stage scales the input + fringe images based on the fringe amplitude. + +\item The Fringe-frame Creation calibration stage combines the scaled + input images using the statistic specified by the user, selected + from one of the following: sample mean, median, and mode, robust + mean, median, and mode, and the clipped mean and median. + +\item The Fringe-frame Creation calibration stage constructs residual + images, in which the master fringe image is applied to the input + images, along with all necessary preceding calibration images. + +\item The Fringe-frame Creation calibration stage measures the + residual fringe amplitude on the residual images. +\end{itemize} + +\subsubsection{Low-spatial-frequency Sky Models} + +The Sky Model calibration stage constructs a sky model image set from +a stack of raw night-time sky images. + +\subsubsection{Flat-field correction Frame Creation} + +The Flat-field correction calibration stage constructs a flat-field +correction image from dithered observations of a stellar field. The +tasks and functions include: + +\begin{itemize} + +\item The Flat-field correction calibration stage constructs a + flat-field correction image from dithered observations of a stellar + field. + +\item The Flat-field correction calibration stage constructs a + flat-field correction image for each filter and camera + independently. + +\item The Flat-field correction calibration stage constructs a + correction image which makes the photometry of multiple observations + of the same stellar source consistent at different locations in the + focal plane. + +\item The Flat-field correction calibration stage constructs corrected + flat-field images using the measured correction. + +\item The Flat-field correction calibration stage determines the + consistency of the corrected flat-field images using the dithered + stellar field observations flattened with the corrected flat-field + image. +\end{itemize} + +\subsubsection{Non-linearity correction} + +The Non-linear correction calibration stage constructs a correction +model for low-level non-linearity effects in the detector. The tasks +and functions include: + +\begin{itemize} + +\item The Non-linear correction calibration stage constructs a + non-linear correction from a collection of images of a constant + source with varying exposure times. + +\item The Non-linear correction calibration stage construct a + non-linear correction which linearizes the detector fluxes + $<0.5\%$. + +\item The Non-linear correction calibration stage determines the + saturation regime, in which the non-linear correction is no longer + consistent to $<0.5\%$. +\end{itemize} + +\subsubsection{Telescope Astrometry Parameters} + +\begin{itemize} +\item The IPP Calibration system constructs static models of the + telescope astrometry parameters (e.g., distortion, detector warps) + once per week. + +\item The IPP Calibration system constructs static models of the + telescope astrometry parameters (e.g., distortion, detector warps) + with an accuracy to produce astrometry consistent to 30 + milliarcsec. + +\item The IPP Calibration system monitors changes in the telescope + astrometry parameters and issue a warning if the parameters change + by more than 2\%. +\end{itemize} + +\subsubsection{Zero-Point Monitoring} + +The IPP Calibration system determines telescope filter and camera +zero-points on a nightly basis with an accuracy sufficient to +determine photometry in the native filter systems to 5 millimags. + +\subparagraph{Astrometry Reference Creation} + +\begin{table} +\begin{center} +\caption{Astrometric Reference Catalogs\label{AstroRefs}} +\begin{tabular}{lrrrrl} +\hline +\hline +Name & scatter limit & proper & depth & Nstars & filters \\ + & (milliarcsec) & motion &(mag) & (millions) & \\ +\hline +Hipparcos & 1 & 2 & 7.3 & 0.1 & {\em V} \\ +Tycho2 & 10 & 1 & 11.5 & 2.5 & {\em B,V} \\ +UCAC-2 & 20 & 1 & 16.0 & 48.0 & {\em R} \\ +USNO-A2.0 & 250 & N/A & 19.0 & 526.2 & {\em B,R} \\ +USNO-B1.0 & 200 & \tbr{20} & 21.0 & 1042.6 & {\em B,R} \\ +2MASS & 70 & N/A & \tbr{15.0} & 470.0 & {\em J,H,K} \\ +\hline +\end{tabular} +\end{center} +\end{table} + +The existing astrometric reference catalogs are known to have +limitations in accuracy as noted in Table~\ref{AstroRefs}. The +internal accuracy of the Pan-STARRS dataset can potentially be much +higher than the external reference catalogs. The IPP must have the +capability of generating an astrometric reference on the basis of the +observations obtained by the AP survey. The IPP must provide the +analysis tools needed to generate the master astrometric reference +catalog. Much of the required functionality is covered by the AP +Database. + +The two basic, necessary ingredients for the construction of the +Astrometric Reference Catalog are: the observed coordinates of stars +and the existing astrometric reference catalogs. +Table~\ref{AstroRefs} lists a subset of the reference catalogs which +we will use at different stages in the analysis process, along with +notes about their accuracy. + +These catalogs must be available and accessible to the AP Database. +It is necessary to have the tools to convert the reference catalog +object coordinates to all of the possible coordinate frames of +relevance in the telescope and camera system, including: +\begin{itemize} +\item Catalog to mean positions +\item Mean to apparent positions +\item Apparent positions + pointing to arbitrary tangent plane coordinates +\item Apparent positions + pointing to focal plane coordinates +\item focal plane to specific detector (OTA) +\item specific detector to detector cell +\end{itemize} + +In addition to the reference catalogs, it will be necessary to +determine and have available for the analysis system a variety of +approximate calibration data, including the telescope and camera +optical distortion models and the variation of the image PSF across +the camera field, as a function of color. + +The other necessary ingredient in the astrometry reference creation is +the observation of stars by Pan-STARRS. The object detections are +produced by several of the analysis stages discussed in the Science Image +Analysis section. The likely measurement of relevance to the +astrometric reference catalog is the object extraction for the +individual, detrended images (section~\ref{IPP:Phase2}). The detected +objects must be matched against the reference catalogs, and it must be +possible to determine fit coefficients as a function of position +alone, or with combinations of magnitude or color. The fitting method +must include robust outlier rejection. It is also necessary to have +information about the objects which are detected in the catalog, but +not the science image or vice-versa, as well as an assessment of the +centroiding errors for each object. It must be possible to plot the +fit residuals against a wide variety of parameters, including the +object positions, magnitudes, colors, etc, and to make subset +selections of the objects on the basis of these parameters. . + +An alternative measurement of the stellar positions is derived from +the guide stars, which are much brighter than the typical saturated +stars. It must be possible to compare the coordinates of the guide +stars with the coordinates of the other stars in the image. It must +also be possible to perform the various fitting steps for the guide +stars rather than for the normal image data. + +\subparagraph{Photometry Reference Creation} + +\begin{table} +\begin{center} +\caption{Photometric Reference Catalogs\label{PhotoRefs}} +\begin{tabular}{lrrr} +\hline +\hline +Name & scatter & depth & filters \\ + & mmag & mag & \\ +\hline +SDSS & 15 & 16 & {\em u,g,r,i,z} \\ +CFHT-LS & \tbr{10} & 18 & {\em u,g,r,i,z} \\ +Landolt & 10-20 & 15 & {\em U,B,V,R,I} \\ +\hline +\end{tabular} +\end{center} +\end{table} + +The IPP must provide the analysis tools needed to generate a master +photometric reference catalog. The tools needed for generation of the +photometric reference catalogs are similar in essence to those used +for the astrometric reference catalog. It is necessary to confront +the observed objects against the existing reference catalogs to +determine the necessary calibrations. Again, much of the required +functionality is covered by the AP Database. + +The necessary ingredients for the construction of the Photometric +Reference Catalog are: the observed magnitudes of stars and the +existing photometric reference catalogs. An internally consistent +magnitude system will be generated as the primary reference catalog. +In addition, comparison of these magnitudes with the reference +magnitudes will allow for the determination of color transformations +and calibrated magnitudes in the reference system. +Table~\ref{PhotoRefs} lists a variety of reference catalogs which may +be used in the process. These catalogs must be available and +accessible to the AP Database. + +The other necessary ingredient in the photometry solution is the +observation of stars with Pan-STARRS. The photometry is determined by +several of the analysis stages discussed in the Science Image Analysis +section. The likely measurement of relevance to the photometric +reference catalog is the object extraction for the individual, +detrended images (section~\ref{IPP:Phase2}). It is necessary to have +the tools to convert between different photometric systems, including: +\begin{itemize} +\item instrumental to nominal detector magnitude +\item nominal detector magnitude to average filter system +\item average filter system to reference photometry system +\end{itemize} +These transformations are based on a set of measured coefficients for +the color and airmass dependency of the measurement. In addition to +these types of transformations, it is necessary to have the ability to +measure and apply relative photometry corrections. + +The detected objects must be matched against the reference catalogs, +and it must be possible to determine fit coefficients as a function of +airmass, magnitude, color and detector coordinates, or with +combinations of the above. The fitting method must include robust +outlier rejection. It is also necessary to perform exclusions on the +basis of object locations, instrumental magnitudes, observed and +reference errors, and time of the observations. It must be possible to +plot the fit residuals against a wide variety of parameters, including +the object positions, magnitudes, colors, etc, and to select a subset +of the objects on the basis of these parameters. It will likely be +necessary to maintain individual color transformations for each +detector and filter combination to a single internal system for each +filter. + +An alternative measurement of the stellar photometry is derived from +the guide stars, which are much brighter than the typical saturated +stars. It must be possible to relate the magnitudes of the guide +stars with the magnitudes of the other stars in the image. It must +also be possible to perform the above fitting steps for the guide +stars rather than for the normal image data. + +\subsubsection{Modules} + +In order to encapsulate functionality, the analysis stages are +constructed of a sequence of steps. The analysis stages consist of +scripts in a high-level language, likely to be either Python or Perl, +which executes a sequence of C-level functions. The C-level functions +called by the script are called {\em modules} and represent basic data +analysis operations. + +\subsubsection{Pan-STARRS IPP Library} + +In order to facilitate testing and development, and to encourage +flexibility, the IPP is built in a layered fashion. The lowest level +functions are written in C and collected together into a Pan-STARRS +library, \code{PSLib}. + +The Pan-STARRS Data Library consists of C structures describing the +basic data types needed by the IPP and C functions which perform the +basic data manipulation operations. The library is organized into +four topics: System Utilities, Basic Data Collections, Data +Manipulation, and Astronomy-Specific Functions. The Modules are +constructed using these low-level Library functions as needed. + +\subsection{Summary of Derived Requirements} + +\begin{enumerate} + +\item The IPP Image Server shall accept raw images from the summit at + a sustained rate of 1 exposure (4 FPAs or 8~GB) per \tbr{40 seconds}. +\label{IPP:DeReq:1} + +\item The IPP Metadata Database shall accept metadata from the summit + at a sustained rate of \tbr{1 MB per 40 second}. +\label{IPP:DeReq:2} + +\item The IPP Calibration Analysis shall produce master calibration + images from the raw calibration images in less \tbr{2 hours}. +\label{IPP:DeReq:3} + +\item Master calibration images shall not introduce systematic + uncertainties in the photometry greater than \tbr{0.2\%}. +\label{IPP:DeReq:4} + +\item The IPP Science Analysis shall pre-process the science images + with the master calibration images at a sustained rate of 1 exposure + per \tbr{40 seconds}. +\label{IPP:DeReq:5} + +\item The IPP Science Analysis shall merge multiple pre-processed + science images into stacked images with corresponding signal-to-noise + maps at a sustained rate of 1 exposure per \tbr{40 seconds}. +\label{IPP:DeReq:6} + +\item The IPP Science Analysis shall excise pixels from the input + images which are outliers for the ensemble of corresponding pixels + with an efficiency of $> 99$\%. +\label{IPP:DeReq:7} + +\item The IPP Science Analysis shall merge the cleaned images into the + static sky image, and update the corresponding exposure (S/N) maps, + at a sustained rate of 1 exposure per \tbr{40 seconds}. +\label{IPP:DeReq:8} + +\item The IPP Science Analysis shall detect and measure parameters of +objects on the pre-processed science images. +\label{IPP:DeReq:9} + +\item The IPP Science Analysis shall detect and measure parameters of +objects on the stacked science images. +\label{IPP:DeReq:10} + +\item The IPP Science Analysis shall detect and measure parameters of +objects on the difference images. +\label{IPP:DeReq:11} + +\item The IPP Science Analysis shall detect and measure parameters of +objects on the static sky images. +\label{IPP:DeReq:12} + +\item The IPP Science Analysis shall determine astrometry of the + detected objects relative to an external astrometric reference with + an accuracy of \tbr{750 mas} (for bright objects) in the + Commissioning phase of the telescope. +\label{IPP:DeReq:13} + +\item The IPP Science Analysis shall determine astrometry of the + detected objects relative to an external astrometric reference with + an accuracy of \tbr{250 mas} (for bright objects) during the + construction of the Pan-STARRS Astrometric Reference Catalog. +\label{IPP:DeReq:14} + +\item The IPP Science Analysis shall determine astrometry of the + detected objects relative to the Pan-STARRS Astrometric Reference + with an accuracy of \tbr{100 mas} (for bright objects) during normal + operations. +\label{IPP:DeReq:15} + +\item The IPP Science Analysis shall determine photometry of the + detected objects within an internal photometric system with scatter + less than \tbr{25 millimags} (for bright objects) during the + Commissioning phase of the telescope in photometric weather. +\label{IPP:DeReq:16} + +\item The IPP Science Analysis shall determine photometry of the + detected objects within an internal photometric system with scatter + less than \tbr{10 millimags} (for bright objects) during the + construction of the Pan-STARRS Photometric Reference Catalog in + photometric weather. +\label{IPP:DeReq:17} + +\item The IPP Science Analysis shall determine photometry of the + detected objects within an internal photometric system with scatter + less than \tbr{5 millimags} (for bright objects) during normal + operations in photometric weather. +\label{IPP:DeReq:18} + +\item The IPP Science Analysis shall determine photometry of the + detected objects in an external photometric system with scatter less + than \tbr{10 millimags} (for bright objects) during normal operations + in photometric weather. +\label{IPP:DeReq:19} + +\item The IPP Reference Creation System shall produce an astrometric + reference catalog from the extracted objects within 6 months of the + end of the AP Survey. +\label{IPP:DeReq:20} + +\item The IPP Reference Creation System shall produce an astrometric + reference catalog with an absolute accuracy of \tbr{100 mas} and a + local relative accuracy of \tbr{30 mas} for bright objects. +\label{IPP:DeReq:21} + +\item The IPP Reference Creation System shall produce an astrometric + reference catalog with proper motions measurements for + non-solar-system objects with an accuracy of \tbr{20 mas / year} for + unsaturated, bright stars. +\label{IPP:DeReq:22} + +\item The IPP Reference Creation System shall produce a photometric + reference catalog from the extracted point-source objects within 6 + months of the end of the AP Survey. +\label{IPP:DeReq:23} + +\item The IPP Reference Creation System shall produce a photometric + reference catalog with a consistency across the sky of \tbr{5 + millimag}. +\label{IPP:DeReq:24} + +\item The IPP Reference Creation System shall produce a photometric + reference catalog with an absolute calibration to the external + system (defined by \tbr{SDSS} and the CFHT Legacy Survey Standards) + with an accuracy of \tbr{10 millimag} (for bright objects). +\label{IPP:DeReq:25} + +\item The IPP shall publish the static sky images to the Pan-STARRS +PSPS on a time-scale of \tbr{6 month}. +\label{IPP:DeReq:26} + +\item The IPP shall publish the detected objects to the Pan-STARRS +PSPS on a time-scale of \tbr{1 month}. +\label{IPP:DeReq:27} + +\item The IPP shall publish the IPP and OTIS metadata to the +Pan-STARRS PSPS on a time-scale of \tbr{1 week}. +\label{IPP:DeReq:28} + +\item The IPP shall provide to the MOPS subsystem the detected + single-occurrence transient objects \tbr{by the end of every night}. +\label{IPP:DeReq:29a} + +\item The IPP shall provide to the MOPS subsystem the metadata +appropriate to the images from which single-occurrence transient +objects were detected \tbr{by the end of every night}. +\label{IPP:DeReq:29b} + +\item The IPP shall provide to external Pan-STARRS clients the + detected objects within \tbr{5 minute} after the image is obtained. +\label{IPP:DeReq:29c} + +\item The IPP shall store the raw images for a period of \tbr{1 month}. +\label{IPP:DeReq:30} + +\item The IPP shall store the detected objects for a minimum of \tbr{1 year}. +\label{IPP:DeReq:31} + +\item The IPP shall store the metadata for the lifetime of the project. +\label{IPP:DeReq:32} +\end{enumerate} + +\subsection{Internal Interfaces} + +The IPP has internal interfaces between several of the architectural +components and between the architectural components and the analysis +stages. + +\begin{enumerate} + +\item IPP Scheduler - IPP Controller. The IPP Scheduler must +send to the IPP Controller information about the tasks to be +performed and must receive from the IPP Controller descriptions of the +success or failure of these tasks. + +\item IPP Scheduler - Metadata DB. The IPP Scheduler must query the +Metadata DB to determine an appropriate course of action. The IPP +Scheduler must send result and status information to the Metadata DB. + +\item IPP Controller - Analysis Tasks. The IPP Controller must +initiate the Analysis Tasks and monitor their output and exit status. + +\item Analysis Tasks - Metadata DB. The Analysis Tasks must be able +to query the Metadata DB as needed to extract metadata needed for a +given task. The Analysis Tasks must be able to send results and +updates to the Metadata DB. + +\item Analysis Tasks - Image Server. The Analysis Tasks must be able +to extract relevant images from the Image Server. The Analysis Tasks +must be able to send output images to the Image Server. + +\item Analysis Tasks - AP DB. The Analysis Tasks must be able +to extract information related to specific objects from the +Astrometric and Photometric Database. The Analysis Tasks must be able +to send result detections to the AP Database. +\end{enumerate} + +\subsection{Requirements Trace Matrix} +\begin{center} +\footnotesize +\begin{tabular}{|l|l|l|l|} +\hline +\multicolumn{2}{|c|}{\bf Subsystem Requirements} & +\multicolumn{2}{|c|}{\bf System/Subsystem Requirements} \\ \hline +\FRN{\bf Number} & +\FRS{\bf Caption} & +\FRN{\bf Number} & +\FRS{\bf Caption} \\ \hline + +\ref{IPP:Req:1} & \FRS{Photometrically consistent images to 1\%} & \ref{OpsReq:D} & \FRS{Photometric precision of 0.01 mag} \\ \hline +\ref{IPP:Req:2} & \FRS{Photometrically calibrated images to 1\%} & \ref{OpsReq:D} & \FRS{Photometric precision of 0.01 mag} \\ \hline +\ref{IPP:Req:3} & \FRS{Absolute astrometry to 100 mas} & \ref{OpsReq:E} & \FRS{Absolute Astrometric precision of 100 mas} \\ \hline +\ref{IPP:Req:4} & \FRS{Relative astrometry to 30 mas} & \ref{OpsReq:F} & \FRS{Relative Astrometric precision of 100 mas} \\ \hline +\ref{IPP:Req:5} & \FRS{Background correction to 1\%} & \ref{OpsReq:G} & \FRS{remove foregrounds to $<$ 1\% of background} \\ \hline +\ref{IPP:Req:6} & \FRS{Construct Static Sky for $g$ filter images} & \ref{OpsReq:J} & \FRS{construct static sky images} \\ \hline +\ref{IPP:Req:7} & \FRS{Construct Static Sky for $r$ filter images} & \ref{OpsReq:J} & \FRS{construct static sky images} \\ \hline +\ref{IPP:Req:8} & \FRS{Construct Static Sky for $i$ filter images} & \ref{OpsReq:J} & \FRS{construct static sky images} \\ \hline +\ref{IPP:Req:9} & \FRS{Construct Static Sky for $z$ filter images} & \ref{OpsReq:J} & \FRS{construct static sky images} \\ \hline +\ref{IPP:Req:10} & \FRS{Construct Static Sky for $y$ filter images} & \ref{OpsReq:J} & \FRS{construct static sky images} \\ \hline +\ref{IPP:Req:11} & \FRS{Construct Static Sky for $w$ filter images} & \ref{OpsReq:J} & \FRS{construct static sky images} \\ \hline +\ref{IPP:Req:12} & \FRS{Detect \& classify objects on science images} & \ref{OpsReq:P} & \FRS{classify detected objects } \\ \hline +\ref{IPP:Req:13} & \FRS{Detect \& classify objects on science image stacks} & \ref{OpsReq:P} & \FRS{classify detected objects } \\ \hline +\ref{IPP:Req:14} & \FRS{Detect \& classify objects on static sky images} & \ref{OpsReq:P} & \FRS{classify detected objects } \\ \hline +\ref{IPP:Req:15} & \FRS{Detect \& classify transients} & \ref{OpsReq:P} & \FRS{classify detected objects} \\ \hline +\ref{IPP:Req:16} & \FRS{Degrade image size by $<$ 10 mas} & \ref{SysDeReq:2}, alloc. & \FRS{Degrade image size by \PSiqdeg\ of median seeing} \\ \hline +\ref{IPP:Req:17} & \FRS{Process images arriving at cadence of 40 seconds} & \ref{OpsReq:B} & \FRS{processing for transients within 5 min} \\ \hline +\ref{IPP:Req:18} & \FRS{Limit false alarm rate for transients} & \ref{OpsReq:M} & \FRS{false alarm rate of $<1\%$} \\ \hline +\ref{IPP:Req:19} & \FRS{Publish static sky images to PSPS} & \ref{OpsReq:Q} & \FRS{Data Products shall be made available} \\ \hline +\ref{IPP:Req:20} & \FRS{Publish detected objects to PSPS} & \ref{OpsReq:Q} & \FRS{Data Products shall be made available} \\ \hline +\ref{IPP:Req:21} & \FRS{Publish metadata to PSPS} & \ref{OpsReq:Q} & \FRS{Data Products shall be made available} \\ \hline +\ref{IPP:Req:22} & \FRS{Provide access to preferred science clients} & \ref{SysDeReq:9} & \FRS{allow interface to preferred science clients} \\ \hline +\ref{IPP:Req:23} & \FRS{Store raw images for 1 month} & allocated & \FRS{} \\ \hline +\ref{IPP:Req:24} & \FRS{Store detected objects for 1 year} & allocated & \FRS{} \\ \hline +\ref{IPP:Req:25} & \FRS{Store metadata for project lifetime} & allocated & \FRS{} \\ \hline +\end{tabular} +\end{center} + +\begin{center} +\footnotesize +\begin{tabular}{|l|l|l|l|} +\hline +\multicolumn{2}{|c|}{\bf Derived Subsystem Requirements} & +\multicolumn{2}{|c|}{\bf Top-level Subsystem Requirements} \\ \hline +\FRN{\bf Number} & +\FRS{\bf Caption} & +\FRN{\bf Number} & +\FRS{\bf Caption} \\ \hline + +\ref{IPP:DeReq:1} & \FRS{Accept images from summit at rate...} & \ref{IPP:Req:17} & \FRS{Process images arriving at cadence of 40 seconds} \\ \hline +\ref{IPP:DeReq:2} & \FRS{Accept metadata from summit at rate...} & \ref{IPP:Req:17} & \FRS{Process images arriving at cadence of 40 seconds} \\ \hline +\ref{IPP:DeReq:3} & \FRS{Produce master calibration images in...} & allocated & \FRS{} \\ \hline +\ref{IPP:DeReq:4} & \FRS{Master cal. image introduce less than $1\%$} & \ref{IPP:Req:1} & \FRS{Photometrically consistent images to 1\%} \\ \hline +\ref{IPP:DeReq:5} & \FRS{Pre-process science image at rate...} & \ref{IPP:Req:17} & \FRS{Process images arriving at cadence of 40 seconds} \\ \hline +\ref{IPP:DeReq:6} & \FRS{Merge images into stacked images at rate...} & \ref{IPP:Req:17} & \FRS{Process images arriving at cadence of 40 seconds} \\ \hline +\ref{IPP:DeReq:7} & \FRS{Excise $>99\%$ of outlier pixels from stack} & \ref{IPP:Req:18} & \FRS{Limit false alarm rate for transients} \\ \hline +\ref{IPP:DeReq:8} & \FRS{Merge cleaned images into static sky at rate...} & \ref{IPP:Req:17} & \FRS{Process images arriving at cadence of 40 seconds} \\ \hline +\ref{IPP:DeReq:9} & \FRS{Measure objects on pre-processed images} & \ref{IPP:Req:12} & \FRS{Detect \& classify objects on science images} \\ \hline +\ref{IPP:DeReq:10} & \FRS{Measure objects on stacked images} & \ref{IPP:Req:13} & \FRS{Detect \& classify objects on science image stacks} \\ \hline +\ref{IPP:DeReq:11} & \FRS{Measure objects on difference images} & \ref{IPP:Req:15} & \FRS{Detect \& classify transients} \\ \hline +\ref{IPP:DeReq:12} & \FRS{Measure objects on static sky images} & \ref{IPP:Req:14} & \FRS{Detect \& classify objects on static sky images} \\ \hline +\ref{IPP:DeReq:13} & \FRS{astrometric accuracy for commissioning phase} & allocated & \FRS{} \\ \hline +\ref{IPP:DeReq:14} & \FRS{astrometric accuracy for reference catalog phase} & allocated & \FRS{} \\ \hline +\ref{IPP:DeReq:15} & \FRS{astrometric accuracy for normal operations} & \ref{IPP:Req:3} & \FRS{Absolute astrometry to 100 mas} \\ \hline +\ref{IPP:DeReq:16} & \FRS{photometric accuracy for commissioning phase} & allocated & \FRS{} \\ \hline +\ref{IPP:DeReq:17} & \FRS{photometric accuracy for reference catalog phase} & allocated & \FRS{} \\ \hline +\ref{IPP:DeReq:18} & \FRS{relative photometric accuracy for normal operations} & \ref{IPP:Req:2} & \FRS{Photometrically calibrated images to 1\%} \\ \hline +\ref{IPP:DeReq:19} & \FRS{absolute photometric accuracy for normal operations} & \ref{IPP:Req:2} & \FRS{Photometrically calibrated images to 1\%} \\ \hline +\ref{IPP:DeReq:20} & \FRS{astrometric reference within 6 mo} & \ref{IPP:Req:3}, allocated & \FRS{Absolute astrometry to 100 mas} \\ \hline +\ref{IPP:DeReq:21} & \FRS{astrometric reference astrometry accuracy} & \ref{IPP:Req:3} & \FRS{Absolute astrometry to 100 mas} \\ \hline +\ref{IPP:DeReq:22} & \FRS{astrometric reference proper motion accuracy} & \ref{IPP:Req:3} & \FRS{Absolute astrometry to 100 mas} \\ \hline +\ref{IPP:DeReq:23} & \FRS{photometric reference within 6 mo} & \ref{IPP:Req:2}, allocated & \FRS{Photometrically calibrated images to 1\%} \\ \hline +\ref{IPP:DeReq:24} & \FRS{photometric reference global consistency} & \ref{IPP:Req:2} & \FRS{Photometrically calibrated images to 1\%} \\ \hline +\ref{IPP:DeReq:25} & \FRS{photometric reference absolute accuracy} & \ref{IPP:Req:2} & \FRS{}Photometrically calibrated images to 1\% \\ \hline +\ref{IPP:DeReq:26} & \FRS{publish static sky images every 6 mo.} & \ref{IPP:Req:19} & \FRS{Publish static sky images to PSPS} \\ \hline +\ref{IPP:DeReq:27} & \FRS{publish detected objects every 1 mo.} & \ref{IPP:Req:20} & \FRS{Publish detected objects to PSPS} \\ \hline +\ref{IPP:DeReq:28} & \FRS{publish metadata every 1 week} & \ref{IPP:Req:21} & \FRS{Publish metadata to PSPS} \\ \hline +\ref{IPP:DeReq:29a} & \FRS{provide transients to MOPS rate...} & \ref{IPP:Req:22} & \FRS{Provide access to preferred science clients} \\ \hline +\ref{IPP:DeReq:29b} & \FRS{provide metadat to MOPS at rate...} & \ref{IPP:Req:22} & \FRS{Provide access to preferred science clients} \\ \hline +\ref{IPP:DeReq:29c} & \FRS{provide transients to other clients at rate...} & \ref{IPP:Req:22} & \FRS{Provide access to preferred science clients} \\ \hline +\ref{IPP:DeReq:30} & \FRS{store raw images for 1 month} & \ref{IPP:Req:23} & \FRS{Store raw images for 1 month} \\ \hline +\ref{IPP:DeReq:31} & \FRS{store detected objects for 1 year} & \ref{IPP:Req:24} & \FRS{Store detected objects for 1 year} \\ \hline +\ref{IPP:DeReq:32} & \FRS{store metadata for project lifetime} & \ref{IPP:Req:25} & \FRS{Store metadata for project lifetime} \\ \hline +\end{tabular} +\end{center} + +\endnotesection + +\ifwhole\else + +\subsection{Notes}{ + +There are several system concepts and goals which must be defined +before the IPP section: + +\begin{enumerate} +\item relationship between FPA image, OTA, and cell +\item concept of a `major frame' (observationally defined group of + images). +\item distinction of a images grouped together for a science goal from + images grouped together for processing purposes. + +\item timing budget: need to specify total delay allowed from shutter + close to specific science result, and break down for the stages: + readout, OTIS-IPP communication, OTIS-IPP data transfer, IPP + processing Phase 2 and Phase 4, IPP-client science pipeline + communication. + +\item{discuss the astrometry and photometry errors introduced at each + stage of the analysis} + +\item mention future pre-processing at the summmit? + +\item does the Top-Level Description need a discussion of the basic goals of the image analysis? + +\end{enumerate} + +\subsection{Definitions} + +\subsection{External Interfaces} + +\end{document} +\fi