Index: trunk/doc/design/ippSSDD.tex =================================================================== --- trunk/doc/design/ippSSDD.tex (revision 6055) +++ trunk/doc/design/ippSSDD.tex (revision 6167) @@ -1,3 +1,3 @@ -%%% $Id: ippSSDD.tex,v 1.6 2006-01-19 10:58:19 eugene Exp $ +%%% $Id: ippSSDD.tex,v 1.7 2006-01-22 09:54:47 eugene Exp $ \documentclass[panstarrs]{panstarrs} @@ -15,5 +15,5 @@ % allow paragraphs to be listed in TOC for now -\setcounter{tocdepth}{4} +\setcounter{tocdepth}{3} \begin{document} @@ -777,9 +777,5 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsection{AP Database} - -\tbd{this section needs to be updated with current implemetation; the - DVO SDD contains much of this information, but needs to be fleshed - out in places.} +\subsection{DVO : the AP Database} \subsubsection{Corresponding Requirements} @@ -787,15 +783,14 @@ The AP Database must meet the requirements specified in Section 3.4.3 of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The specified design -is chosen to meet requirements 3.4.3.1 and 3.4.3.2. In order to meet -the throughput requirements, the AP Database will be distributed -across 10 Nodes independent of the Image Server Nodes. An alternative -organization of the database which will be studied will have the AP -Database co-located with the Image Server Phase 4 Nodes. +is chosen to meet requirements 3.4.3.1 and 3.4.3.2. The IPP is +modifying the Elixir program 'DVO' to perform the role of the IPP AP +Database. \subsubsection{Overview} -The AP (Astrometry \& Photometry) Database is a CSCI which stores data -related to astronomical objects derived from various sources with a -variety of associations. The AP Database deals with two related +DVO, the Desktop Virtual Observatory, is a software system which +stores data related to astronomical objects derived from various +sources, and provides mechanisms to related multiple detections +together as astronomical objects. DVO deals with two related concepts: {\em objects} and {\em detections}. The {\em objects} are descriptions of astronomical objects while the {\em detections} are @@ -809,65 +804,66 @@ must be accepted as they are reported. -The AP Database stores the collections of detections which were -derived from specific images from any of the analysis stages. It -provides a mechanism to determine the image from which a specific -detection was derived, and in conjunction with the Image Server locate -the corresponding data file. The AP Database also makes it possible -to extract all detections derived from a specific image and to -determine quantities such as the pixel coordinates of the detection on -the image. - -The AP Database also has the capability to associate multiple -detections of a specific object. Several major classes of objects -will be present, each of which must be handled correctly. - -First, the most distant stars, compact galaxies, and QSOs will have -nearly fixed locations relative to other distant stars, with only -small deviations for individual measurements. The association between -multiple detections of such objects is made on the basis of their -coincident positions. The AP Database determines the average position -of the object and the deviations of the individual detections from -that average on the basis of the ensemble of individual detection. - -Second, solar system objects do not have a fixed location. Detections +DVO stores the collections of detections which were derived from +specific images. It provides a mechanism to determine the image from +which a specific detection was derived, and in conjunction with the +Image Server locate the corresponding data file. DVO also makes it +possible to extract all detections derived from a specific image and +to determine quantities such as the pixel coordinates of the detection +on the image. + +DVO also has the capability to associate multiple detections of a +specific object. Several major classes of objects will be present, +each of which must be handled correctly. DVO distinguished the +following types of objects. + +{\bf Stars, compact galaxies, and QSOs} will have nearly fixed +locations relative to other distant stars, with only small deviations +for individual measurements. The association between multiple +detections of such objects is made on the basis of their coincident +positions. DVO determines the average position of the object and the +deviations of the individual detections from that average on the basis +of the ensemble of individual detection. + +{\bf Solar System Objects} do not have a fixed location. Detections of such objects are linked by their orbits, and depend on both the -position and the time of the image. The AP Database does not attempt -to make this link; this is the role of the MOPS system. However, it -has the ability to accept identifications made externally with -specified detections and to return the identifier of the moving object +position and the time of the image. DVO does not attempt to make this +link; this is the role of the MOPS system. However, it has the +ability to accept identifications made externally with specified +detections and to return the identifier of the moving object associated with the specific detections. These associations also include descriptive information such as the offset of the detection from the predicted location of the detection based on the orbit. This -functionality is required to allow the AP Database to ignore known -moving object detections from other types of queries. - -Third, objects in the general vicinity of the solar system fall in -between these first two classes of objects. Their proper motion and -parallax response is significant enough ($>0.2$ arcsec in 1 year) that -they are not well-described by an average location and a collection of -offsets. These objects are described by a distance and a proper -motion vector. The AP Database provides the association between the -specific detections and an average object which includes finite -parallax and proper motion. - -Fourth, many detections, especially in their initial states, will not -be associated with a specific astronomical object of any of the above -classes and are treated as orphans. Most of these will be spurious -(not representing real objects), some will be from solar system -objects for which orbits are not yet determined, some will be from -faint stars near the detection limits, and some will be from -short-term transients which have only been detected once. The AP -Database maintains these detections until they have been associated -with one of the objects above. The AP Database provides mechanisms by -which individual detections may be migrated back and forth between the -orphan state and association with an astronomical object. - -For every object, and all orphaned detections, the AP Database also -provides the capability to determine the images containing the -location of the object but for which no detection was made. The -minimum set of information which must be carried for these +functionality is required to allow DVO to ignore known moving object +detections from other types of queries. + +{\bf High-proper-motion objects} in the general vicinity of the solar +system fall in between these first two classes of objects. Their +proper motion and parallax response is significant enough ($>0.2$ +arcsec in 1 year) that they are not well-described by an average +location and a collection of offsets. These objects are better +described by a distance and a proper motion vector. DVO provides the +association between the specific detections and an average object +which includes finite parallax and proper motion. + +{\bf Orphaned detections} are not associated with a specific +astronomical object of any of the above classes. Most of these will +be spurious (not representing real objects), some will be from solar +system objects for which orbits are not yet determined, some will be +from faint stars near the detection limits, and some will be from +short-term transients which have only been detected once. DVO +maintains these detections until they have been associated with one of +the objects above. DVO provides mechanisms by which individual +detections may be migrated back and forth between the orphan state and +association with an astronomical object. + +DVO stores the information about the detection, the related objects, +and the images which provided the measurements. For every detection, +DVO provides the mechanisms to link the detection back to the image +which supplied it. DVO also provides the capability to determine the +images containing a specific location but for which no detection was +made. The minimum set of information which must be carried for these non-detections is the image and the associated object or orphan. -The AP Database also stores the relationships between various +DVO also stores the relationships between various photometric systems and the evolution of that relationship. It provides mechanisms to convert between the measured instrumental @@ -880,152 +876,220 @@ various reference systems appropriate for those filters. +\subsubsection{Photometric systems and the DVO Photcodes} + +One of the major roles of DVO is to relate different photometric +measurements made with different instruments and detectors together. +We may have observations made with the same basic filters, but using a +number of different detectors. We may have observations from +different telescopes in similar filters. We may have reference data +related to some filter, but obtained and published by other observers. +We would like to related these measurements together in optimal ways, +making use of whatever information we have available. DVO provides +several mechanisms to enable these relationships. + +We identify three distinct types of photometry measurements within +DVO: +\begin{itemize} +\item {\bf reference photometry} These measurements are provided by + external observers. For reference photometry, we do not have access + to very must information used to determine the magnitudes of the + objects of interest. We have the reference magnitudes corresponding + to a type of filter, and presumably some information of the error on + the measurement. We might possibly know the epoch of the + observations, but not necessarily. +\item {\bf detection photometry} This is our primary measurement of + interest: the photometry of objects measured from images which we + have processed. More specifically, the detection photometry is an + instantaneous measurement from a specific image with well-known + properties, such as exposure time, airmass, instrument source, etc. +\item {\bf internal photometry} With the application of an appropriate + zero point and other calibration terms, any detection photometry can + be calibrated to represent a measurement in a well-known photometric + system. The internal photometry measurements are calibrated to be + on a photometric system which represents a consistent system for a + particular telescope or collection of data, minimizing the + calibration transformations necsessary. +\end{itemize} + +Defining the relationships between the different types of measurements +is part of the process of photometric calibration. DVO uses the +concept of the 'photcode' to identify the source of the photometry, +and to define the relationships between different photometry sources. +A photcode identifies a photometric system: for the detection +photometry measurments, each combination of telescope, camera, filter, +and detector is associated with a unique photcode; there are also +unique photcodes for the internal photometry systems and any distinct +external reference source. + +As a concrete example, consider the Pan-STARRS PS-1 system. There +will be three different cameras in use at different times: GPC-1, +TC-3, and the SkyProbe camera. There are at least 6 filter systems: +{\it grizy} and {\it w}. The SkyProbe camera has a single CCD, TC-3 +has 16 different detectors, and GPC-1 has up to 64 different devices. +Each of these combinations is potentially a different photometric +system, so a different photcode is defined for each combination. +These photcodes would have names such as: GPC1.02.r (r filter with the +GPC1 camera and OTA 02) or SP1.00.g (SkyProbe 1, g filter). These +($64 \times 6 + 16 \times 6 + 5 = 485$) photcodes are all identified +as 'detection' photcodes, specifying that detection photometry is +associated with them + +There are also 6 different internal photometric systems of interest, +namely those associated with the 6 named filters, {\it grizy} and {\it +w}. Each of these 6 systems is identified with an internal photcode. +The internal photcodes are further distinguished as 'primary' or +'secondary', which specifies how the DVO system stores average +quantities related to these types of photcodes (see the discussion of +the tables below). + +Finally, there may be multiple external photometric systems of +interest, some of which are related to the major internal photometry +systems, some of which are not. For example, the Pan-STARRS project +may refer to photometry from the SDSS secondary standards, the SDSS +data releases, Johnson photometry from Landolt (1992), observations +from 2MASS in $JHK$, USNO-B observations, and so forth. Each of these +photometric systems is assoiciated with a different photcode; only +some of these are relevant to the detection or internal photometry +system. + +Within DVO, the detection and internal photcodes each define a +relationships as well as a specific photometric system. Associated +with each of these photcodes are the parameters of the photometry +transformation from the photometric system of the photcode to another +photometric system. For the detection photcodes, the parameters +define the transformation to the equivalent internal photcode system. +The currently-defined transformation parameters consist of the +following photometry equation: +% +\[ +M_\lambda = m_\lambda + C_\lambda + K_\lambda (\mbox{airmass} - 1) + \sum_{i = 1}^{i < N} +A_{\lambda,i} (\mbox{color}_\lambda - \mbox{color}_{o,\lambda})^i +\] +% +where $C_r$ represents the zero-point of the transformation, $K_r$ +represents the slope of the airmass trend, $\mbox{airmass}$ is the +airmass for a given measurement, $\mbox{color}$ is the color of the +source of interest (as identified below), $\mbox{color}_r$ is the +reference color for sources in this photometry system, and $A_{r,i}$ +is the coefficient of the $i$ power of the color difference. Up to +fourth order color terms are currently allowed. For any photcode, the +color is defined as the difference of the measurements in two other +photcodes, usually two 'internal' photcodes. The photcode information +also specified the equivalent photcode to which the transformation corresponds. + +For the detection photcodes, the target of the transformation must be +an internal photcode. For the internal photcodes, the target of the +transformation is an external reference photcode system. This +restriction implies that the internal photometry may only be +transformed (and thus compared with) a single external reference. +This is in fact the best practice as far as photometric calibration is +concerned: the 'standard' observations from different references +should always be treated as different photometric systems. To allow +for the relationship of the internal photometry to multiple sources of +reference photometry, an additional set of photcodes are defined which +identify 'alternative' transformations for the internal photcodes. + +It is important to note that not all of the photometry transformation +parameters identified above are relevant for each of the three major +types of photcode. The detection photcodes will in general make use +of all of these elements, though the order of the color transformation +will hopefully be limited if the different devices are sufficiently +similar. For the transformation from the internal photcodes, which +are derivative in some way of the detection photcodes, the airmass +component is invalid: for a single measurement, the +detection-to-internal transformation has already removed the airmass +trend; for an averaged internal photometric measurement, no single +airmass corresponds to the observations. Finally, no transformation +parameters are defined for the reference photcodes at this time. + +DVO provides methods by which these photometry transforamtions are +automatically applied. The specific measurements (detection +photometry) are stored in the database tables as instrumental +magnitudes, and any operation which examines these measurements must +make use of the APIs to convert to an appropriate common system. A +further complication to note is that the photcodes defined above are +static; they do not include any information about changes to the +system sensitivity. This information is carried externally to the +photcode calibration information; the transformations defined by the +photcodes must be considered the {\em starting point} for any +photometric analysis. An additional adjusment can be applied. + +The detections from a specific image may all have a 'calibration' +offset applied which bring the measured photometry into a common +relative system. This calibration offset is associated with the image +and may be a function of position on the detector. The tables which +carry the individual measurements also include the calibration +magnitude appropriate for each measurement to speed up the application +of this offset. In a well-calibrated collection of photometry, all of +the detection measurements will have a measured calibration magnitude, +yielding a collection of internal photometry measurements which are +all consistent. An additional piece of information is the zero-point +history, which tracks the system-wide variations in the average +sensitivity. The zero-point history can be used to predict the +calibration magnitudes for any observation which is not tied directly +via relative photometry to the rest of the photometric observations. + +Putting all of these pieces together, the photometry APIs in DVO can +be used to return any of the following types of photometric +measurements: +\begin{itemize} +\item raw instrumental magnitudes for any detection + +\item 'catalog' magnitudes, applying only the airmass and static + zero-point calibrations to a detection magnitude; this is useful to + test the detector-color transformation. + +\item 'system' measurements, applying the complete static + transformation for a detection magnitude to the internal photometry + system; for photometric weather and no zero-point variations, this + would be a measurement in the internal photometry system. + +\item 'relative' magnitudes, applying the measured calibration offset + to the calibrated detection magnitude determined above; in a + well-calibrated system, this represents a consistent internal + photometry measurement. + +\item 'calibrated' magnitudes, correcting the measure detection + photometry by applying the transformation from the internal + magnitude system to the external reference magntiude system. + +\item 'average' magntiudes, the raw internal photometry magnitudes + (note the distinction between the 'average' quantities, which are + derived from a collection of detections an the 'relative' quantities + which represent an instantenous measurement in the same system). + +\item 'reference' magnitudes, in which the 'average' internal + photometry values are transformed to the refernce magnitude system. +\end{itemize} +The complexity of these transformations is necessary to allow the +examination of the trends of actual measurements with external +parameters. + +\subsubsection{DVO Database Tables} + \begin{figure} \begin{center} -\resizebox{4.5in}{!}{\includegraphics{pics/APDB}} -\caption{AP DB components} -\label{fig:APDBComponents} +\resizebox{4.5in}{!}{\includegraphics{pics/dvo.01.ps}} +\caption{\label{fig:DVOtables} \small Data types managed by DVO} \end{center} \end{figure} -The AP Database provides interfaces to extract lists of objects and -detections based on various query parameters. It provides the -capability to extract all detections associated with a specific -object, all non-detections of that object, all non-detections of an -orphan, and summary statistics from these collections. It will also -return all objects or detections within specified spatial regions -including regions bounded by great circles (RA,DEC; GLAT,GLON; -ELAT,ELON) and regions described by a location and a search radius. -It will also return the image parameters associated with a specific -detection including image coordinates of the detection, exposure time, -time and date of the detection, etc. - -As shown in Figure~\ref{fig:APDBComponents}, the IPP AP Database -consists of the following components: - -\begin{itemize} -\item AP Database database tables -\item AP Database database engine -\item AP Database servers -\item AP Database client APIs -\end{itemize} - -\subsubsection{AP Database Tables} - -Table~\ref{tab:APDBTables} lists the tables used by the AP Database. The -contents of these tables are outlined in -Appendix~\ref{sec:APDBTableContents}. Below, the use of these tables by -the AP Database software is discussed below. Several of the tables -are not just simple tables in the database but are instead table -groups divided into many subtables, each of which represents a portion -of the sky (a {\tt region}). These subtables may also be distributed -across different computers to distribute the processing load. - -\paragraph{Images Table Group} - -The {\tt Images} table group lists all of the images which provided -the data in the AP Database. These tables are subdivided by region on -the sky. In general, the images listed in this table correspond to -the Chips. This group of tables includes sufficient astrometric -parameters to represent the coordinates of the detections to a -sufficient accuracy. Parallel to the Images table is the Mosaic -table. This table is very similar to the Images table, but defines -the Mosaic which corresponds to a group of Images. The parameters -include the astrometric information needed to define the camera -distortion. - -\paragraph{Image Overlaps Table Group} - -The specific subtable of {\tt Images} which contains a given image is -the one which contains the center pixel of that image. An additional -table group, {\tt Image Overlaps} (with the same subtable organization -as the {\tt Images} subtables), lists images which overlap that -specific subtable. Thus, given a particular coordinate, in order to -find that images which overlap that coordinate, it is necessary to -search the images in the {\tt Images} subtable which includes that -coordinate, and all images in the {\tt ImageOverlaps} subtable for -that coordinate. - -\begin{table}[hb] -\begin{center} -\caption{AP Database Tables\label{tab:APDBTables}} -\begin{tabular}{ll} -\hline -\hline -{\bf Table Name} & {\bf Description} \\ -\hline -Images & The images that have objects in the DB. \\ -Image Overlaps & Image regions which are touched by specific images. \\ -Objects & The objects --- average properties of multiple detections of the same object. \\ -Average Magnitudes & Average photometry in multiple filters \\ -Solar System Objects & Identification of solar system objects \\ -Matched Detections & Detections of sources in an image identified with an Object. \\ -Orphaned Detections & Detections of sources in an image not identified with an Object. \\ -Non-detections & Non-detections of objects in an image. \\ -Regions & spatial distribution of tables \\ -Filters & Filters understood by the system. \\ -Photcodes & Transformations between different photometric systems \\ -Zero Points & History of Zero-point \& Airmass terms \\ -Distortion Models & History of Optical Distortion terms \\ -Database Hosts & computers used to store the tables \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\paragraph{Objects Table Group} - -The {\tt Objects} table group (also divided by region) stores the -average parameters for each astronomical object. Certain details of -this table have not yet been specified. In particular, objects with -significant parallax and/or proper motion may potentially be stored in -a distinct table. Solar system object identifications, to the extent -average properties are maintained in the AP Database, will certainly -be stored in a separate table. - -\paragraph{Average Magnitudes Table Group} - -A related table, also divided into the same regions, is the {\tt -Average Magnitudes} table. In this table, there are multiple rows per -object, one for each of the primary filters of interest for which -photometric averaging is performed. This organization makes the -number of primary (averaged) filters a configurable value. - -\paragraph{Matched Detections Table Group} - -The {\tt Matched Detections} table stores all of the measurements of -astronomical objects on specific images. This table includes all -detections associated with the average {\tt Objects}. As discussed -below, bright objects (above a configuration-specified signal-to-noise -level) are defined object even if only one detection has been found at -that position. Faint orphaned objects are not added to this list or -the list of objects. The different types of detections (P2, -P4$\Delta$, P4$\Sigma$) are distinguished by their photometry codes. -(This is only valid if the AP Database does not store different -quantities for these types of detections.) - -\paragraph{Orphaned Detections Table Group} - -The {\tt Orphaned Detections} table stores the detections which have -not been correlated with an existing object. This table is only -populated for objects below a configuration-specified signal-to-noise -limit (e.g., 5$\sigma$). Bright orphaned detections are assigned an -object and added to the {\tt Matched Detections} table. - -\paragraph{Non-detections Table Group} - -The {\tt Non-detections} table stores information about detection -failures for each object. If an image is added to the database which -overlaps an object but the object is not detected, an entry is made in -this table. In practice, this table may store only the most recent -non-detection and the total number, or a similar reduced set of -non-detection statistics. - -\paragraph{Regions Table} +Figure~\ref{fig:DVOtables} illustrates the data managed by DVO, and +Table~\ref{tab:DVOtables} provides a complete listing. The contents +of these tables are outlined in Appendix~\ref{sec:DVOTableContents}. +Below, the use of these tables by DVO software is discussed below. +Several of the tables are not just simple tables in the database but +are instead table groups divided into many subtables, each of which +represents a portion of the sky (a {\tt region}). These subtables may +also be distributed across different computers to distribute the +processing load. + +\paragraph{Sky Regions Table} The {\tt Regions} table is used to subdivide the tables of images, -objects, and detections, etc, as discussed above. The AP Database +objects, and detections, etc, as discussed above. DVO divides the sky into a hierarchy of regions (portions of the sky) each of which is in turn subdivided into smaller portions. Since nearly -all interactions with the AP Database performed by the IPP are limited +all interactions with DVO performed by the IPP are limited in spatial coverage, subdividing the tables allows a specific interaction to search only a small subset of the data. The table of @@ -1049,9 +1113,131 @@ \begin{figure} \begin{center} -\resizebox{6in}{!}{\includegraphics{pics/APDBRegions}} -\caption{AP DB Regions and Image / Object tables} -\label{fig:APDBRegions} +\resizebox{6in}{!}{\includegraphics{pics/dvo.02.ps}} +\caption{DVO Regions and Image / Object tables} +\label{fig:DVOskyregions} \end{center} \end{figure} + +\paragraph{Images Table Group} + +The {\tt Images} table group lists all of the images which provided +the data in DVO. These tables are subdivided by region on +the sky. In general, the images listed in this table correspond to +the Chips. This group of tables includes sufficient astrometric +parameters to represent the coordinates of the detections to a +sufficient accuracy. Parallel to the Images table is the Mosaic +table. This table is very similar to the Images table, but defines +the Mosaic which corresponds to a group of Images. The parameters +include the astrometric information needed to define the camera +distortion. + +\paragraph{Image Overlaps Table Group} + +The specific subtable of {\tt Images} which contains a given image is +the one which contains the center pixel of that image. An additional +table group, {\tt Image Overlaps} (with the same subtable organization +as the {\tt Images} subtables), lists images which overlap that +specific subtable. Thus, given a particular coordinate, in order to +find that images which overlap that coordinate, it is necessary to +search the images in the {\tt Images} subtable which includes that +coordinate, and all images in the {\tt ImageOverlaps} subtable for +that coordinate. + +\begin{table}[hb] +\begin{center} +\caption{DVO Database Tables\label{tab:DVOtables}} +\begin{tabular}{ll} +\hline +\hline +{\bf Table Name} & {\bf Description} \\ +\hline +Images & The images that have objects in the DB. \\ +Image Overlaps & Image regions which are touched by specific images. \\ +Objects & The objects --- average properties of multiple detections of the same object. \\ +Average Magnitudes & Average photometry in multiple filters \\ +Solar System Objects & Identification of solar system objects \\ +Matched Detections & Detections of sources in an image identified with an Object. \\ +Orphaned Detections & Detections of sources in an image not identified with an Object. \\ +Non-detections & Non-detections of objects in an image. \\ +SkyRegions & spatial distribution of tables \\ +Filters & Filters understood by the system. \\ +Photcodes & Transformations between different photometric systems \\ +Zero Points & History of Zero-point \& Airmass terms \\ +Distortion Models & History of Optical Distortion terms \\ +Database Hosts & computers used to store the tables \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\subsection{Objects Table Group} + +\begin{table} +\begin{center} +\caption{DBO Detection Classes \& Object Parameters\label{tab:APdetections}} +\begin{tabular}{lrrrr} +\hline +\hline +Object Parameter & P2 & P4S & P4D & SS \\ +\hline +PSF x,y, covar, $\alpha,\delta$ & + & + & + & + \\ +PSF mag, $\sigma_{\rm mag}$ & + & + & + & + \\ +star/gal sep & + & + & + & + \\ +$\sigma_x$, $\sigma_y$, $\theta$ & + & + & + & + \\ +local sky data & + & + & + & + \\ +Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ +S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ +W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ +exp. spaced aps., Poisson noise, variance & - & - & - & + \\ +\hline +\end{tabular} +\end{center} +\end{table} + +The {\tt Objects} table group (also divided by region) stores the +average parameters for each astronomical object. Certain details of +this table have not yet been specified. In particular, objects with +significant parallax and/or proper motion may potentially be stored in +a distinct table. Solar system object identifications, to the extent +average properties are maintained in DVO, will certainly +be stored in a separate table. + +\paragraph{Average Magnitudes Table Group} + +A related table, also divided into the same regions, is the {\tt +Average Magnitudes} table. In this table, there are multiple rows per +object, one for each of the primary filters of interest for which +photometric averaging is performed. This organization makes the +number of primary (averaged) filters a configurable value. + +\paragraph{Matched Detections Table Group} + +The {\tt Matched Detections} table stores all of the measurements of +astronomical objects on specific images. This table includes all +detections associated with the average {\tt Objects}. As discussed +below, bright objects (above a configuration-specified signal-to-noise +level) are defined object even if only one detection has been found at +that position. Faint orphaned objects are not added to this list or +the list of objects. The different types of detections (P2, +P4$\Delta$, P4$\Sigma$) are distinguished by their photometry codes. +(This is only valid if DVO does not store different +quantities for these types of detections.) + +\paragraph{Orphaned Detections Table Group} + +The {\tt Orphaned Detections} table stores the detections which have +not been correlated with an existing object. This table is only +populated for objects below a configuration-specified signal-to-noise +limit (e.g., 5$\sigma$). Bright orphaned detections are assigned an +object and added to the {\tt Matched Detections} table. + +\paragraph{Non-detections Table Group} + +The {\tt Non-detections} table stores information about detection +failures for each object. If an image is added to the database which +overlaps an object but the object is not detected, an entry is made in +this table. In practice, this table may store only the most recent +non-detection and the total number, or a similar reduced set of +non-detection statistics. \paragraph{Other Reference Tables} @@ -1062,94 +1248,257 @@ photometry system may consist of a detector, telescope, and specific filter, or it may be a derived photometry system. The {\tt Database -Machines} table identifies all of the computers available to the AP -Database. - -\subsubsection{AP Database servers} - -The AP Database functions on a group of computers, with portions of -the tables stored on separate machines, as described above. The -association between a machine and the corresponding table or part of -the sky is defined by the Region table. Each machine has a -corresponding AP Database server which runs on that machine to -interact with the tables available on that machine. Two possible -interaction models are considered. - -{\bf Option A:} A client chooses one of the machines and sends its -query or data to that machine. The server then uses the region table -to determine which machines contain the relevant portion of the sky. -Data to be added to the database is divided into corresponding region -chunks and sent to the appropriate servers. Queries are redirected to -the appropriate server(s). The original server may collect the -results and return them to the original client. - -{\bf Option B:} The client downloads the region table and performs the -division of the data into appropriate subsets. The subsets are then -sent to the corresponding servers by the client. - -The differences between these models is small. The first option may -make the code more testable, placing all of the logic in the servers -and making each server symmetric. The smaller tables (ie, Region, -Filters, etc) could either be downloaded from a single server or -replicated to all AP DB servers. For these reasons, Option A will be -used for the PS-1 IPP. \tbd{update this in light of the addstar - client / server implementation} - -\subsubsection{AP Database engine} - -The backend database engine for the AP Database stores the tables and -provides them to the servers on demand. The AP Database will use a -\code{mysql} database engine for this function. - -\subsubsection{AP DB Client operations} - -The AP Database client interactions consist of a collection of basic -queries of the database, along with more complex operations to perform -particular tasks. The complex operations are listed below. - -\paragraph{Insert Image \& Detection Set (addstar)} - -One of the most basic operations needed by the AP Database is to -insert a collection of detections derived from a specific image, and -add the definition of that image to the database. This operation is -critical in terms of the processing throughput. After the detections -have been assigned to the appropriate regions, they are matched -against all objects in the {\tt Objects} table. Matches are performed -only on the basis of positional coincidence, using a matching radius -which may depend on the image astrometry errors, or may be a fixed -distance. Any matched detections are added to the {\tt Matched -Detections} table. Any unmatched detections brighter than the Faint -Detection cut-off are specified as a new {\tt Object} and also added -to the {\tt Matched Detections} table. Any faint unmatched detections -are added to the {\tt Orphaned Detections} table. This division is -important because it allows the automatic association of new -detections with existing bright objects while limiting the I/O volume -required to make the detections. In general, there will be many fewer -{\tt Objects} than {\tt Detections}, and there will be fewer bright -orphans than faint orphans. - -\paragraph{Insert Reference Objects (addrefs)} - -This operation is very similar to the previous one. A collection of -reference objects are added to the database as a collection of -detections. The reference photometry should in general be given its -own photometry code. The reference data is different from the image -detection set because the associated image information is not -included. Thus, no corresponding images are added to the database. - -\paragraph{Determine Relative Photometry in region (relphot)} +Machines} table identifies all of the computers available to DVO. + +\subsubsection{Database Table I/O} + +\begin{figure} +\begin{center} +\resizebox{4.5in}{!}{\includegraphics{pics/dvo.03.ps}} +\caption{\label{fig:DVOformats} \small DVO Table I/O } +\end{center} +\end{figure} + +DVO allows for a flexible representation of its data on disk. Data +may be written to disk in one four possible mode: RAW, FITS MEF, FITS +SPLIT, and MYSQL. These modes define the overall organization of the +data on disk. In the RAW mode, the data is written to disk in a +pseudo-FITS table format which consists of a simple FITS header +describing the layout followed by the binary data in a block. This +storage mode is maintained for historical reasons. There are also two +types of FITS modes in which the data tables are written as valid FITS +Binary Tables. In the SPLIT format, every data table is written as a +separate file, while in the MEF format, the object and detection +tables are bundled together into a single FITS file with multiple +table extensions. The MEF format has the advantage of minimize the +proliferation of files, while the SPLIT format is required to make use +of the fastest read/write capabilities of DVO. DVO makes use of these +raw data formats as a throughput risk mitigation strategy. As +discussed below, this strategy has proven very successful. + +There are also multiple formats in which the data may be stored. The +different formats define which specific database table columns are +stored and with what numerical format and precision. +Figure~\ref{fig:DVOformat} illustrates the conversion process which +DVO performs when loading in the data. When DVO loads data from a +file-based table (FITS or RAW), it first loads from the disk file into +a data structure representing the external format in use. The +external structure is then converted into the internal format. The +internal structure is always specified to be the superset of all +external data formats. This capability allows DVO to maintain +backwards compatibility with data tables written with early versions. +As DVO is extended and new elements are added to the tables, it is +only necessary to define the methods to convert the new internal table +into the external table. In addition, DVO makes use of autocoded +table manipulation and I/O APIs which are generated for each data +structure based on a descriptive table. This makes it easy to add new +data types and input/output methods without significant re-coding. + +\tbd{DVO mysql table storage is not yet implemented} + +\subsubsection{addstar : Insert Image \& Detection Set} + +\begin{figure} +\begin{center} +\resizebox{4.5in}{!}{\includegraphics{pics/dvo.04.ps}} +\caption{\label{catalog} \small a figure } +\end{center} +\end{figure} + +One of the most basic operations needed by DVO is to insert a +collection of detections derived from a specific image, and add the +definition of that image to the database. This operation is critical +in terms of the processing throughput. After the detections have been +assigned to the appropriate regions, they are matched against all +objects in the {\tt Objects} table. Matches are performed only on the +basis of positional coincidence, using a matching radius which may +depend on the image astrometry errors, or may be a fixed distance. +Any matched detections are added to the {\tt Matched Detections} +table. Any unmatched detections brighter than the Faint Detection +cut-off are specified as a new {\tt Object} and also added to the {\tt +Matched Detections} table. Any faint unmatched detections are added +to the {\tt Orphaned Detections} table. This division is important +because it allows the automatic association of new detections with +existing bright objects while limiting the I/O volume required to make +the detections. In general, there will be many fewer {\tt Objects} +than {\tt Detections}, and there will be fewer bright orphans than +faint orphans. + +A wide range of options are available to addstar. These can be used +to modify the object matching rules, to reduce the number of tables +which are updated, to specify the output data format, and so forth. A +few options modify the behavoir in substantial ways, as discussed in +the two sections below. + +\tbd{flesh out discussion of the options} + +\paragraph{Insert Reference Objects} + +\code{addstar -ref (filename)} + +This mode of addstar reads a text file and adds the listed objects to +the database as a reference photcode type. A collection of reference +objects are added to the database as a collection of detections. The +reference photometry should in general be given its own photometry +code. The reference data is different from the image detection set +because the associated image information is not included. Thus, no +corresponding images are added to the database. + +\paragraph{Insert Catalog Objects} + +\code{addstar -cat (name) -region ra ra dec dec} + +In this mode, any of several all-sky or large-scale reference catalogs +are used for the input sources. The catalog objects are added to the +database as reference objects. The valid catalogs consist of 2MASS, +USNO, GSC. Tycho and USNO-B will be added shortly. Specific +photcode names are defined for each of these catalogs, and must be +appropriately requested and defined in the photcode table. The +optional region restriction limits the insert to a subset of the sky. +The user does not always want to add 50GB of 2MASS detections to any +DVO database... + +\paragraph{Addstar Client/Server Interactions} + +DVO currently uses stand-alone programs which are run from the command +line (like addstar, or the programs listed below), or it works with +the interactive DVO shell, which allows the user to query portions of +the database. These programs all interact with the database tables +directly, making use of file locking to prevent conflicts. + +Unlike the other DVO programs (currently), it is possible to run +addstar as a client/server system. In this configuration, the program +\code{addstard} is launched to run in the background as a server. It +monitors a socket waiting for clients to contact it. The client +program, \code{addstarc} appears to the user identical to the +stand-alone addstar. However, rather than directly insert data into +the database, \code{addstarc} contacts the addstar server and sends it +the detections and associated image data (along with the information +about the user options). The daemons accepts the incoming data and +then loads this data into the database, just as the stand-alone +addstar does. + +The purpose of the addstar client/server design is three-fold. First, +the client can be used by processes to send data to the DVO database +and then immediately exit. The addstar loading process is one of the +more time-critical functions within the IPP. However, unlike the +other portions of the IPP, the addstar processes must operate in +serial, at least when they are updating the same portion of the sky +(or the image table). If the IPP analysis routines all needed to run +the stand-alone addstar program, they would eventually block waiting +for each addstar to complete, preventing other processing from +continuing. The addstar client / server model allows the processing +node to invoke the addstar client, sending the data to the addstar +server. The addstar server will then be the entity that manages the +serialization of the incoming data stream. The addstar server has two +threads which run in parallel. One thread monitors the socket and +accepts new data sets from addstar clients, adding the data to an +internal queue. The other thread pulls data off of the queue and +updates the database with the data. + +A second advantage of the client/server interaction is that only the +new detections need to be sent across the network. To update the +database, addstar must load the average objects for the region from +the database tables. In the stand-alone mode, the addstar program +loads this data via NFS across the network from whatever device stores +the addstar tables. In the client/server model, the addstar server +always runs locally on the machine which holds the database tables. +Thus, for the server, all database access is local disk access. + +The final advantage of the client/server model is that it enables the +parallel database model, which is not yet implemented as of Jan +2006. In this model, there are multiple addstar servers. Each one has +a fraction of the sky in the local tables. The identification of +which table is managed by this host/addstar server is stored in the +SkyRegion table. The addstar server simply accepts incoming +detections from the addstar clients. Any detections which it receives +which fall within the boundaries of tables that it manages are updated +as normal. The server then identifies the other addstar servers which +are responsible for the other detections. It then sends these +detections to those servers using the same socket communication used +by the addstar clients. The addstar server must also be ready accept +detections from other addstar servers. This relationship is +completely parallel, and any addstar client may send its data to any +addstar server, letting the servers hash out who owns what. The only +difficulty with this model is in handling sources near the boundaries +of the tables. Note that this issues exists whether those tables are +distributed across multiple machines or not. + +Addstar uses the following strategy to handle detections on the table +boundaries. Detections are first added to each table completely +ignoring the neighboring tables. A detection which is close to the +boundary may either be associated with an average object contained +within the table, or not. If it is, the detection is associated with +that average object. If not, a new average object is created at the +location of the detection. So far, this process is identical to the +behavoir in the middle of the table. One a longer time-scale, a +process is run which mediates the table boundaries. In this analysis, +the two neighboring tables are simultaneously examined. The border +region, in a strip wider than the correlation radius, is examined in +detail. If two objects within the border region fall within 2x the +correlation radius of each other, their individual detections are +re-examined. These detections are re-added to a temporary table which +encompases the overlap. the resulting objects will in general have +detections from either side of the boundary. The average objects are +kept within the table as normal, but the detections are allowed to +migrate between the tables to stay with their object. \tbd{this +boundary cleanup process is not implemented to date}. + +\subsubsection{Relphot : Relative Photometry Analysis} This operation uses the overlaps of images and multiple observations of the same objects to determine the relative photometry zero-points -for a collection of images. This is a task that wil be run much more +for a collection of images. This is a task that is run much more infrequently than the object insertion tasks. -\paragraph{Determine Consistent Photometry Zero Points (uniphot)} +The relphot analysis is currently performed with a single Sky region +as the starting point. All images (or all chips from all mosaic +iamges) which overlap the sky region are identified in the image +table. This set of images are considered set A. Next, all skyregions +which are overlapped by all of these images are selected. Finally, +all additional images which overlapped the new regions only are +selected. These are considered as image set B. The image selections +are also restricted to images of a single, user-selected photcode. + +All of the objects and detections which are contributed by the images +in sample A are extracted from the average and measure tables. Only a +subset of the detections for which the S/N is greater than a +user-selected limit are kept. Other restrictions, such as time range +or instrumental magnitude ranges may also be specified. The +collection of average objects, their detections, and the images from +which they were derived now define a system of photometry equations. +In this system, every image has a calibration offset magnitude +($M_{cal}$), every object has an average magnitude in a relative +system ($M_{rel}$), and every detection of that object has a magnitude +defined by the equation $M = M_{rel} + M_{cal}$. The goal is to solve +for the values of $M_{ref}$ and $M_{cal}$. + +There are two points to note about this operation. First, the system +of equations is generally much too large to solve directly; we must +use an iterative technique to converge on a solution. Second, it is +important in the analysis to use robust averaging and identify +detections, stars, or images which are deviant in some way. These +should be marked and given set weight in the solution. These cases +may represent poorly measured objects (perhaps detections on or near a +bad column), variable stars, and images obtained in poor weather +conditions. + +Relphot can also be used to determine the mosaic grid used to generate +photometrically corrected flats (-grid option). + +\subsubsection{Uniphot : Zero Point Analysis} This operation uses the time history of relative photometry zero points for images and the spatial overlap information to determine a best set of image zero points which have a specific time scale for the -atmospheric stability. - -\paragraph{Determine Distortion and Static Astrometry Model (mosastro)} +atmospheric stability. This analysis would be used after relative +photometry has been determined for data in DVO. This analysis +currently is defined to unify the zero points of a collection of +disjoint regions; additional modifications will be needed to +simultaneously determine consistent zero points and relative +photometry corrections for a collection of images distributed over a +large range in space and time, but still with significant +overlap. distritions with subustanailaccount for the c + +\subsubsection{Global Astrometry Analysis} This operation uses the reference and image detections to determine an @@ -1164,40 +1513,69 @@ ideal flat focal plane. . -\begin{table} -\begin{center} -\caption{AP Detection Classes \& Object Parameters\label{tab:APdetections}} -\begin{tabular}{lrrrr} -\hline -\hline -Object Parameter & P2 & P4S & P4D & SS \\ -\hline -PSF x,y, covar, $\alpha,\delta$ & + & + & + & + \\ -PSF mag, $\sigma_{\rm mag}$ & + & + & + & + \\ -star/gal sep & + & + & + & + \\ -$\sigma_x$, $\sigma_y$, $\theta$ & + & + & + & + \\ -local sky data & + & + & + & + \\ -Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ -S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ -W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ -exp. spaced aps., Poisson noise, variance & - & - & - & + \\ -\hline -\end{tabular} -\end{center} -\end{table} - -\subsubsection{Throughput} - -The AP Database design partly driven by the need to make the -detection-object associations quickly and to processes the incoming -detections at a sufficiently high rate to meet the throughput -requirements. For each upload of the object detections from a -complete FPA, the AP Database must match roughly $1.4 \times 10^{6}$ -detections from an FPA with roughly $6.4 \times 10^{6}$ objects, -including orphaned bright detections. This corresponds to roughly 640 -MB, if each object uses 100 bytes for its descriptive informations -(more than is currently specified in the Object table). With a -throughput of 100 MB/s for reads from a RAID, the AP Database can -perform the data read in a fraction of a second if the data is -distributed across 10 computers. +\subsubsection{DVO shell} + +The DVO Shell is a user-tool for examining the visualizing data stored +by DVO. The DVO Shell uses the Opihi shell language structure (see +also PanTasks, Section~\ref{pantasks}), which provides a rich data +analysis language. The shell language provides the user with +capabilities to define new commands, set and manipulate scalar, +vector, and image structures, and plot 2D graphics, including +projections of the sky. In addition, the DVO shell is aware of the +DVO data tables and provides access mechanisms to these tables. The +following is a brief overview of these table access features. + +DVO provides several ways to access the information stored in the +database. Several simple commands allow the user to extract 1-D +information directly from one of the primary database tables. The +fundamental such commands are: +\begin{itemize} +\item imextract +\item avextract +\item mextract +\end{itemize} + +These commands allow the user to extract data from one of the columns +represented by the image table(s), the average object tables, or the +measurement tables. The extraction places the resulting data into a +vector data elements, which may be used to make plots or perform +analysis. The user may constrain the query with spatial selection, by +photcode, by time ranges, and so forth. Some examples: +\begin{verbatim} + avextract all ra : select ra for all objects in displayed region + avextract all g : select g magnitudes + avextract all mag -photcode r : select r magnitudes + avextract all Xm -photcode r : select chisq values for r average mags +\end{verbatim} + +Beyond these basic vector extractions, the user may perform more +complex extract operations such as color-based selections. For +example, color-color diagrams can be easily made by extracting the +colors from the average or measurement tables and plotting the +resulting vectors. The \code{ccd} commands extracts a specified pair +of colors for all objects with that color pair from the specified data +region. Similarly, the command \code{cmd} extracts a color and a +magnitude into a pair of vectors. Both commands may specify any of +the different types of magnitudes (relative, calibrated, etc) +discussed above. + +An additional class of DVO Shell commands perform more complex +graphical operation. For example, the command \code{images}, plots +the images which the specified region on the plotting too. Other +commands allow the user to extract the images or the database tables +which overlap specified locations. + +\begin{figure} +%\resizebox{4.5in}{!}{\includegraphics{pics/polar}} +\caption{\label{polar} \small Map of +the sky in polar project, and images added to database. } +\end{figure} + +\begin{figure} +%\resizebox{4.5in}{!}{\includegraphics{pics/fullsky}} +\caption{\label{allsky} \small Map of the entire sky, and images added to database. } +\end{figure} + +Some examples of using the DVO shell to perform visualization are +given in Figures~\ref{polor} and \ref{allsky}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1924,14 +2302,9 @@ up the analysis stages are touched upon in Section~\ref{sec:PanTasks}, which discusses the IPP Scheduler program, PanTasks. They are -discussed in more detail in the document 'ippTools'. +discussed in more detail in Section~\ref{ippTools}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Phase 1: image processing preparation} - -\tbd{need to add a discussion of Phase 0} - -\tbd{need to incorporate a discussion of ppImage, etc as distinct from - the ``phases''} The Phase 1 analysis stage is performed on each science exposure (each @@ -2794,19 +3167,4 @@ year, or an average rate of $\sim$2 Mpix per second, or $< 1$\% of the object analysis in the other analysis stages. - -\section{IPPtools} - -Above, we discussed PanTasks, the IPP scheduler which determines the -new jobs to run and distributes them to computers across the network. -PanTasks is a general tool; by it self it does not define the specific -analysis tasks that the IPP requires. The previous few sections -discussed in detail the analysis which is performed by the IPP -analysis stages. IPPtools is the collection of PanTasks scripts, -Metadata Database interaction programs, and other tools used to -definet the specific analysis stages of the IPP. - -\tbd{this section needs to be fleshed out with a summary of the - ippTools functions. The stand-alone IPPTools document gives a - detailed discussion of these issues}. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -3254,4 +3612,1108 @@ stage and the image combination stage with robust outlier rejection. \tbd{Paul: flesh this out!} + +\subsection{Command Sequences} + +It is useful in order to understand the analysis sequence to examine +the complete series of processing steps involved in the analysis +stages discussed above. We first illustrate the Phase 1-3 sequence, +giving the commands which start with a raw image available on disk and +results in a collection of detrended chip images, a high-quality +astrometric calibration, and a collection of object detections. + +\subsubsection{Phase 1-3 Analysis Commands} + +In the example below, we imagine a GPC image available on disk with +the exposure ID 654321.obj and chip IDs 00 through 88. The IPP design +does not mandate specific naming convensions for the exposure IDs and +the chip IDs; these values are opaque strings supplied by the data +source (eg, GPC). Below, in Section~\ref{ipptools}, we discuss how +the names and inputs are constructed, and how the relationships are +tracked between an exposure and the data containers which make up the +exposure. Also, the details of directory naming and organization are +just examples, though some nightly folder scheme is a likely option. +For the moment, these are assumed to be known by the system + +Also discussed in Section~\ref{ipptools} it the concept of multiple +analysis passes for a data element. Within the IPP, any data may be +processed multiple times; the system tracks each attempt to process a +particular set of data, tracking the analysis versions numbers which +increment sequentially for each new attempt. In the sequence below, +we are performing the first analysis attempt on the data, so the +version numbers are all 0. + +\begin{verbatim} +Phase 1: + ppImage -recipe PHASE1 + -inglob file:/data/2006.11.01/7654321o/7654321o.??.fits + -out_astro file:/data/2006.11.01/7654321o/7654321o.XX.P1.00.ast.fits + +Phase 2: + ppImage -recipe PHASE2 + -infile file:/data/2006.11.01/7654321o/7654321o.24.fits + -in_astro file:/data/2006.11.01/7654321o/7654321o.XX.P1.00.ast.fits + -outfile file:/data/2006.11.01/7654321o/7654321o.24.P2.00.img.fits + -outmask file:/data/2006.11.01/7654321o/7654321o.24.P2.00.msk.fits + -outvar file:/data/2006.11.01/7654321o/7654321o.24.P2.00.var.fits + -objects file:/data/2006.11.01/7654321o/7654321o.24.P2.00.cmf.fits + -thumb file:/data/2006.11.01/7654321o/7654321o.24.P2.00.thm.fits + -binned file:/data/2006.11.01/7654321o/7654321o.24.P2.00.bin.fits + +Phase 3: + ppImage -recipe PHASE3 + -inglob file:/data/2006.11.01/7654321o/7654321o.??.P2.00.cmf.fits + -out_astro file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.ast.fits + -objects file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.cmf.fits + + ppImage -recipe MOSAIC + -inglob file:/data/2006.11.01/7654321o/7654321o.??.P2.00.thm.fits + -outjpeg file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.thm.jpeg + + ppImage -recipe MOSAIC + -inglob file:/data/2006.11.01/7654321o/7654321o.??.P2.00.bin.fits + -outjpeg file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.bin.jpeg +\end{verbatim} + +In this example, the data are supplied by their file names in the UNIX +file system. We are invoking the capability of ppImage to accept a +glob to supply a list of files. In the Phase 1 stage, ppImage is used +with the PHASE1 recipe to load data from a full mosaic (the files +specified by the glob) and produce a single astrometry calibration +file. The \code{XX} is supplied for the full-mosaic output files just +to make the output data products appear in a more easily readable +fashion is a directory listing. Also, note that we attach the +\code{.fits} extension to all of these output files to make the data +type more explicit to the reader. Again, none of these convensions +are required by the analysis programs. + +The Phase 2 analysis uses the Phase 1 astrometry to improve the +astrometric starting guess. The analysis for Phase 2 is illustrated +for just a single chip (24), though presumably equivalent commands are +executed for the other 63 chips. The six output files selected in +this example include the detrended image (\code{*.img.fits}), along +with the corresponding mask (\code{*.msk.fits}) and variance images +(\code{*.var.fits}) and the photometry results file +(\code{*.cmf.fits}). Note that the output object file contains the +astrometric solution parameters from this analysis stage. This +process also constructs two smaller version output images: the binned +image (\code{*.bin.fits}) and the thumbnail (\code{*.thm.fits}). The +binning scale for these images is specified in the recipe for the +camera; the first of these for GPC would likely be binned 32x32, while +the second would probably be binned 320x320. + +For the Phase 3 analysis stage, three actual analyses are illustrated. +In the first, the photometry results files are identified by file glob +and the result is an improved astrometric model for the camera and +optics in our astrometry parameter table format. The object files are +also grouped into a single multi-extension file along with the +astrometric calibration. + +In the second and third analysis examples, the collection of chip +binned and thumbnail images are loaded by file glob, mosaic-ed +together into a single image, and written to disk as a JPEG. These +images are used by the pipeline tracking tool, ippMonitor. The binned +image results in a full GPC image represented by 1200x1200 pixels; the +thumbnail yields a 120x120 representation of the full GPC. + +The example above would be sufficient if we were processing a small +number of images by hand or for test purposes. However, the IPP is +designed to be more flexible about the physical location of the data +files that this illustration permits. The use of nebulous allows us +to use a similar naming scheme and yet place the actual data files on +different hardware depending on the chip ID (among other +possibilities). To convert the filename version above to a version in +which the files are stored on Nebulous simply requires changing the +\code{file:} tag to \code{neb:}. The analysis programs recognize this +tag to indicate a file available from Nebulous, and make a request to +Nebulous for the actual file names. Nebulous can supply files based +on a name match much like the file glob. Nebulous also allows the +storage object ID to include path-like elements, allowing a structured +organization of the files within Nebulous (which does not reflect a +{\em physical location} relationship). + +\subsubsection{Basic Detrend Creation Commands} + +In the following example, we examine the steps to produce master +detrend images. First, a few important points to note about this +process. The construction of a master detrend frame (bias, flat, etc) +involves combining a number of individual frames of an appropriate +type of exposure, possibly after some preparatory processing. For +example, in building a twilight flat-field image, 5 or 10 (or however +many) raw flat-field images are first masked and bias corrected before +being combined. To build a night-time fringe-frame image, a +collection of raw night-time images are bias, dark, and flat-corrected +before they are combined. In the combination, it may be necessary to +apply some scaling and/or offset correction to the images to place +them on a common footing. For example, in the construction of a +master flat-field image, the individual images must be normalized in a +consistent fashion; in building a master fringe frame, the fringe +amplitude must be used as part of the scaling applied to the input +images. In a mosaic camera, if individual chips are analysed +independently, the resulting master chip images may require +re-normalized to place the results on a common, consistent footing. + +Beyond the details of the analysis steps, there is the question of the +choice of input images. This choice is extremely dependent on the +implementation for a particular camera, telescope, type of detrend +image, etc. The analysis {\em process} should not be designed to make +strong assumptions about the selection of the input data. In the IPP, +the definition of the selection rules is part of the input +configuration information and the scheduling rules, and can be +considered outside of the discussion of the analysis commands. The +IPP provides a tool, part of the \code{dettools} suite, which examines +the Metadata Database tables for raw images of the appropriate type to +select input images based on selection options such as time range, +filter, camera, chip, exposure type, airmass, exposure time, etc. In +the discussion below, we assume that some selection is made with +\code{dettools}, resulting in a collection of input exposures and +their corresponding chips. These lists are placed in tables which are +then provided as part of the input to the analysis programs below; the +corresponding images used as part of these inputs are also saved in +Metadata Database tables as discussed in Section~\ref{ipptools}. In +practice, the database tables provide the primary source; the list +files are constructed from these tables and are simply intermediate +data sources for the analysis programs. + +Another important distinction to clarify in the detrend processing is +between the detrend {\em run}, the {\em version}, and the {\em +iteration}. These issues are discussed further in +Section~\ref{ipptools}. Briefly, though, there are the following +concepts to keep in mind: The detrend {\em run} is a particular +attempt to construct a master detrend image. One run defines a +collection of selection criteria for the initial set of input images. +The resulting master detrend image is given an identifier, equivalent +to the exposure ID. If the same selection criteria are used multiple +times (eg, for multiple experiments on the analysis recipe used to +construct the image), the same detrend ID may be used for multiple +detrend runs. In this case, each new detrend run is given a different +{\em version} number, equivalent to the version numbers used to track +the science analysis passes. For the detrend image construction, this +concept must go one level further, however. In order to produce a +single validated master detrend image, it is necessary in general to +produce multiple intermediate attempts. The intermediate master +frames are applied to the input images; the statistics of the residual +images are then used to select a subset of the input images, rejecting +poor quality or deviant images. This processing is a form of +image-level outlier rejection, and is particularly necessary for input +images which result from observations of the sky (eg, twilight flats +or night-time fringe frames); images obtained using stable calibration +sources may not require this level of iterative processing. This type +of analysis can also be used to determine if a new master frame is +needed (all input images internally consistent, but deviant from the +current best master) or if conditions are unacceptable to produce a +new master (all input images mutually inconsistent). In order to +track these multiple analysis passes, the IPP infrastructure assigns +iteration numbers to the data products associated with a particular +detrend run and version. + +One final point to address is the issue of the validity domain of a +detrend image. The end result of the detrend run is a master detrend +image of a particular type, e.g., a master r' flat-field image for +GPC-1. As a result of the input selection criteria, the resulting +master detrend frame will have a primary domain of validity, which +consists of a particular camera, telescope, set of chips, and which +may include a time range, filter, airmass range, etc. The primary +domain of validity defines those images which would be best processed +with the particular master detrend image. Beyond this primary domain +of validity is a wider, relaxed domain of partial validity. + +Clearly, a SkyProbe flat-field image would be inapporiate in all +context to be applied to a GPC image. Likewise, a GPC-1 $r'$ flat would +be inappropriate for a GPC-1 $g'$ science image. However, in some +circumstances, it is appropriate or desireable to apply a detrend +frame to an image outside of its primary domain of validity. For +example, if a master flat-field image was produced using input images +from a certain week, an image from a different week may be viewed as +outside the primary domain; however, for some experiments or because a +flat-field in the appropriate time range could not be produced, it may +be acceptable to apply the out-of-date flat-field image to the science +image. In general, any image of the appropriate type, camera, filter +(if a valid construct) and detector is in the partial domain of +validity as a detrend image with those same values. The detrend +creation system assigns a primary domain of validity to the masters +which it creates; it is the choice of the analysis routines to apply +images from a more relaxed domain, if necessary or desired, or to +require the primary domain and yield an error if it is not available. + +Below we give the series of analysis commands used to construct a +master detrend frame. In this example, we construct a master $r'$ +flat-field image for GPC1, using the detrend images for week 050. In +practice, we will likely build detrend frames on a nightly basis, but +the choice of timescale will depend to some extent on the observing +process and the stability of the system. In this example, we +construct a detrend ID using the camera, type, filter, and week +number, though this choice is completely arbitrary. We also +illustrate the example for one of the input images with exposure ID +7654321f and for chip 24. In this example, this is the third time +this image has been used for the analysis, thus the processing results +for this frame/chip are given a version number of 02. + +\begin{verbatim} +dettools -define [selection criteria] -detID GPC1.flat.r.w050.00 + +ppImage -recipe MKDET.PROCESS + -infile file:/data/2006.11.01/7654321f/7654321f.24.fits + -output file:/data/2006.11.01/7654321f/7654321f.24.PC.02.img.fits + +ppMerge -recipe MKDET.STACK + -inlist file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c24.list + -output file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c24.fits + +ppNorm -recipe MKDET.NORM + -inglob file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c??.fits + +ppImage -recipe MKDET.RESID + -infile file:/data/2006.11.01/7654321f/7654321f.24.PC.02.img.fits + -output file:/data/2006.11.01/7654321f/7654321f.24.RS.06.img.fits + -thumb file:/data/2006.11.01/7654321f/7654321f.24.RS.06.thm.fits + -binned file:/data/2006.11.01/7654321f/7654321f.24.RS.06.bin.fits + +ppImage -recipe MOSAIC + -inglob file:/data/2006.11.01/7654321o/7654321o.??.RS.06.bin.fits + -outjpeg file:/data/2006.11.01/7654321o/7654321o.XX.RS.06.bin.jpeg + +ppImage -recipe MOSAIC + -inglob file:/data/2006.11.01/7654321o/7654321o.??.RS.06.thm.fits + -outjpeg file:/data/2006.11.01/7654321o/7654321o.XX.RS.06.thm.jpeg + +ppImage -recipe MKDET.MOSAIC. + -inglob file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c??.fits + -outjpgt file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.thm.jpeg + -outjpgb file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.bin.jpeg + +\end{verbatim} + +Again, this data illustrates the use of files in the UNIX file system; +the substitution of \code{neb:} for \code{file:} will inform the +programs to retrieve the file names via Nebulous. In this example, we +use output names for the intermediate images which are equivalent to +those used for the science processing; the version numbers used for +these data products are sequential over all detrend runs which use +those input frames. Also, for the detrend image products, we have +added the 'v', 'n', 'c' tags to clarify which of the two-digit numbers +represents a version (v), an iteration (n), and a chip ID (c). Note +that this sequence of analysis steps makes heavy use of +\code{ppImage}, with different choices of the recipe and the output +options to change the behavior somewhat. However, all of the uses of +\code{ppImage} represented here are consistent with the primary +responsibilities of \code{ppImage}: read in the file or mosaic image +into the correct level of the image hierarchy, perform a detrend +analysis (including rebinning in this category), re-structure the +collection of image arrays as described by the recipe, write out the +image in the desired format. + +\section{IPPTools} + +PanTasks is the IPP tool which manages the sequencing of data analysis +steps and, with the related tool `PControl', distributes the data +processing across a cluster of computers. However, by itself, PanTasks +does not determine the organization of data or the analysis sequencing +for a particular pipeline. This level of information is contained +within specific PanTasks scripts. To use the tasks defined by a set +of PanTasks scripts, additional helper programs are needed. This +section discusses these programs and the PanTasks scripts used by the +IPP. + +IPPTools is a collection of programs, Metadata Database table +definitions, and PanTasks scripts used to define the actual data +organization and the sequencing of operations by the IPP. Within the +IPP, the Metadata Database is used to store the analysis state, as +well as result processing data points. This section discusses the +tasks needed to define each of the IPP analysis stages (Phase 1-4, +detrend creation, etc) and examines the relevant MDDB tables. + +\subsection{Persistent vs Ephemeral State in PanTasks} + +\begin{figure} +\begin{center} +\includegraphics[scale=0.85]{pics/ipptools.01.ps} +\caption{\label{queues} PanTasks queues and MDDB tables} +\end{center} +\end{figure} + +The IPP, a fairly complex analysis system, uses PanTasks to select +jobs, distribute them to the cluster, and harvest the results. It +uses the Metadata Database to record the results of a given analysis +step, and to determine which jobs must be performed when. + +There are some subtleties in the interaction between PanTasks, the +Metadata Database tables which store the system state, and the jobs +which are currently being performed. There is a choice to be made +between rigorously maintaining the system state in the Metadata DB at +all times or keeping an intermediate set of state tables. Keeping the +exact system state in the Metadata DB tables would require many extra +queries to/from the database and may introduce additional latencies +which are undesirable. This is because any attempt by PanTasks to +initiate a new job would require PanTasks to mark the corresponding +data item in the Metadata DB (the item which acts as the trigger) with +a `pending' state, and then mark it again as `done' when the job +actually completes. This also has the drawback that, if the system +crashes (eg, hardware failure), some initial process would be required +on start up to find all Metadata DB items which are in the `pending' +state (examining all possible items which can be in such a state) and +reset them to the `new' state. + +We implement an alternative in which PanTasks maintains an internal, +ephemeral stack of the pending jobs, and only updates the system state +entries in the Metadata DB when jobs are actually completed. In this +scenario, as far as the Metadata DB tables are concerned, data items +transition only between a `new' and a `done' state. Any jobs which +are pending when the system crashes or the power is lost are simply +dropped, and will be automatically re-constructed when the system +restarts. In this paradigm, no intermediate operation state is saved, +and no partially completed job can be recovered. Since the IPP is +defined in terms of a fine granularity, with jobs lasting no more than +30 - 120 seconds, crashes under this model will not have a large +impact on the data processing. + +Figure~\ref{queues} illustrates this ephemeral vs persistent state +information and the interrelation between the metadata tables and +PanTasks. The left-hand portion of the diagram illustrates the +recommended interaction between the metadata database tables and +PanTasks' internal queues. Some table in the metadata database +defines a list of data items which are to be processed by some +analysis job. PanTasks uses a two-step approach to define the +analysis jobs based on this list. First, one task queries the MDDB +for a list of pending items, adds the returned items to an internal +PanTasks queue. The process of adding the elements to the queue is +defined so that only unique items are added: already existing items +are skipped. The entries in the queue consist of the data items of +interest and an internal temporary state. At first, this would be +`pending'. A second tasks pops `pending' entries one-by-one from this +internal queue, submits a job based on the entry, and sets the +temporary state in the internal queue to `running'. The internal +state is needed to prevent PanTasks from re-submitting a job for the +same data item before the first job is done or assessed. Since the +job make take an arbitrary amount of time, PanTasks requires a +mechanism to remember which data items it has already submitted. When +the job eventually completes, the metadata database table is updated +noting the completion. This may be done either by the job itself or +by PanTasks as part of the job exit rules. In addition, the state of +the entry in the queue can be set to either `done' or the entry can be +simply removed from the queue. + + +The purpose of this interaction is to maintain the temporary state +information within non-persistent elements of PanTasks rather than +using the metadata database tables to store this information. This +concept has two advantages. First, PanTasks internal queues are in +memory and relatively small, thus interfacing with them is quite fast +for PanTasks -- this should reduce the system latency. Second, by +keeping this information non-persistent, the system responds correctly +to stopping and restarting PanTasks. Any jobs which have not been +completed will not be marked in the database, and will be restarted +naturally by PanTasks. The alternative, of writing a temporary state +marker in the database would require PanTasks, on startup, to +initially clean all database tables of these temporary state markers. + +The right-hand portion of the diagram illustrates this process using +the process of copying the images from the summit as an example. The +metadata database table of interest in this case is the list of +pending images, with entries supplied by a job which queries the +summit data systems. The job which is actually performed is a remote +copy of the image file from the location specified by the summit data +system to the appropriate location within the IPP Image Server +(Nebulous). (As an alternative to the above, the `pending images' +table may be part of the summit database system, and the `get images' +command may query the summit directly. In this scenario, the `copy +image' command reports to the summit data system that an individual +image file has been copied.) + +In the rest of this document, the use of PanTasks internal queues to +manage the temporary data states is glossed over and assumed part of +the tasks defined in the process. + + +\subsection{IPP Pipelines Overview} + +The IPP as a whole performs all of the image analysis functions +required by the Pan-STARRS telescopes, including images from the full +Gigapixel camera (or cameras), the test camera TC-3, and the SkyProbe +camera. The IPP is designed to be very flexible, with instrument +specific details isolated in configuration files associated with the +different cameras known to the system. As a result, the organization +of the top level analysis infrastructure must be sufficiently general +that a wide range of cameras can be accomodated. We have a few +general principles regarding constraints on the data to be processed +which are used to guide the IPP design and developement: + +\begin{itemize} +\item {\bf Camera Focal Plane Hierarchy} The IPP analysis programs + assume that the images to be processed are obtained by a camera + which can be represented by our Camera Focal-Plane Hierarchy of data + structures. This hierarchy is discussed in detail in the Modules + SDRS, and defines a top-level {\em Focal-Plane Array (FPA)}, which + may contain 1 or more {\em Chips}, each of which may contain one or + more {\em Cells}. An {\em FPA} is identified as having a single + optical system feeding photons to the detectors. A {\em Chip} is + identified as a unit of data all deriving from a single detector + (piece of silicon), while a {\em Cell} is identified as a collection + of pixels read out as a continuous cartesian grid. Finally, a + single collection of data from an {\em FPA} may include multiple + {\em Readouts} from any or all of the {\em Cells}. + +\item {\bf Exposures vs Groups} The processing presumes that the data + is organized into {\em exposures} and exposure {\em groups}. An + exposure represents the data from a single FPA, with the possible + subdivision of the exposure into multiple readouts for some or all + of the cells. Exposure {\em Groups} are any group of exposures + which are related together in some way; the definition of the {\em + Groups} may be provided by the observers, or they may be derived + from the characteristics of the exposures. The use of a particular + {\em group} depends on the context of that group. A few examples of + exposure groups: + + \begin{itemize} + \item a dithered sequence of exposures to be stacked for cosmetics + and improved signal-to-noise. + \item a twilight flat-field sequence. + \item all images of the same filter within a 10 degree region to be + used to construct an sample astrometric reference. + \end{itemize} + +\item {\bf Image Files (imfiles) vs Exposures} Any single exposure + may consist of a number of different data files. The number of {\em + imfiles} for a given exposure will depend on the camera, as will the + data organization within those image files. Also, a particular + camera will supply files corresponding to one of the particular + Focal-Plane Hierarchy elements. The IPP analysis must be able to + interpret the incoming data correctly. +\end{itemize} + +As discussed elsewhere, there are several major types of analysis +performed by the IPP. For the purposes of data organization and +parallel processing efficiencies, we have identified the following +divisions of the analysis tasks. These will be discuss in much more +detail below. + +\begin{itemize} +\item {\bf Science Image Analysis} : This represents the analysis + performed on the images obtained by the telescope, and generally + performed in real-time, night-by-night. The science image analysis + tasks are further subdivided as follows: + + \begin{itemize} + \item {\bf Phase 1} : The full focal-plane array is examined quickly + to determine an initial astrometric calibration. In this step, the + OTA guide stars may be used as the astrometric reference; if none + are available, predicted bright star positions are examined. This + step is only used for mosaic images, and may be skipped if no guide + stars are available {\em and} the astrometric calibration for the + telescope / camera is reliable (better than 10 arcseconds). + + \item {\bf Phase 2} : Each image file is analysed independently: the + image is detrended (bias, dark, flat, fringe, etc), sources are then + detected to a modest level, improved astrometric calibration is + performed. + + \item {\bf Phase 3} : The collection of sources measured from all of + the image files for the camera are used to determine a global + astrometric, and possibly photometric, solution for the exposure. + This step is only required for mosaic cameras. + + \item {\bf Phase 4.1} : An exposure group consisting of images + obtained in a specific region of the sky are merged together. In + this step, the images are first warped to a common pixel grid, defined by + the static sky images. The collection of images are then used to + construct a single, cleaned image by rejecting the outliers from the + source images in the stack. The corresponding static sky pixels are + then used to construct a difference image from the resulting stack. + + \item {\bf Magic} : In this step, the difference images are examined + to find the trailed images introduced by artificial satelites. + These so-called {\em streaks} are excised from the difference + images, as well as all of the source images which were used to + generate the difference images; the public data sources are updated + with the precise, correct time. Note that this step requires that + separate difference images be generated for each of the input + images, a step which would be skipped if {\em magic} were avoided. + Also note that, until {\em magic} is performed, the publically + available time has a limited precision (probably $\sim 1$ minute + errors). This step is only necessary in the operational IPP system + given the restrictions from the Air Force. + + \item {\bf Phase 4.2} : After {\em magic} the final difference and + the final cleaned stacked image are produced and objects in both + images are detected. The difference sources are used to mask the + extreme outliers in the cleaned stack, which is then used to update + the Static Sky images. + \end{itemize} + + \item {\bf Static Sky Image Analysis} : While the science image + analysis is performed as images are availablef, the static sky image + analysi occurs on a very different timescale. In steady state, the + full static sky analysis will take place over the course of a full + year. At any given time, the portion of the sky corresponding to the + location of the sun will be under-going the analysis. In practice, + for PS-1, the static sky is produced in a somewhat different fashion + than in the steady-state model. In PS-1, the different survey + strategies introduce very different update rates for the static sky. + At one extreme, the AP Survey will not have enough data for a + complete static sky analysis until nearly 22 months after the survey + begins. At the other extreme, the deep survey, which observes a much + smaller portion of the sky, may best be analysed quite frequently. + These details are part of the science guidelines of the PS-1 surveys, + and are beyond the scope of this document. Rather, the IPP Static + Sky Image Analysis must provide the capability of defining the static + sky analysis in a flexible and dynamic fashion. + +\item {\bf Basic Detrend Creation Analysis} : The analysis of most of + the detrend data is grouped together in a common analysis stage. + The differences between the analysis of the bias, dark, flat, and + fringe images is primarily one of how the input images are + pre-processed, what statistic is used to characterize a given input + image, how the input images are scaled before being combined, and + what normalization is applied to the resulting image. All of these + types of detrend images can thus be processed with a single analysis + pipeline which is made aware of these minor differences. This stage + is never the less fairly complex, and as a result is subdivided into + several compenents, as discussed below. + +\item {\bf Other analyses} There are a number of other tasks which the + IPP must perform that are not well-defined by the different analysis + types discussed above. Some analysis tasks are not automatically + triggered, and are thus outside the scope of this document; these + are the tasks which are more properly considered as research + projects than analysis systems. The other important automatic tasks + are: + \begin{itemize} + \item {\bf Summit Copy} : In this stage, the data source or data + sources are queried for new exposures and image files, which are + then copied to the IPP data area. This stage also includes the + copying of other metadata which are not included in the image + files. + + \item {\bf Image Classification} : new images which are introduced + to the IPP are examined by this analysis stage and placed in the + appropriate table for processing. This step includes a small + amount of accumulating statistics about the images. + + \item {\bf Data File management} : a few tasks are necessary to + monitor and maintain the clustered storage system. These tasks + include the automatic duplication and deletion of different types + of files from Nebulous, the file storage archive. This also + includes automatic redistribution of machine assignments as + hardware is added or removed from the system. This collection of + tasks also includes monitoring of system parameters to alert + people in case of dangerous hardware situations. + + \item {\bf Irregular Calibration Data} certain types of + calibration information is extracted on different intervals from + the more regular detrend images. These types of calibration data + include improved telescope pointing models, astrometric + calibrations, photometric calibrations, flat-field correction + frames. + \end{itemize} +\end{itemize} + +\subsection{Tables, Tasks and Tools} + +The following sections discuss the database tables, the tasks within +PanTasks, and the collection of programs used by PanTasks to examine +and manipulate the state tables. These later programs do not, in +general, perform any in depth analysis; instead they perform actions +such as selecting from one table images ready for analysis in a +following processing step. This collection of tools is grouped under +the name of the {\tt ippTools}, and consists of a separate tool for +each of the different major analysis steps. + +The {\tt ippTools} make use of {\em glueforge} to simplify the +management of the database table schema. Glueforge provides a single +mechanism to generate a collection of C data structures, database +tables, database access APIs, and I/O routines from a simple table +description configuration file. All APIs generated by {\em glueforge} +for the same type of interaction have common naming schemes. This +technique has several important advantages. It makes the writing of C +database interactions very quick and easy. It also makes it easy to +modify the database schema without disrupting the software +development. Finally, it provides a simple, self-documenting source +for data structure of multiple types which can be shared between +programs or platforms. + +Within the following diagrams, we illustrate the database tables used +to track the state of the IPP. We also show the commands provided by +{\tt ippTools} to connect the tables. Finally, we show the IPP tasks +which initiate the different analysis steps. The following set of +diagrams uses several consistent features. The blue-and-grey +rectangles define the metadata database tables. The blue section +contains the table name, while the grey section lists a minimal subset +of the table columns. The ellipses represent programs (or program +portions in some cases) executed by PanTasks. The blue filled +ellipses represent the {\tt ippTools} commands which are executed +locally on the computer hosting PanTasks. The grey-blue ellipses +represent the commands executed on the parallel cluster, monitored by +{\tt pcontrol}. The green ellipses represent commands executed by +hand for testing and manual intervention. + +In most of the analysis tasks, we use a two-table approach to the data +in order to avoid excessive latencies. One table is used to track +quantities which are still pending for a particular stage. When the +analysis is completed, these items are moved from the 'pending' tables +to corresponding 'done' tables. Although this introduces a somewhat +higher number of tables and complexity, it will avoid the system from +slowing down as the number of data items grows with time. The pending +tables are searched repeatedly by the {\tt ippTools} programs as they +attempt to select new data of interest. In contrast, the done tables +are searched much less frequently. + +\subsection{Summit Copy Tasks} + +\begin{figure} +\begin{center} +\includegraphics[scale=0.85]{pics/ipptools.02.ps} +\caption{\label{pcopy} Summit Copy Tasks} +\end{center} +\end{figure} + +Figure~\ref{pcopy} illustrates the MDDB tables used to copy data +(images and metadata tables) from the summit. The left-hand portion +of the diagram shows the tables involved in copying images from the +summit system. The table of pending image files lists the URLs of the +individual image files available for transfer, along with their +associated exposure ID and the camera which generated the image. Two +other entries assist in interpreting the file: the class and the class +ID. The final entry in this table is the current copy state of the +file, can have the value of `ready' or `copied'. + +The class defines the data grouping represented by this image file and +may have values of: FPA, Chip, Cell. This value indicates that the +provided image file represents the specified portion of the camera +FPA. If the value is FPA, the file represents data from a complete +FPA, though the file may contain pixel data in multiple extensions or +other groupings to be identified later. If the value is chip, the +file contains only data for a single chip, presumably of multiple +chips available, and equivalently for Cell. Further discussion of the +FPA image hierarchy is given in the IPP documents (eg, Modules SDRS). +The class ID gives the identifier used to name the class level +corresponding to this file. This value is necessary to make decisions +on how to copy the data based on the chip / cell before the data is +available to IPP components. Table~\ref{classes} lists likely values +for the class and class ID for some common cameras. The system +described is sufficiently flexible to allow us to transfer the GPC +images by cell if we eventually decide that is more efficient. + +The copy process copies the file from the given URL to the appropriate +IPP node and adds an entry to the table of new image files, consisting +of the same information as the pending image file table, though with a +new value for the URL. This URL may be an explicit filename, a +reference to an entry in the image server, or a web address, or +located on the image server (marked with file:, neb:, and http:, +respectively). (TBD: other possible file storage types? perhaps the +path could be abstracted without going to the level of the image +server? eg: ref:DIR0001/file0001.fits might be in a directory which +is defined in a table of directories.) After an image file is +successfully copied, the corresponding state in the `pending chip' +table is updated from `ready' to `copied'. + +\begin{table} +\begin{center} +\caption{Camera and Data Classes\label{classes}} +\begin{tabular}{llll} +\hline +\hline +camera & class & classID \\ +\hline +GPC & chip & chip02 \\ +skyprobe & fpa & sp01 \\ +Megacam & fpa & MegacamSpliced \\ +Suprime & chip & chip0 \\ +\hline +\end{tabular} +\end{center} +\end{table} + +The right hand portion of this diagram illustrates the process of +copying a metadata table. The table of pending tables lists the URLs +for the tables which are ready, a unique table ID for each table, and +the table type. The copy function copies the listed table and uploads +the data to the IPP version of the same metadata database. Two +examples of metadata tables needed by the IPP for the basic image +processing system are illustrated: the table of new exposures and the +table of pending matches. The first lists the exposures which are +avilable from the summit system, and all represent entries which are +available from the Image server. the second represents the matches +between exposure IDs and chips + +\subsection{Phase 0} + +\begin{figure} +\begin{center} +\includegraphics[scale=0.85]{pics/ipptools.03.ps} +\caption{\label{phase0} Phase 0 Tasks} +\end{center} +\end{figure} + +Figure~\ref{phase0} illustrates phase 0, in which the image files are +categorised, examined for summary information and basic statistics, +and moved to the later phase 'pending' tables to trigger further +analysis. The command {\tt p0search -pending} examines the `new +imfiles' and 'new exposure' tables. It selects images from this table +which have not yet been examined (state is `new'). These are returned +to PanTasks, which sends each image file to a separate analysis node +running the {\tt p0search -update} command. With this command, the +file header is examined and relevant metadata is extracted (eg, RA, +DEC, times, and so forth to be defined later). The process may also +select a portion of the image pixel data to determine a rough bias and +background level. These statistics, whether derived from the header +or the pixel values, are placed along with image summary information +in the `raw image files' table, and the state field of the `new image +files' table is set to `ready'. + +The {\tt p0search -update} command is also responsible for moving the +exposures to the tables used for triggering the analysis process. If +the image class is FPA, the image can be advanced without waiting for +any other image files. If the class is Chip or Cell, the process must +also examine the `new exposure' table for this exposure ID. The +number of class files available for this exposure is listed in this +table. The process must the select all image files matching the +exposure ID with state of `ready' and compare the number avalable to +the number expected. If the two match, then a new exposure is ready. +Based on the image type (from the most recently examined image file +header or new exp table?), the exposure is added to the `raw exposure' +table for images of that type. The allowed types are `detrend', (all +bias, dark, flat images), `object', `focus'(??), etc. (** The +different tables represent different analysis modes. This process +also adds an entry to the exp ID / image file match **). This process +also adds all science (OBJECT) exposures to the P1 exposure table (for +mosaic data) or the P2 chip table (for single detector data). These +tables are used to trigger the Phase 1 and Phase 2 analysis stages. + +\subsection{Phase 1} + +\begin{figure} +\begin{center} +\includegraphics[scale=0.85]{pics/ipptools.04.ps} +\caption{\label{phase1} Phase 1 Tasks} +\end{center} +\end{figure} + +Figure~\ref{phase1} shows the tables involved in running the Phase 1 +analysis stage. There are paths for exposures to enter the analysis +automatically from the Phase 0 analysis (arrow on left) or to be added +manually based on a selection from the raw exposure table. Exposures +to be analysed by Phase 1 are added to the P1 exposure table with the +state `new'. Exposures may be added multiple times for processing and +reprocessing. The P1 done exposure table keeps a record of the old +attempts for debugging and analysis. Each time an exposure is added +to the P1 exp table, it is given a new, unique version number, +allowing the system as a whole to track different analysis attempts. +This method is used in all of the image analysis stages (and +extrapolated to iterations in the detrend analysis steps below). The +top portion of the diagram shows the use of the command {\tt p1search +-define} to select and submit an exposure or a group of exposures, +potentially selected on the basis of a query from the raw science +exposure table. + +The P1 pending exposure table is examined by {\tt p1search -pending} +to select the new exposures, which are sent to PanTasks. PanTasks +initiates a separate analysis job (p1astro) for each exposure, which +are sent to the parallel processing nodes. Within the analysis job, +the chips (image files) associated with the exposure are select from +the raw image file table. The analysis examines the contents of these +files, either extract the guide star information from the image files +(GS table extension) or searches for and centroids the pixels on +appropriate bright stars. The analysis results in astrometric +calibration terms which are written to the astrometric calibration +file for this exposure. The location of the astrometric calibration +file and the statistics of the measurement are written back to the P1 +exposure table. The images associated with exposures which are +successfully processed by P1 are then added to the P2 image table, +which is used to trigger the Phase 2 analysis. This last step is +performed by the command {\tt p1search -done}, which is executed +regularly to search for completed Phase 1 jobs. + +\subsection{Phase 2} + +\begin{figure} +\begin{center} +\includegraphics[scale=0.85]{pics/ipptools.05.ps} +\caption{\label{phase2} Phase 2 Tasks} +\end{center} +\end{figure} + +Figure~\ref{phase2} shows the tables involved in running the P2 +analysis stage. There are paths for images to enter the analysis +automatically from the P1 analysis (arrow on left) or to be added +manually based on a selection from the raw exposure and raw image file +tables. Image files to be analysed by Phase 2 are added to the P2 +pending imfiles table with the state `new'. When images are added to +this table, a single entry is also added to the P2 exposure table +listing the P1 and P2 versions for this exposure. These version +numbers must be integers starting with 1. If this image did not have +a P1 analysis, the P1 version is set to 0. Exposures may be added +multiple times for processing and reprocessing. The P2 image table +keeps a record of the old attempts for debugging and analysis. As +with P1, each time a collection of associated images from an exposure +is added to the P2 image table, it is given a new, unique version +number, allowing the system as a whole to track different analysis +attempts. Note that these version numbers are unique for each {\em +exposure} processed by Phase 2, not just for any image file. The top +portion of the diagram illustrates the behavior of the commands {\tt +p2search -define} and {\tt p2search -quick}. The first may be used to +re-submit the images for an exposure or a group of exposures, +potentially selected on the basis of a query from the raw science +exposure and raw image file tables. The second version sends images +files directly to PanTasks for processing; these entries will not be +included in the processing tables, and is used only for testing +purposes. + +The P2 pending image table is examined with the command {\tt p2search + -pending} to select the `new' images. These images are used by +PanTasks to generate P2 analysis jobs, running the analysis command +{\tt ppImage}. The P2 analysis uses the input url to find and load +the image file. The url may be a file on disk, an entry in the image +server, Nebulous, etc. The master detrend images matching the +specific science image and the conditions are selected by examining +the table of master detrend frames. The specific detrend image files +are selected by using the master detrend ID to select the matching the +entries in the table of master detrend files. After the analysis, the +output image, mask, and FITS table of objects, including the +astrometry calibration, are written back to the P2 image table, along +with summary statistics from the P2 analysis. The state is also +updated (to `done'). + +The completed images are examined by the command {\tt p2search -done}, +and when all image files for a single exposure are completed, this +command migrates them to the P2 done table. This process is also +responsible for populating the P3 pending tables so exposures may be +processing by Phase 3. + +\subsection{Phase 3} + +\begin{figure} +\begin{center} +\includegraphics[scale=0.85]{pics/ipptools.06.ps} +\caption{\label{phase3} Phase 3 Tasks} +\end{center} +\end{figure} + +Figure~\ref{phase3} illustrates the tables and commands involved in +the Phase 3 analysis. The P3 pending exposure table lists the +exposure ID, the P3 analysis version, the P2 analysis version to be +used as input to this P3 analysis, and the recipe to be used. The +command {\tt p3search -pending} extracts exposures from this table and +provides them to PanTasks for processing. PanTasks launches a Phase 3 +analysis (the command {\tt psastro}?) for each exposure. In this +analysis, the P2 exposure and image tables are used, in conjunction +with the P2 version information, to select the P2 output measured +objects and the astrometric calibrations from P2 and P1. These +measured objects are matched with the reference catalog objects, and +calibrated astrometry {\em and eventually photometry} is produced for +the full exposure. The location of the resulting astometry +calibration table is stored back in the P3 exposure table. If the +recipe file specifies, the 2-D photometric and background / fringe +corrections may also be performed at this stage. Since these analyses +require reference data, the recipe may also be used to skip these +analysis if such reference data is unavailable or unreliable. At the +end of Phase 3, the objects from the exposure are inserted into the +photometry database (this is not shown). + +The astrometric calibration portion of Phase 3 is principally needed +for a mosaic camera. For single-chip cameras, Phase 3 may be used to +perform the photometric calibration and simply pass the astrometric +results along to the output file to be listed in the P3 exposure +table. In this way, later stages of the analysis (ie, Phase 4) can +use the P3 exposure table as input for all cameras, even if all the +funcionality of Phase 3 is not necessary for that camera. This would +be the case for the skyprobe camera, for example. + +\subsection{Phase 4} + +\begin{figure} +\begin{center} +\includegraphics[scale=0.85]{pics/ipptools.07.ps} +\caption{\label{phase4} Phase 4 Tasks} +\end{center} +\end{figure} + +At the end of Phase 3, the images are ready for Phase 4. The Phase 3 +diagram shows the output line adding the exposures to be processed by +Phase 4 to a Phase 4 table. However, this line is just for +illustration purposes. The rules for initiating a Phase 4 run are +somewhat more complicated than those for running Phases 1-3. Groups +of exposures which have an appropriate overlap should be chosen for +the Phase 4 analysis. In the steady-state period of PS-4, it may be +straightforward to choose the exposure groups: they would simply be +the exposures obtained nearly simultaneously by the four separate +cameras. The circumstance for PS-1 will be much more complicated (and +even PS-4 will probably be more complex than it seems at first +glance). For example, in PS-1, we will not have a static sky for most +of the AP Survery. In this circumstance, we cannot run P4, at least +until after the complete AP Reference catalog is built, and +potentially all exposures re-run through Phase 3. It may be useful +for the AP Survey data to split the Phase 4 analysis into two stages: +image combination and image differencing. It may even be the case +that only the combination portion of Phase 4 is performed on the AP +Survey data. + +More generally, the image groups selected for Phase 4 analysis may be +chosen on the basis of a query of the AP Database (DVO) with some +rules. + +\note{Phase 4 run can be defined by selecting an observation group, a + set of exposures, or a set of rules related to a spatial region (eg, + region, time range, and filter}. + +\note{Phase 4 discussion (and diagram) needs more work} + +\subsection{Analysis Version and Recipes} + +Note that each of the stages P1-P4 refer to the processing version +from the previous stage. This allows the processing stage to request +the correct version of the results from the previous stage, and makes +it possible to run and re-run the analysis at any stage without +deleting the earlier results. As different analysis attempts are +performed for a given image, the versions branch out. + +Also note that at every stage, the entries include a recipe +identifier. This is used to select the analysis recipe which should +be used for this version. By default, the recipe should be set to the +current best recipe (use a default name for this?). This feature +allows the user to run test analyses with variations on the recipe +without altering the analysis system. For example, it is possible to +use a different flat-field set by specifying alternate rules for the +flat-field selection in a recipe file. If it is necessary to run the +P1-P3 analysis with the raw master flats, for example, the user simply +defines that selection in the recipe file and submits the images of +interest to P1 (or P2, etc), with the corresponding entry for the +recipe. + +The recipe file may also be used to specify alternative analysis paths +and desitinations. For example, it is not necessary that all analysis +stops with P4: the recipe file may be used to halt the analysis at P2 +or P3. In addition, the recipe file may be used to specify an +alternative destination for the output results. For example, to +generate the photometric flat-field correction frame from a collection +of dithered images, the user may not want the photometry results in +the main DVO database. By using the recipe to set an alternative DVO +database target, and by specifying the use of the raw master flat +rather than the corrected one, the analysis of the dithered images is +kept isolated from the other photometry database results. The +resulting photometry may be used to construct the new, corrected +flat-field images, and the processing of the same images using the new +flat-field images may be sent to the master DVO database. + +\subsection{Basic Detrend Creation} + +\begin{figure} +\begin{center} +\includegraphics[scale=0.85]{pics/ipptools.08.ps} +\caption{\label{detrend} Detrend Creation Tasks} +\end{center} +\end{figure} + +Figure~\ref{detrend} illustrates the tables needed for the generic +detrend construction process, using the flat-field construction as an +example. This diagram is somewhat more complex than the preceeding +versions. In this diagram, both single jobs and multiple jobs are +represented by the process elements (the blue ellipses). In some +cases, more that one task will be needed to perform the function +illustrated by a single process task. The complexity of this diagram +is enhanced by the need for multiple iterations and both single chip +and full mosaic processing. At the moment, the distinction between +mosaic and single chip cameras is not specifically discussed. +Finally, the triggers which initiate a specific detrend analysis are +glossed over. + +The detrend analysis is initiated by choosing a type of detrend image +to be constructed and by specifying the criteria which will be used to +select the input raw detrend frames for the construction. For +example, these criteria could specify that all twilight flat images +over a certain period of days, perhaps with restrictions on the flux +levels or the time-from-sunset of the images. The detrend analysis +run is given an ID (det ID) which will also be used to identify the +resulting master detrend frame. + +Given the definition of a master detrend run, the input exposures are +selected from the raw detrend exposure table, and written to the input +detrend exposure table. In the next step, the corresponding image +files are selected from the table of raw image files. Since there +will be a different set of input raw images for each attempt at +creating a master detrend image, and since any given attempt may use +some of the same input images as any other attempt, a separate table +of input raw images is constructed. + +Each of the input raw images may be pre-processed before it may be +used to construct the detrend frame. For example, the input +flat-field images should (probably) be dark- and bias-corrected before +they are stacked. The information about these input processed images +is written to the input images table. If no processing is needed, +this step simply copies the appropriate information to the table, and +points back to the raw image, rather than a processed version. + +The input processed images are combined (stacked) to create a master +detrend image for the particular data element defined by the image +class (chip/cell/fpa). At this stage, not all input images should +necessarily be included in the stack. If residual statistics have +been measured for the input images (say, using a prior stack), then +some of the input image may be excluded. The table of residual images +is used to guide this process. The information describing the +resulting master image is written to the master images table. + +The statistics of the master detrend images must examined so that any +necessary renormalizations may be performed. For example, after +stacking the individual flat images, the resulting stacks must be +renomalized to account for the different ranges of input image fluxes. +This analysis is least-squares solution in which an appropriate scale +is determined for each input exposure and a separate gain is +determined for each of the chips or cells in the camera. This +analysis can only performed after all image stacks (ie, for all chips) +have been constructed. The resulting information is written to the +table of master detrend frames. + +Once the master detrend is constructed, the master detrend images may +be used to construct residual images for each of the input images. +These residual statistics, as well as the locations of the residual +images and other related data products (jpeg thumbnails?) are written +to the residual image table. Note the red arrow which by-passes the +stack construction and merge steps and skips directly to the residual +analysis. In some cases, it may be useful to have the input images +confronted with an existing detrend image, and the resulting residual +values used to guide the rest of the process. For example, in the +flat-field analysis, applying an earlier flat can result in a very +good first-pass rejection of poor input images. The logic to make +this leap must be part of PanTasks, since each of the individual +blocks represent complete processing jobs. + +Finally, the residual statistics from the complete mosaic (all input +images, all chips) are used to assess the quality of the newly +constructed master detrend image, and to potentially modify the +selection of input images. This latter process is performed by +marking the state of the residual images from this iteration. The +stacking process always examines the state information for the +residual images from the previous iteration, if it exists, when +constructing the master stack. Once a master detrend frame has been +judged of high enough quality, the state of the entry for the frame in +the master detrend frames table is set to an appropriate value to tell +the other routines that this image should be used as a master detrend. +The exact choice of which master detrend frame is used for a given +science image depends on the recipe along with information such as the +time period used or the conditions used. + +Note that, although this discussion focuses on the construction of +flat-field images, the same structure should be capable of +constructing the biases, dark, fringes, etc. In some cases, as noted +above, the `process' stage is a null operation. + +\begin{figure} +\begin{center} +\includegraphics[scale=0.85]{pics/ipptools.09.ps} +\caption{\label{detprocess} Detrend Creation : Process Tasks} +\end{center} +\end{figure} + +\begin{figure} +\begin{center} +\includegraphics[scale=0.85]{pics/ipptools.10.ps} +\caption{\label{detresid} Detrend Creation : Residual Tasks} +\end{center} +\end{figure} + +\begin{figure} +\begin{center} +\includegraphics[scale=0.85]{pics/ipptools.11.ps} +\caption{\label{detstack} Detrend Creation : Stack and Norm} +\end{center} +\end{figure} \section{Interfaces}