IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Ignore:
Timestamp:
Jan 21, 2006, 11:54:47 PM (20 years ago)
Author:
eugene
Message:

fleshed out further

File:
1 edited

Legend:

Unmodified
Added
Removed
  • trunk/doc/design/ippSSDD.tex

    r6055 r6167  
    1 %%% $Id: ippSSDD.tex,v 1.6 2006-01-19 10:58:19 eugene Exp $
     1%%% $Id: ippSSDD.tex,v 1.7 2006-01-22 09:54:47 eugene Exp $
    22\documentclass[panstarrs]{panstarrs}
    33
     
    1515
    1616% allow paragraphs to be listed in TOC for now
    17 \setcounter{tocdepth}{4}
     17\setcounter{tocdepth}{3}
    1818
    1919\begin{document}
     
    777777%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    778778
    779 \subsection{AP Database}
    780 
    781 \tbd{this section needs to be updated with current implemetation; the
    782   DVO SDD contains much of this information, but needs to be fleshed
    783   out in places.}
     779\subsection{DVO : the AP Database}
    784780
    785781\subsubsection{Corresponding Requirements}
     
    787783The AP Database must meet the requirements specified in Section 3.4.3
    788784of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005).  The specified design
    789 is chosen to meet requirements 3.4.3.1 and 3.4.3.2.  In order to meet
    790 the throughput requirements, the AP Database will be distributed
    791 across 10 Nodes independent of the Image Server Nodes.  An alternative
    792 organization of the database which will be studied will have the AP
    793 Database co-located with the Image Server Phase 4 Nodes.
     785is chosen to meet requirements 3.4.3.1 and 3.4.3.2.  The IPP is
     786modifying the Elixir program 'DVO' to perform the role of the IPP AP
     787Database. 
    794788
    795789\subsubsection{Overview}
    796790
    797 The AP (Astrometry \& Photometry) Database is a CSCI which stores data
    798 related to astronomical objects derived from various sources with a
    799 variety of associations.  The AP Database deals with two related
     791DVO, the Desktop Virtual Observatory, is a software system which
     792stores data related to astronomical objects derived from various
     793sources, and provides mechanisms to related multiple detections
     794together as astronomical objects.  DVO deals with two related
    800795concepts: {\em objects} and {\em detections}.  The {\em objects} are
    801796descriptions of astronomical objects while the {\em detections} are
     
    809804must be accepted as they are reported.
    810805
    811 The AP Database stores the collections of detections which were
    812 derived from specific images from any of the analysis stages.  It
    813 provides a mechanism to determine the image from which a specific
    814 detection was derived, and in conjunction with the Image Server locate
    815 the corresponding data file.  The AP Database also makes it possible
    816 to extract all detections derived from a specific image and to
    817 determine quantities such as the pixel coordinates of the detection on
    818 the image.
    819 
    820 The AP Database also has the capability to associate multiple
    821 detections of a specific object.  Several major classes of objects
    822 will be present, each of which must be handled correctly.
    823 
    824 First, the most distant stars, compact galaxies, and QSOs will have
    825 nearly fixed locations relative to other distant stars, with only
    826 small deviations for individual measurements.  The association between
    827 multiple detections of such objects is made on the basis of their
    828 coincident positions.  The AP Database determines the average position
    829 of the object and the deviations of the individual detections from
    830 that average on the basis of the ensemble of individual detection.
    831 
    832 Second, solar system objects do not have a fixed location.  Detections
     806DVO stores the collections of detections which were derived from
     807specific images.  It provides a mechanism to determine the image from
     808which a specific detection was derived, and in conjunction with the
     809Image Server locate the corresponding data file.  DVO also makes it
     810possible to extract all detections derived from a specific image and
     811to determine quantities such as the pixel coordinates of the detection
     812on the image.
     813
     814DVO also has the capability to associate multiple detections of a
     815specific object.  Several major classes of objects will be present,
     816each of which must be handled correctly.  DVO distinguished the
     817following types of objects.
     818
     819{\bf Stars, compact galaxies, and QSOs} will have nearly fixed
     820locations relative to other distant stars, with only small deviations
     821for individual measurements.  The association between multiple
     822detections of such objects is made on the basis of their coincident
     823positions.  DVO determines the average position of the object and the
     824deviations of the individual detections from that average on the basis
     825of the ensemble of individual detection.
     826
     827{\bf Solar System Objects} do not have a fixed location.  Detections
    833828of such objects are linked by their orbits, and depend on both the
    834 position and the time of the image.  The AP Database does not attempt
    835 to make this link; this is the role of the MOPS system.  However, it
    836 has the ability to accept identifications made externally with
    837 specified detections and to return the identifier of the moving object
     829position and the time of the image.  DVO does not attempt to make this
     830link; this is the role of the MOPS system.  However, it has the
     831ability to accept identifications made externally with specified
     832detections and to return the identifier of the moving object
    838833associated with the specific detections.  These associations also
    839834include descriptive information such as the offset of the detection
    840835from the predicted location of the detection based on the orbit.  This
    841 functionality is required to allow the AP Database to ignore known
    842 moving object detections from other types of queries.
    843 
    844 Third, objects in the general vicinity of the solar system fall in
    845 between these first two classes of objects.  Their proper motion and
    846 parallax response is significant enough ($>0.2$ arcsec in 1 year) that
    847 they are not well-described by an average location and a collection of
    848 offsets.  These objects are described by a distance and a proper
    849 motion vector.  The AP Database provides the association between the
    850 specific detections and an average object which includes finite
    851 parallax and proper motion.
    852 
    853 Fourth, many detections, especially in their initial states, will not
    854 be associated with a specific astronomical object of any of the above
    855 classes and are treated as orphans.  Most of these will be spurious
    856 (not representing real objects), some will be from solar system
    857 objects for which orbits are not yet determined, some will be from
    858 faint stars near the detection limits, and some will be from
    859 short-term transients which have only been detected once.  The AP
    860 Database maintains these detections until they have been associated
    861 with one of the objects above.  The AP Database provides mechanisms by
    862 which individual detections may be migrated back and forth between the
    863 orphan state and association with an astronomical object.
    864 
    865 For every object, and all orphaned detections, the AP Database also
    866 provides the capability to determine the images containing the
    867 location of the object but for which no detection was made.  The
    868 minimum set of information which must be carried for these
     836functionality is required to allow DVO to ignore known moving object
     837detections from other types of queries.
     838
     839{\bf High-proper-motion objects} in the general vicinity of the solar
     840system fall in between these first two classes of objects.  Their
     841proper motion and parallax response is significant enough ($>0.2$
     842arcsec in 1 year) that they are not well-described by an average
     843location and a collection of offsets.  These objects are better
     844described by a distance and a proper motion vector.  DVO provides the
     845association between the specific detections and an average object
     846which includes finite parallax and proper motion.
     847
     848{\bf Orphaned detections} are not associated with a specific
     849astronomical object of any of the above classes.  Most of these will
     850be spurious (not representing real objects), some will be from solar
     851system objects for which orbits are not yet determined, some will be
     852from faint stars near the detection limits, and some will be from
     853short-term transients which have only been detected once.  DVO
     854maintains these detections until they have been associated with one of
     855the objects above.  DVO provides mechanisms by which individual
     856detections may be migrated back and forth between the orphan state and
     857association with an astronomical object.
     858
     859DVO stores the information about the detection, the related objects,
     860and the images which provided the measurements.  For every detection,
     861DVO provides the mechanisms to link the detection back to the image
     862which supplied it.  DVO also provides the capability to determine the
     863images containing a specific location but for which no detection was
     864made.  The minimum set of information which must be carried for these
    869865non-detections is the image and the associated object or orphan.
    870866
    871 The AP Database also stores the relationships between various
     867DVO also stores the relationships between various
    872868photometric systems and the evolution of that relationship.  It
    873869provides mechanisms to convert between the measured instrumental
     
    880876various reference systems appropriate for those filters.
    881877
     878\subsubsection{Photometric systems and the DVO Photcodes}
     879
     880One of the major roles of DVO is to relate different photometric
     881measurements made with different instruments and detectors together.
     882We may have observations made with the same basic filters, but using a
     883number of different detectors.  We may have observations from
     884different telescopes in similar filters.  We may have reference data
     885related to some filter, but obtained and published by other observers.
     886We would like to related these measurements together in optimal ways,
     887making use of whatever information we have available.  DVO provides
     888several mechanisms to enable these relationships.
     889
     890We identify three distinct types of photometry measurements within
     891DVO:
     892\begin{itemize}
     893\item {\bf reference photometry}  These measurements are provided by
     894  external observers.  For reference photometry, we do not have access
     895  to very must information used to determine the magnitudes of the
     896  objects of interest.  We have the reference magnitudes corresponding
     897  to a type of filter, and presumably some information of the error on
     898  the measurement.  We might possibly know the epoch of the
     899  observations, but not necessarily. 
     900\item {\bf detection photometry} This is our primary measurement of
     901  interest: the photometry of objects measured from images which we
     902  have processed.  More specifically, the detection photometry is an
     903  instantaneous measurement from a specific image with well-known
     904  properties, such as exposure time, airmass, instrument source, etc. 
     905\item {\bf internal photometry} With the application of an appropriate
     906  zero point and other calibration terms, any detection photometry can
     907  be calibrated to represent a measurement in a well-known photometric
     908  system.  The internal photometry measurements are calibrated to be
     909  on a photometric system which represents a consistent system for a
     910  particular telescope or collection of data, minimizing the
     911  calibration transformations necsessary.
     912\end{itemize}
     913
     914Defining the relationships between the different types of measurements
     915is part of the process of photometric calibration.  DVO uses the
     916concept of the 'photcode' to identify the source of the photometry,
     917and to define the relationships between different photometry sources.
     918A photcode identifies a photometric system: for the detection
     919photometry measurments, each combination of telescope, camera, filter,
     920and detector is associated with a unique photcode; there are also
     921unique photcodes for the internal photometry systems and any distinct
     922external reference source. 
     923
     924As a concrete example, consider the Pan-STARRS PS-1 system.  There
     925will be three different cameras in use at different times: GPC-1,
     926TC-3, and the SkyProbe camera.  There are at least 6 filter systems:
     927{\it grizy} and {\it w}.  The SkyProbe camera has a single CCD, TC-3
     928has 16 different detectors, and GPC-1 has up to 64 different devices.
     929Each of these combinations is potentially a different photometric
     930system, so a different photcode is defined for each combination.
     931These photcodes would have names such as: GPC1.02.r (r filter with the
     932GPC1 camera and OTA 02) or SP1.00.g (SkyProbe 1, g filter).  These
     933($64 \times 6 + 16 \times 6 + 5 = 485$) photcodes are all identified
     934as 'detection' photcodes, specifying that detection photometry is
     935associated with them
     936
     937There are also 6 different internal photometric systems of interest,
     938namely those associated with the 6 named filters, {\it grizy} and {\it
     939w}. Each of these 6 systems is identified with an internal photcode.
     940The internal photcodes are further distinguished as 'primary' or
     941'secondary', which specifies how the DVO system stores average
     942quantities related to these types of photcodes (see the discussion of
     943the tables below). 
     944
     945Finally, there may be multiple external photometric systems of
     946interest, some of which are related to the major internal photometry
     947systems, some of which are not.  For example, the Pan-STARRS project
     948may refer to photometry from the SDSS secondary standards, the SDSS
     949data releases, Johnson photometry from Landolt (1992), observations
     950from 2MASS in $JHK$, USNO-B observations, and so forth.  Each of these
     951photometric systems is assoiciated with a different photcode; only
     952some of these are relevant to the detection or internal photometry
     953system.
     954
     955Within DVO, the detection and internal photcodes each define a
     956relationships as well as a specific photometric system.  Associated
     957with each of these photcodes are the parameters of the photometry
     958transformation from the photometric system of the photcode to another
     959photometric system.  For the detection photcodes, the parameters
     960define the transformation to the equivalent internal photcode system.
     961The currently-defined transformation parameters consist of the
     962following photometry equation:
     963%
     964\[
     965M_\lambda = m_\lambda + C_\lambda + K_\lambda (\mbox{airmass} - 1) + \sum_{i = 1}^{i < N}
     966A_{\lambda,i} (\mbox{color}_\lambda - \mbox{color}_{o,\lambda})^i
     967\]
     968%
     969where $C_r$ represents the zero-point of the transformation, $K_r$
     970represents the slope of the airmass trend, $\mbox{airmass}$ is the
     971airmass for a given measurement, $\mbox{color}$ is the color of the
     972source of interest (as identified below), $\mbox{color}_r$ is the
     973reference color for sources in this photometry system, and $A_{r,i}$
     974is the coefficient of the $i$ power of the color difference.  Up to
     975fourth order color terms are currently allowed.  For any photcode, the
     976color is defined as the difference of the measurements in two other
     977photcodes, usually two 'internal' photcodes.  The photcode information
     978also specified the equivalent photcode to which the transformation corresponds.
     979
     980For the detection photcodes, the target of the transformation must be
     981an internal photcode.  For the internal photcodes, the target of the
     982transformation is an external reference photcode system.  This
     983restriction implies that the internal photometry may only be
     984transformed (and thus compared with) a single external reference.
     985This is in fact the best practice as far as photometric calibration is
     986concerned: the 'standard' observations from different references
     987should always be treated as different photometric systems.  To allow
     988for the relationship of the internal photometry to multiple sources of
     989reference photometry, an additional set of photcodes are defined which
     990identify 'alternative' transformations for the internal photcodes.
     991
     992It is important to note that not all of the photometry transformation
     993parameters identified above are relevant for each of the three major
     994types of photcode.  The detection photcodes will in general make use
     995of all of these elements, though the order of the color transformation
     996will hopefully be limited if the different devices are sufficiently
     997similar.  For the transformation from the internal photcodes, which
     998are derivative in some way of the detection photcodes, the airmass
     999component is invalid: for a single measurement, the
     1000detection-to-internal transformation has already removed the airmass
     1001trend; for an averaged internal photometric measurement, no single
     1002airmass corresponds to the observations.  Finally, no transformation
     1003parameters are defined for the reference photcodes at this time.
     1004
     1005DVO provides methods by which these photometry transforamtions are
     1006automatically applied.  The specific measurements (detection
     1007photometry) are stored in the database tables as instrumental
     1008magnitudes, and any operation which examines these measurements must
     1009make use of the APIs to convert to an appropriate common system.  A
     1010further complication to note is that the photcodes defined above are
     1011static; they do not include any information about changes to the
     1012system sensitivity.  This information is carried externally to the
     1013photcode calibration information; the transformations defined by the
     1014photcodes must be considered the {\em starting point} for any
     1015photometric analysis.  An additional adjusment can be applied. 
     1016
     1017The detections from a specific image may all have a 'calibration'
     1018offset applied which bring the measured photometry into a common
     1019relative system.  This calibration offset is associated with the image
     1020and may be a function of position on the detector.  The tables which
     1021carry the individual measurements also include the calibration
     1022magnitude appropriate for each measurement to speed up the application
     1023of this offset.  In a well-calibrated collection of photometry, all of
     1024the detection measurements will have a measured calibration magnitude,
     1025yielding a collection of internal photometry measurements which are
     1026all consistent.  An additional piece of information is the zero-point
     1027history, which tracks the system-wide variations in the average
     1028sensitivity.  The zero-point history can be used to predict the
     1029calibration magnitudes for any observation which is not tied directly
     1030via relative photometry to the rest of the photometric observations.
     1031
     1032Putting all of these pieces together, the photometry APIs in DVO can
     1033be used to return any of the following types of photometric
     1034measurements:
     1035\begin{itemize}
     1036\item raw instrumental magnitudes for any detection
     1037
     1038\item 'catalog' magnitudes, applying only the airmass and static
     1039  zero-point calibrations to a detection magnitude; this is useful to
     1040  test the detector-color transformation.
     1041
     1042\item 'system' measurements, applying the complete static
     1043  transformation for a detection magnitude to the internal photometry
     1044  system; for photometric weather and no zero-point variations, this
     1045  would be a measurement in the internal photometry system.
     1046
     1047\item 'relative' magnitudes, applying the measured calibration offset
     1048  to the calibrated detection magnitude determined above; in a
     1049  well-calibrated system, this represents a consistent internal
     1050  photometry measurement.
     1051
     1052\item 'calibrated' magnitudes, correcting the measure detection
     1053  photometry by applying the transformation from the internal
     1054  magnitude system to the external reference magntiude system.
     1055
     1056\item 'average' magntiudes, the raw internal photometry magnitudes
     1057  (note the distinction between the 'average' quantities, which are
     1058  derived from a collection of detections an the 'relative' quantities
     1059  which represent an instantenous measurement in the same system).
     1060
     1061\item 'reference' magnitudes, in which the 'average' internal
     1062  photometry values are transformed to the refernce magnitude system. 
     1063\end{itemize}
     1064The complexity of these transformations is necessary to allow the
     1065examination of the trends of actual measurements with external
     1066parameters.
     1067
     1068\subsubsection{DVO Database Tables}
     1069
    8821070\begin{figure}
    8831071\begin{center}
    884 \resizebox{4.5in}{!}{\includegraphics{pics/APDB}}
    885 \caption{AP DB components}
    886 \label{fig:APDBComponents}
     1072\resizebox{4.5in}{!}{\includegraphics{pics/dvo.01.ps}}
     1073\caption{\label{fig:DVOtables} \small Data types managed by DVO}
    8871074\end{center}
    8881075\end{figure}
    8891076
    890 The AP Database provides interfaces to extract lists of objects and
    891 detections based on various query parameters.  It provides the
    892 capability to extract all detections associated with a specific
    893 object, all non-detections of that object, all non-detections of an
    894 orphan, and summary statistics from these collections.  It will also
    895 return all objects or detections within specified spatial regions
    896 including regions bounded by great circles (RA,DEC; GLAT,GLON;
    897 ELAT,ELON) and regions described by a location and a search radius.
    898 It will also return the image parameters associated with a specific
    899 detection including image coordinates of the detection, exposure time,
    900 time and date of the detection, etc.
    901 
    902 As shown in Figure~\ref{fig:APDBComponents}, the IPP AP Database
    903 consists of the following components:
    904 
    905 \begin{itemize}
    906 \item AP Database database tables
    907 \item AP Database database engine
    908 \item AP Database servers
    909 \item AP Database client APIs
    910 \end{itemize}
    911 
    912 \subsubsection{AP Database Tables}
    913 
    914 Table~\ref{tab:APDBTables} lists the tables used by the AP Database.  The
    915 contents of these tables are outlined in
    916 Appendix~\ref{sec:APDBTableContents}.  Below, the use of these tables by
    917 the AP Database software is discussed below.  Several of the tables
    918 are not just simple tables in the database but are instead table
    919 groups divided into many subtables, each of which represents a portion
    920 of the sky (a {\tt region}).  These subtables may also be distributed
    921 across different computers to distribute the processing load.
    922 
    923 \paragraph{Images Table Group}
    924 
    925 The {\tt Images} table group lists all of the images which provided
    926 the data in the AP Database.  These tables are subdivided by region on
    927 the sky.  In general, the images listed in this table correspond to
    928 the Chips.  This group of tables includes sufficient astrometric
    929 parameters to represent the coordinates of the detections to a
    930 sufficient accuracy.  Parallel to the Images table is the Mosaic
    931 table.  This table is very similar to the Images table, but defines
    932 the Mosaic which corresponds to a group of Images.  The parameters
    933 include the astrometric information needed to define the camera
    934 distortion.
    935 
    936 \paragraph{Image Overlaps Table Group}
    937 
    938 The specific subtable of {\tt Images} which contains a given image is
    939 the one which contains the center pixel of that image.  An additional
    940 table group, {\tt Image Overlaps} (with the same subtable organization
    941 as the {\tt Images} subtables), lists images which overlap that
    942 specific subtable.  Thus, given a particular coordinate, in order to
    943 find that images which overlap that coordinate, it is necessary to
    944 search the images in the {\tt Images} subtable which includes that
    945 coordinate, and all images in the {\tt ImageOverlaps} subtable for
    946 that coordinate.
    947 
    948 \begin{table}[hb]
    949 \begin{center}
    950 \caption{AP Database Tables\label{tab:APDBTables}}
    951 \begin{tabular}{ll}
    952 \hline
    953 \hline
    954 {\bf Table Name} & {\bf Description} \\
    955 \hline
    956 Images               & The images that have objects in the DB. \\
    957 Image Overlaps       & Image regions which are touched by specific images. \\
    958 Objects              & The objects --- average properties of multiple detections of the same object. \\
    959 Average Magnitudes   & Average photometry in multiple filters \\
    960 Solar System Objects & Identification of solar system objects \\
    961 Matched Detections   & Detections of sources in an image identified with an Object. \\
    962 Orphaned Detections  & Detections of sources in an image not identified with an Object. \\
    963 Non-detections       & Non-detections of objects in an image. \\
    964 Regions              & spatial distribution of tables \\
    965 Filters              & Filters understood by the system. \\
    966 Photcodes            & Transformations between different photometric systems \\
    967 Zero Points          & History of Zero-point \& Airmass terms \\
    968 Distortion Models    & History of Optical Distortion terms \\
    969 Database Hosts       & computers used to store the tables \\
    970 \hline
    971 \end{tabular}
    972 \end{center}
    973 \end{table}
    974 
    975 \paragraph{Objects Table Group}
    976 
    977 The {\tt Objects} table group (also divided by region) stores the
    978 average parameters for each astronomical object.  Certain details of
    979 this table have not yet been specified.  In particular, objects with
    980 significant parallax and/or proper motion may potentially be stored in
    981 a distinct table.  Solar system object identifications, to the extent
    982 average properties are maintained in the AP Database, will certainly
    983 be stored in a separate table. 
    984 
    985 \paragraph{Average Magnitudes Table Group}
    986 
    987 A related table, also divided into the same regions, is the {\tt
    988 Average Magnitudes} table.  In this table, there are multiple rows per
    989 object, one for each of the primary filters of interest for which
    990 photometric averaging is performed.  This organization makes the
    991 number of primary (averaged) filters a configurable value.
    992 
    993 \paragraph{Matched Detections Table Group}
    994 
    995 The {\tt Matched Detections} table stores all of the measurements of
    996 astronomical objects on specific images.  This table includes all
    997 detections associated with the average {\tt Objects}.  As discussed
    998 below, bright objects (above a configuration-specified signal-to-noise
    999 level) are defined object even if only one detection has been found at
    1000 that position.  Faint orphaned objects are not added to this list or
    1001 the list of objects.  The different types of detections (P2,
    1002 P4$\Delta$, P4$\Sigma$) are distinguished by their photometry codes.
    1003 (This is only valid if the AP Database does not store different
    1004 quantities for these types of detections.)
    1005 
    1006 \paragraph{Orphaned Detections Table Group}
    1007 
    1008 The {\tt Orphaned Detections} table stores the detections which have
    1009 not been correlated with an existing object.  This table is only
    1010 populated for objects below a configuration-specified signal-to-noise
    1011 limit (e.g., 5$\sigma$).  Bright orphaned detections are assigned an
    1012 object and added to the {\tt Matched Detections} table.
    1013 
    1014 \paragraph{Non-detections Table Group}
    1015 
    1016 The {\tt Non-detections} table stores information about detection
    1017 failures for each object.  If an image is added to the database which
    1018 overlaps an object but the object is not detected, an entry is made in
    1019 this table.  In practice, this table may store only the most recent
    1020 non-detection and the total number, or a similar reduced set of
    1021 non-detection statistics.
    1022 
    1023 \paragraph{Regions Table}
     1077Figure~\ref{fig:DVOtables} illustrates the data managed by DVO, and
     1078Table~\ref{tab:DVOtables} provides a complete listing.  The contents
     1079of these tables are outlined in Appendix~\ref{sec:DVOTableContents}.
     1080Below, the use of these tables by DVO software is discussed below.
     1081Several of the tables are not just simple tables in the database but
     1082are instead table groups divided into many subtables, each of which
     1083represents a portion of the sky (a {\tt region}).  These subtables may
     1084also be distributed across different computers to distribute the
     1085processing load.
     1086
     1087\paragraph{Sky Regions Table}
    10241088
    10251089The {\tt Regions} table is used to subdivide the tables of images,
    1026 objects, and detections, etc, as discussed above.  The AP Database
     1090objects, and detections, etc, as discussed above.  DVO
    10271091divides the sky into a hierarchy of regions (portions of the sky) each
    10281092of which is in turn subdivided into smaller portions.  Since nearly
    1029 all interactions with the AP Database performed by the IPP are limited
     1093all interactions with DVO performed by the IPP are limited
    10301094in spatial coverage, subdividing the tables allows a specific
    10311095interaction to search only a small subset of the data.  The table of
     
    10491113\begin{figure}
    10501114\begin{center}
    1051 \resizebox{6in}{!}{\includegraphics{pics/APDBRegions}}
    1052 \caption{AP DB Regions and Image / Object tables}
    1053 \label{fig:APDBRegions}
     1115\resizebox{6in}{!}{\includegraphics{pics/dvo.02.ps}}
     1116\caption{DVO Regions and Image / Object tables}
     1117\label{fig:DVOskyregions}
    10541118\end{center}
    10551119\end{figure}
     1120
     1121\paragraph{Images Table Group}
     1122
     1123The {\tt Images} table group lists all of the images which provided
     1124the data in DVO.  These tables are subdivided by region on
     1125the sky.  In general, the images listed in this table correspond to
     1126the Chips.  This group of tables includes sufficient astrometric
     1127parameters to represent the coordinates of the detections to a
     1128sufficient accuracy.  Parallel to the Images table is the Mosaic
     1129table.  This table is very similar to the Images table, but defines
     1130the Mosaic which corresponds to a group of Images.  The parameters
     1131include the astrometric information needed to define the camera
     1132distortion.
     1133
     1134\paragraph{Image Overlaps Table Group}
     1135
     1136The specific subtable of {\tt Images} which contains a given image is
     1137the one which contains the center pixel of that image.  An additional
     1138table group, {\tt Image Overlaps} (with the same subtable organization
     1139as the {\tt Images} subtables), lists images which overlap that
     1140specific subtable.  Thus, given a particular coordinate, in order to
     1141find that images which overlap that coordinate, it is necessary to
     1142search the images in the {\tt Images} subtable which includes that
     1143coordinate, and all images in the {\tt ImageOverlaps} subtable for
     1144that coordinate.
     1145
     1146\begin{table}[hb]
     1147\begin{center}
     1148\caption{DVO Database Tables\label{tab:DVOtables}}
     1149\begin{tabular}{ll}
     1150\hline
     1151\hline
     1152{\bf Table Name} & {\bf Description} \\
     1153\hline
     1154Images               & The images that have objects in the DB. \\
     1155Image Overlaps       & Image regions which are touched by specific images. \\
     1156Objects              & The objects --- average properties of multiple detections of the same object. \\
     1157Average Magnitudes   & Average photometry in multiple filters \\
     1158Solar System Objects & Identification of solar system objects \\
     1159Matched Detections   & Detections of sources in an image identified with an Object. \\
     1160Orphaned Detections  & Detections of sources in an image not identified with an Object. \\
     1161Non-detections       & Non-detections of objects in an image. \\
     1162SkyRegions           & spatial distribution of tables \\
     1163Filters              & Filters understood by the system. \\
     1164Photcodes            & Transformations between different photometric systems \\
     1165Zero Points          & History of Zero-point \& Airmass terms \\
     1166Distortion Models    & History of Optical Distortion terms \\
     1167Database Hosts       & computers used to store the tables \\
     1168\hline
     1169\end{tabular}
     1170\end{center}
     1171\end{table}
     1172
     1173\subsection{Objects Table Group}
     1174
     1175\begin{table}
     1176\begin{center}
     1177\caption{DBO Detection Classes \& Object Parameters\label{tab:APdetections}}
     1178\begin{tabular}{lrrrr}
     1179\hline
     1180\hline
     1181Object Parameter & P2 & P4S & P4D & SS \\
     1182\hline
     1183PSF x,y, covar, $\alpha,\delta$               & + & + & + & + \\
     1184PSF mag, $\sigma_{\rm mag}$                   & + & + & + & + \\
     1185star/gal sep                                  & + & + & + & + \\
     1186$\sigma_x$, $\sigma_y$, $\theta$              & + & + & + & + \\
     1187local sky data                                & + & + & + & + \\
     1188Petrosian R, M, $R_{50}$, $R_{90}$            & - & + & - & + \\
     1189S\'ersic R, M, AB, $\phi$, $\nu$              & - & + & - & + \\
     1190W.L. $\gamma_1$, $\gamma_2$, pol. terms       & - & - & - & + \\
     1191exp. spaced aps., Poisson noise, variance     & - & - & - & + \\
     1192\hline
     1193\end{tabular}
     1194\end{center}
     1195\end{table}
     1196
     1197The {\tt Objects} table group (also divided by region) stores the
     1198average parameters for each astronomical object.  Certain details of
     1199this table have not yet been specified.  In particular, objects with
     1200significant parallax and/or proper motion may potentially be stored in
     1201a distinct table.  Solar system object identifications, to the extent
     1202average properties are maintained in DVO, will certainly
     1203be stored in a separate table. 
     1204
     1205\paragraph{Average Magnitudes Table Group}
     1206
     1207A related table, also divided into the same regions, is the {\tt
     1208Average Magnitudes} table.  In this table, there are multiple rows per
     1209object, one for each of the primary filters of interest for which
     1210photometric averaging is performed.  This organization makes the
     1211number of primary (averaged) filters a configurable value.
     1212
     1213\paragraph{Matched Detections Table Group}
     1214
     1215The {\tt Matched Detections} table stores all of the measurements of
     1216astronomical objects on specific images.  This table includes all
     1217detections associated with the average {\tt Objects}.  As discussed
     1218below, bright objects (above a configuration-specified signal-to-noise
     1219level) are defined object even if only one detection has been found at
     1220that position.  Faint orphaned objects are not added to this list or
     1221the list of objects.  The different types of detections (P2,
     1222P4$\Delta$, P4$\Sigma$) are distinguished by their photometry codes.
     1223(This is only valid if DVO does not store different
     1224quantities for these types of detections.)
     1225
     1226\paragraph{Orphaned Detections Table Group}
     1227
     1228The {\tt Orphaned Detections} table stores the detections which have
     1229not been correlated with an existing object.  This table is only
     1230populated for objects below a configuration-specified signal-to-noise
     1231limit (e.g., 5$\sigma$).  Bright orphaned detections are assigned an
     1232object and added to the {\tt Matched Detections} table.
     1233
     1234\paragraph{Non-detections Table Group}
     1235
     1236The {\tt Non-detections} table stores information about detection
     1237failures for each object.  If an image is added to the database which
     1238overlaps an object but the object is not detected, an entry is made in
     1239this table.  In practice, this table may store only the most recent
     1240non-detection and the total number, or a similar reduced set of
     1241non-detection statistics.
    10561242
    10571243\paragraph{Other Reference Tables}
     
    10621248photometry system may consist of a detector, telescope, and specific
    10631249filter, or it may be a derived photometry system.  The {\tt Database
    1064 Machines} table identifies all of the computers available to the AP
    1065 Database.
    1066 
    1067 \subsubsection{AP Database servers}
    1068 
    1069 The AP Database functions on a group of computers, with portions of
    1070 the tables stored on separate machines, as described above.  The
    1071 association between a machine and the corresponding table or part of
    1072 the sky is defined by the Region table.  Each machine has a
    1073 corresponding AP Database server which runs on that machine to
    1074 interact with the tables available on that machine.  Two possible
    1075 interaction models are considered. 
    1076 
    1077 {\bf Option A:} A client chooses one of the machines and sends its
    1078 query or data to that machine.  The server then uses the region table
    1079 to determine which machines contain the relevant portion of the sky.
    1080 Data to be added to the database is divided into corresponding region
    1081 chunks and sent to the appropriate servers.  Queries are redirected to
    1082 the appropriate server(s).  The original server may collect the
    1083 results and return them to the original client.
    1084 
    1085 {\bf Option B:} The client downloads the region table and performs the
    1086 division of the data into appropriate subsets.  The subsets are then
    1087 sent to the corresponding servers by the client. 
    1088 
    1089 The differences between these models is small.  The first option may
    1090 make the code more testable, placing all of the logic in the servers
    1091 and making each server symmetric.  The smaller tables (ie, Region,
    1092 Filters, etc) could either be downloaded from a single server or
    1093 replicated to all AP DB servers.  For these reasons, Option A will be
    1094 used for the PS-1 IPP.  \tbd{update this in light of the addstar
    1095   client / server implementation}
    1096 
    1097 \subsubsection{AP Database engine}
    1098 
    1099 The backend database engine for the AP Database stores the tables and
    1100 provides them to the servers on demand.  The AP Database will use a
    1101 \code{mysql} database engine for this function.
    1102 
    1103 \subsubsection{AP DB Client operations}
    1104 
    1105 The AP Database client interactions consist of a collection of basic
    1106 queries of the database, along with more complex operations to perform
    1107 particular tasks.  The complex operations are listed below.
    1108 
    1109 \paragraph{Insert Image \& Detection Set (addstar)}
    1110 
    1111 One of the most basic operations needed by the AP Database is to
    1112 insert a collection of detections derived from a specific image, and
    1113 add the definition of that image to the database.  This operation is
    1114 critical in terms of the processing throughput.  After the detections
    1115 have been assigned to the appropriate regions, they are matched
    1116 against all objects in the {\tt Objects} table.  Matches are performed
    1117 only on the basis of positional coincidence, using a matching radius
    1118 which may depend on the image astrometry errors, or may be a fixed
    1119 distance.  Any matched detections are added to the {\tt Matched
    1120 Detections} table.  Any unmatched detections brighter than the Faint
    1121 Detection cut-off are specified as a new {\tt Object} and also added
    1122 to the {\tt Matched Detections} table.  Any faint unmatched detections
    1123 are added to the {\tt Orphaned Detections} table.  This division is
    1124 important because it allows the automatic association of new
    1125 detections with existing bright objects while limiting the I/O volume
    1126 required to make the detections.  In general, there will be many fewer
    1127 {\tt Objects} than {\tt Detections}, and there will be fewer bright
    1128 orphans than faint orphans.
    1129 
    1130 \paragraph{Insert Reference Objects (addrefs)}
    1131 
    1132 This operation is very similar to the previous one.  A collection of
    1133 reference objects are added to the database as a collection of
    1134 detections.  The reference photometry should in general be given its
    1135 own photometry code.  The reference data is different from the image
    1136 detection set because the associated image information is not
    1137 included.  Thus, no corresponding images are added to the database.
    1138 
    1139 \paragraph{Determine Relative Photometry in region (relphot)}
     1250Machines} table identifies all of the computers available to DVO.
     1251
     1252\subsubsection{Database Table I/O}
     1253
     1254\begin{figure}
     1255\begin{center}
     1256\resizebox{4.5in}{!}{\includegraphics{pics/dvo.03.ps}}
     1257\caption{\label{fig:DVOformats} \small DVO Table I/O }
     1258\end{center}
     1259\end{figure}
     1260
     1261DVO allows for a flexible representation of its data on disk.  Data
     1262may be written to disk in one four possible mode: RAW, FITS MEF, FITS
     1263SPLIT, and MYSQL.  These modes define the overall organization of the
     1264data on disk.  In the RAW mode, the data is written to disk in a
     1265pseudo-FITS table format which consists of a simple FITS header
     1266describing the layout followed by the binary data in a block.  This
     1267storage mode is maintained for historical reasons.  There are also two
     1268types of FITS modes in which the data tables are written as valid FITS
     1269Binary Tables.  In the SPLIT format, every data table is written as a
     1270separate file, while in the MEF format, the object and detection
     1271tables are bundled together into a single FITS file with multiple
     1272table extensions.  The MEF format has the advantage of minimize the
     1273proliferation of files, while the SPLIT format is required to make use
     1274of the fastest read/write capabilities of DVO.  DVO makes use of these
     1275raw data formats as a throughput risk mitigation strategy.  As
     1276discussed below, this strategy has proven very successful.
     1277
     1278There are also multiple formats in which the data may be stored.  The
     1279different formats define which specific database table columns are
     1280stored and with what numerical format and precision.
     1281Figure~\ref{fig:DVOformat} illustrates the conversion process which
     1282DVO performs when loading in the data.  When DVO loads data from a
     1283file-based table (FITS or RAW), it first loads from the disk file into
     1284a data structure representing the external format in use.  The
     1285external structure is then converted into the internal format. The
     1286internal structure is always specified to be the superset of all
     1287external data formats.  This capability allows DVO to maintain
     1288backwards compatibility with data tables written with early versions.
     1289As DVO is extended and new elements are added to the tables, it is
     1290only necessary to define the methods to convert the new internal table
     1291into the external table.  In addition, DVO makes use of autocoded
     1292table manipulation and I/O APIs which are generated for each data
     1293structure based on a descriptive table.  This makes it easy to add new
     1294data types and input/output methods without significant re-coding.
     1295
     1296\tbd{DVO mysql table storage is not yet implemented}
     1297
     1298\subsubsection{addstar : Insert Image \& Detection Set}
     1299
     1300\begin{figure}
     1301\begin{center}
     1302\resizebox{4.5in}{!}{\includegraphics{pics/dvo.04.ps}}
     1303\caption{\label{catalog} \small a figure }
     1304\end{center}
     1305\end{figure}
     1306
     1307One of the most basic operations needed by DVO is to insert a
     1308collection of detections derived from a specific image, and add the
     1309definition of that image to the database.  This operation is critical
     1310in terms of the processing throughput.  After the detections have been
     1311assigned to the appropriate regions, they are matched against all
     1312objects in the {\tt Objects} table.  Matches are performed only on the
     1313basis of positional coincidence, using a matching radius which may
     1314depend on the image astrometry errors, or may be a fixed distance.
     1315Any matched detections are added to the {\tt Matched Detections}
     1316table.  Any unmatched detections brighter than the Faint Detection
     1317cut-off are specified as a new {\tt Object} and also added to the {\tt
     1318Matched Detections} table.  Any faint unmatched detections are added
     1319to the {\tt Orphaned Detections} table.  This division is important
     1320because it allows the automatic association of new detections with
     1321existing bright objects while limiting the I/O volume required to make
     1322the detections.  In general, there will be many fewer {\tt Objects}
     1323than {\tt Detections}, and there will be fewer bright orphans than
     1324faint orphans.
     1325
     1326A wide range of options are available to addstar.  These can be used
     1327to modify the object matching rules, to reduce the number of tables
     1328which are updated, to specify the output data format, and so forth.  A
     1329few options modify the behavoir in substantial ways, as discussed in
     1330the two sections below.
     1331
     1332\tbd{flesh out discussion of the options}
     1333
     1334\paragraph{Insert Reference Objects}
     1335
     1336\code{addstar -ref (filename)}
     1337
     1338This mode of addstar reads a text file and adds the listed objects to
     1339the database as a reference photcode type.  A collection of reference
     1340objects are added to the database as a collection of detections.  The
     1341reference photometry should in general be given its own photometry
     1342code.  The reference data is different from the image detection set
     1343because the associated image information is not included.  Thus, no
     1344corresponding images are added to the database.
     1345
     1346\paragraph{Insert Catalog Objects}
     1347
     1348\code{addstar -cat (name) -region ra ra dec dec}
     1349
     1350In this mode, any of several all-sky or large-scale reference catalogs
     1351are used for the input sources.  The catalog objects are added to the
     1352database as reference objects.  The valid catalogs consist of 2MASS,
     1353USNO, GSC.  Tycho and USNO-B will be added shortly.  Specific
     1354photcode names are defined for each of these catalogs, and must be
     1355appropriately requested and defined in the photcode table.  The
     1356optional region restriction limits the insert to a subset of the sky.
     1357The user does not always want to add 50GB of 2MASS detections to any
     1358DVO database...
     1359
     1360\paragraph{Addstar Client/Server Interactions}
     1361
     1362DVO currently uses stand-alone programs which are run from the command
     1363line (like addstar, or the programs listed below), or it works with
     1364the interactive DVO shell, which allows the user to query portions of
     1365the database.  These programs all interact with the database tables
     1366directly, making use of file locking to prevent conflicts. 
     1367
     1368Unlike the other DVO programs (currently), it is possible to run
     1369addstar as a client/server system.  In this configuration, the program
     1370\code{addstard} is launched to run in the background as a server.  It
     1371monitors a socket waiting for clients to contact it.  The client
     1372program, \code{addstarc} appears to the user identical to the
     1373stand-alone addstar.  However, rather than directly insert data into
     1374the database, \code{addstarc} contacts the addstar server and sends it
     1375the detections and associated image data (along with the information
     1376about the user options).  The daemons accepts the incoming data and
     1377then loads this data into the database, just as the stand-alone
     1378addstar does.
     1379
     1380The purpose of the addstar client/server design is three-fold.  First,
     1381the client can be used by processes to send data to the DVO database
     1382and then immediately exit.  The addstar loading process is one of the
     1383more time-critical functions within the IPP.  However, unlike the
     1384other portions of the IPP, the addstar processes must operate in
     1385serial, at least when they are updating the same portion of the sky
     1386(or the image table).  If the IPP analysis routines all needed to run
     1387the stand-alone addstar program, they would eventually block waiting
     1388for each addstar to complete, preventing other processing from
     1389continuing.  The addstar client / server model allows the processing
     1390node to invoke the addstar client, sending the data to the addstar
     1391server.  The addstar server will then be the entity that manages the
     1392serialization of the incoming data stream.  The addstar server has two
     1393threads which run in parallel.  One thread monitors the socket and
     1394accepts new data sets from addstar clients, adding the data to an
     1395internal queue.  The other thread pulls data off of the queue and
     1396updates the database with the data. 
     1397
     1398A second advantage of the client/server interaction is that only the
     1399new detections need to be sent across the network.  To update the
     1400database, addstar must load the average objects for the region from
     1401the database tables.  In the stand-alone mode, the addstar program
     1402loads this data via NFS across the network from whatever device stores
     1403the addstar tables.  In the client/server model, the addstar server
     1404always runs locally on the machine which holds the database tables.
     1405Thus, for the server, all database access is local disk access.
     1406
     1407The final advantage of the client/server model is that it enables the
     1408parallel database model, which is not yet implemented as of Jan
     14092006. In this model, there are multiple addstar servers.  Each one has
     1410a fraction of the sky in the local tables.  The identification of
     1411which table is managed by this host/addstar server is stored in the
     1412SkyRegion table.  The addstar server simply accepts incoming
     1413detections from the addstar clients.  Any detections which it receives
     1414which fall within the boundaries of tables that it manages are updated
     1415as normal.  The server then identifies the other addstar servers which
     1416are responsible for the other detections.  It then sends these
     1417detections to those servers using the same socket communication used
     1418by the addstar clients.  The addstar server must also be ready accept
     1419detections from other addstar servers.  This relationship is
     1420completely parallel, and any addstar client may send its data to any
     1421addstar server, letting the servers hash out who owns what.  The only
     1422difficulty with this model is in handling sources near the boundaries
     1423of the tables.  Note that this issues exists whether those tables are
     1424distributed across multiple machines or not.
     1425
     1426Addstar uses the following strategy to handle detections on the table
     1427boundaries.  Detections are first added to each table completely
     1428ignoring the neighboring tables.  A detection which is close to the
     1429boundary may either be associated with an average object contained
     1430within the table, or not.  If it is, the detection is associated with
     1431that average object.  If not, a new average object is created at the
     1432location of the detection.  So far, this process is identical to the
     1433behavoir in the middle of the table.  One a longer time-scale, a
     1434process is run which mediates the table boundaries. In this analysis,
     1435the two neighboring tables are simultaneously examined.  The border
     1436region, in a strip wider than the correlation radius, is examined in
     1437detail.  If two objects within the border region fall within 2x the
     1438correlation radius of each other, their individual detections are
     1439re-examined.  These detections are re-added to a temporary table which
     1440encompases the overlap.  the resulting objects will in general have
     1441detections from either side of the boundary.  The average objects are
     1442kept within the table as normal, but the detections are allowed to
     1443migrate between the tables to stay with their object.  \tbd{this
     1444boundary cleanup process is not implemented to date}.
     1445
     1446\subsubsection{Relphot : Relative Photometry Analysis}
    11401447
    11411448This operation uses the overlaps of images and multiple observations
    11421449of the same objects to determine the relative photometry zero-points
    1143 for a collection of images.  This is a task that wil be run much more
     1450for a collection of images.  This is a task that is run much more
    11441451infrequently than the object insertion tasks.
    11451452
    1146 \paragraph{Determine Consistent Photometry Zero Points (uniphot)}
     1453The relphot analysis is currently performed with a single Sky region
     1454as the starting point.  All images (or all chips from all mosaic
     1455iamges) which overlap the sky region are identified in the image
     1456table.  This set of images are considered set A.  Next, all skyregions
     1457which are overlapped by all of these images are selected.  Finally,
     1458all additional images which overlapped the new regions only are
     1459selected.  These are considered as image set B.  The image selections
     1460are also restricted to images of a single, user-selected photcode. 
     1461
     1462All of the objects and detections which are contributed by the images
     1463in sample A are extracted from the average and measure tables.  Only a
     1464subset of the detections for which the S/N is greater than a
     1465user-selected limit are kept.  Other restrictions, such as time range
     1466or instrumental magnitude ranges may also be specified.  The
     1467collection of average objects, their detections, and the images from
     1468which they were derived now define a system of photometry equations.
     1469In this system, every image has a calibration offset magnitude
     1470($M_{cal}$), every object has an average magnitude in a relative
     1471system ($M_{rel}$), and every detection of that object has a magnitude
     1472defined by the equation $M = M_{rel} + M_{cal}$.  The goal is to solve
     1473for the values of $M_{ref}$ and $M_{cal}$. 
     1474
     1475There are two points to note about this operation.  First, the system
     1476of equations is generally much too large to solve directly; we must
     1477use an iterative technique to converge on a solution. Second, it is
     1478important in the analysis to use robust averaging and identify
     1479detections, stars, or images which are deviant in some way.  These
     1480should be marked and given set weight in the solution.  These cases
     1481may represent poorly measured objects (perhaps detections on or near a
     1482bad column), variable stars, and images obtained in poor weather
     1483conditions.
     1484
     1485Relphot can also be used to determine the mosaic grid used to generate
     1486photometrically corrected flats (-grid option).
     1487
     1488\subsubsection{Uniphot : Zero Point Analysis}
    11471489
    11481490This operation uses the time history of relative photometry zero
    11491491points for images and the spatial overlap information to determine a
    11501492best set of image zero points which have a specific time scale for the
    1151 atmospheric stability.
    1152 
    1153 \paragraph{Determine Distortion and Static Astrometry Model (mosastro)}
     1493atmospheric stability.  This analysis would be used after relative
     1494photometry has been determined for data in DVO.  This analysis
     1495currently is defined to unify the zero points of a collection of
     1496disjoint regions; additional modifications will be needed to
     1497simultaneously determine consistent zero points and relative
     1498photometry corrections for a collection of images distributed over a
     1499large range in space and time, but still with significant
     1500overlap. distritions with subustanailaccount for the c
     1501
     1502\subsubsection{Global Astrometry Analysis}
    11541503
    11551504This operation uses the reference and image detections to determine an
     
    11641513ideal flat focal plane. .
    11651514
    1166 \begin{table}
    1167 \begin{center}
    1168 \caption{AP Detection Classes \& Object Parameters\label{tab:APdetections}}
    1169 \begin{tabular}{lrrrr}
    1170 \hline
    1171 \hline
    1172 Object Parameter & P2 & P4S & P4D & SS \\
    1173 \hline
    1174 PSF x,y, covar, $\alpha,\delta$               & + & + & + & + \\
    1175 PSF mag, $\sigma_{\rm mag}$                   & + & + & + & + \\
    1176 star/gal sep                                  & + & + & + & + \\
    1177 $\sigma_x$, $\sigma_y$, $\theta$              & + & + & + & + \\
    1178 local sky data                                & + & + & + & + \\
    1179 Petrosian R, M, $R_{50}$, $R_{90}$            & - & + & - & + \\
    1180 S\'ersic R, M, AB, $\phi$, $\nu$              & - & + & - & + \\
    1181 W.L. $\gamma_1$, $\gamma_2$, pol. terms       & - & - & - & + \\
    1182 exp. spaced aps., Poisson noise, variance     & - & - & - & + \\
    1183 \hline
    1184 \end{tabular}
    1185 \end{center}
    1186 \end{table}
    1187 
    1188 \subsubsection{Throughput}
    1189 
    1190 The AP Database design partly driven by the need to make the
    1191 detection-object associations quickly and to processes the incoming
    1192 detections at a sufficiently high rate to meet the throughput
    1193 requirements.  For each upload of the object detections from a
    1194 complete FPA, the AP Database must match roughly $1.4 \times 10^{6}$
    1195 detections from an FPA with roughly $6.4 \times 10^{6}$ objects,
    1196 including orphaned bright detections.  This corresponds to roughly 640
    1197 MB, if each object uses 100 bytes for its descriptive informations
    1198 (more than is currently specified in the Object table).  With a
    1199 throughput of 100 MB/s for reads from a RAID, the AP Database can
    1200 perform the data read in a fraction of a second if the data is
    1201 distributed across 10 computers.
     1515\subsubsection{DVO shell}
     1516
     1517The DVO Shell is a user-tool for examining the visualizing data stored
     1518by DVO. The DVO Shell uses the Opihi shell language structure (see
     1519also PanTasks, Section~\ref{pantasks}), which provides a rich data
     1520analysis language.  The shell language provides the user with
     1521capabilities to define new commands, set and manipulate scalar,
     1522vector, and image structures, and plot 2D graphics, including
     1523projections of the sky.  In addition, the DVO shell is aware of the
     1524DVO data tables and provides access mechanisms to these tables.  The
     1525following is a brief overview of these table access features.
     1526
     1527DVO provides several ways to access the information stored in the
     1528database. Several simple commands allow the user to extract 1-D
     1529information directly from one of the primary database tables. The
     1530fundamental such commands are:
     1531\begin{itemize}
     1532\item imextract
     1533\item avextract
     1534\item mextract
     1535\end{itemize}
     1536
     1537These commands allow the user to extract data from one of the columns
     1538represented by the image table(s), the average object tables, or the
     1539measurement tables.  The extraction places the resulting data into a
     1540vector data elements, which may be used to make plots or perform
     1541analysis.  The user may constrain the query with spatial selection, by
     1542photcode, by time ranges, and so forth.  Some examples:
     1543\begin{verbatim}
     1544   avextract all ra : select ra for all objects in displayed region
     1545   avextract all g  : select g magnitudes
     1546   avextract all mag -photcode r : select r magnitudes
     1547   avextract all Xm -photcode r  : select chisq values for r average mags
     1548\end{verbatim}
     1549
     1550Beyond these basic vector extractions, the user may perform more
     1551complex extract operations such as color-based selections.  For
     1552example, color-color diagrams can be easily made by extracting the
     1553colors from the average or measurement tables and plotting the
     1554resulting vectors.  The \code{ccd} commands extracts a specified pair
     1555of colors for all objects with that color pair from the specified data
     1556region.  Similarly, the command \code{cmd} extracts a color and a
     1557magnitude into a pair of vectors.  Both commands may specify any of
     1558the different types of magnitudes (relative, calibrated, etc)
     1559discussed above.
     1560
     1561An additional class of DVO Shell commands perform more complex
     1562graphical operation.  For example, the command \code{images}, plots
     1563the images which the specified region on the plotting too.  Other
     1564commands allow the user to extract the images or the database tables
     1565which overlap specified locations.
     1566
     1567\begin{figure}
     1568%\resizebox{4.5in}{!}{\includegraphics{pics/polar}}
     1569\caption{\label{polar} \small Map of
     1570the sky in polar project, and images added to database. }
     1571\end{figure}
     1572
     1573\begin{figure}
     1574%\resizebox{4.5in}{!}{\includegraphics{pics/fullsky}}
     1575\caption{\label{allsky} \small Map of the entire sky, and images added to database. }
     1576\end{figure}
     1577
     1578Some examples of using the DVO shell to perform visualization are
     1579given in Figures~\ref{polor} and \ref{allsky}.
    12021580
    12031581%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
     
    19242302up the analysis stages are touched upon in Section~\ref{sec:PanTasks},
    19252303which discusses the IPP Scheduler program, PanTasks.  They are
    1926 discussed in more detail in the document 'ippTools'.
     2304discussed in more detail in Section~\ref{ippTools}.
    19272305
    19282306%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
    19292307
    19302308\subsection{Phase 1: image processing preparation}
    1931 
    1932 \tbd{need to add a discussion of Phase 0}
    1933 
    1934 \tbd{need to incorporate a discussion of ppImage, etc as distinct from
    1935   the ``phases''}
    19362309
    19372310The Phase 1 analysis stage is performed on each science exposure (each
     
    27943167year, or an average rate of $\sim$2 Mpix per second, or $< 1$\% of the
    27953168object analysis in the other analysis stages.
    2796 
    2797 \section{IPPtools}
    2798 
    2799 Above, we discussed PanTasks, the IPP scheduler which determines the
    2800 new jobs to run and distributes them to computers across the network.
    2801 PanTasks is a general tool; by it self it does not define the specific
    2802 analysis tasks that the IPP requires.  The previous few sections
    2803 discussed in detail the analysis which is performed by the IPP
    2804 analysis stages.  IPPtools is the collection of PanTasks scripts,
    2805 Metadata Database interaction programs, and other tools used to
    2806 definet the specific analysis stages of the IPP.
    2807 
    2808 \tbd{this section needs to be fleshed out with a summary of the
    2809   ippTools functions.  The stand-alone IPPTools document gives a
    2810   detailed discussion of these issues}.
    28113169
    28123170%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
     
    32543612stage and the image combination stage with robust outlier rejection.
    32553613\tbd{Paul: flesh this out!}
     3614
     3615\subsection{Command Sequences}
     3616
     3617It is useful in order to understand the analysis sequence to examine
     3618the complete series of processing steps involved in the analysis
     3619stages discussed above.  We first illustrate the Phase 1-3 sequence,
     3620giving the commands which start with a raw image available on disk and
     3621results in a collection of detrended chip images, a high-quality
     3622astrometric calibration, and a collection of object detections. 
     3623
     3624\subsubsection{Phase 1-3 Analysis Commands}
     3625
     3626In the example below, we imagine a GPC image available on disk with
     3627the exposure ID 654321.obj and chip IDs 00 through 88.  The IPP design
     3628does not mandate specific naming convensions for the exposure IDs and
     3629the chip IDs; these values are opaque strings supplied by the data
     3630source (eg, GPC).  Below, in Section~\ref{ipptools}, we discuss how
     3631the names and inputs are constructed, and how the relationships are
     3632tracked between an exposure and the data containers which make up the
     3633exposure.  Also, the details of directory naming and organization are
     3634just examples, though some nightly folder scheme is a likely option.
     3635For the moment, these are assumed to be known by the system
     3636
     3637Also discussed in Section~\ref{ipptools} it the concept of multiple
     3638analysis passes for a data element.  Within the IPP, any data may be
     3639processed multiple times; the system tracks each attempt to process a
     3640particular set of data, tracking the analysis versions numbers which
     3641increment sequentially for each new attempt.  In the sequence below,
     3642we are performing the first analysis attempt on the data, so the
     3643version numbers are all 0.
     3644
     3645\begin{verbatim}
     3646Phase 1:
     3647  ppImage -recipe PHASE1
     3648          -inglob    file:/data/2006.11.01/7654321o/7654321o.??.fits
     3649          -out_astro file:/data/2006.11.01/7654321o/7654321o.XX.P1.00.ast.fits
     3650
     3651Phase 2:
     3652  ppImage -recipe PHASE2
     3653          -infile    file:/data/2006.11.01/7654321o/7654321o.24.fits
     3654          -in_astro  file:/data/2006.11.01/7654321o/7654321o.XX.P1.00.ast.fits
     3655          -outfile   file:/data/2006.11.01/7654321o/7654321o.24.P2.00.img.fits
     3656          -outmask   file:/data/2006.11.01/7654321o/7654321o.24.P2.00.msk.fits
     3657          -outvar    file:/data/2006.11.01/7654321o/7654321o.24.P2.00.var.fits
     3658          -objects   file:/data/2006.11.01/7654321o/7654321o.24.P2.00.cmf.fits
     3659          -thumb     file:/data/2006.11.01/7654321o/7654321o.24.P2.00.thm.fits
     3660          -binned    file:/data/2006.11.01/7654321o/7654321o.24.P2.00.bin.fits
     3661
     3662Phase 3:
     3663  ppImage -recipe PHASE3
     3664          -inglob    file:/data/2006.11.01/7654321o/7654321o.??.P2.00.cmf.fits
     3665          -out_astro file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.ast.fits
     3666          -objects   file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.cmf.fits
     3667
     3668  ppImage -recipe MOSAIC
     3669          -inglob   file:/data/2006.11.01/7654321o/7654321o.??.P2.00.thm.fits
     3670          -outjpeg  file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.thm.jpeg
     3671
     3672  ppImage -recipe MOSAIC
     3673          -inglob   file:/data/2006.11.01/7654321o/7654321o.??.P2.00.bin.fits
     3674          -outjpeg  file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.bin.jpeg
     3675\end{verbatim}
     3676
     3677In this example, the data are supplied by their file names in the UNIX
     3678file system.  We are invoking the capability of ppImage to accept a
     3679glob to supply a list of files.  In the Phase 1 stage, ppImage is used
     3680with the PHASE1 recipe to load data from a full mosaic (the files
     3681specified by the glob) and produce a single astrometry calibration
     3682file.  The \code{XX} is supplied for the full-mosaic output files just
     3683to make the output data products appear in a more easily readable
     3684fashion is a directory listing.  Also, note that we attach the
     3685\code{.fits} extension to all of these output files to make the data
     3686type more explicit to the reader.  Again, none of these convensions
     3687are required by the analysis programs.
     3688
     3689The Phase 2 analysis uses the Phase 1 astrometry to improve the
     3690astrometric starting guess.  The analysis for Phase 2 is illustrated
     3691for just a single chip (24), though presumably equivalent commands are
     3692executed for the other 63 chips.  The six output files selected in
     3693this example include the detrended image (\code{*.img.fits}), along
     3694with the corresponding mask (\code{*.msk.fits}) and variance images
     3695(\code{*.var.fits}) and the photometry results file
     3696(\code{*.cmf.fits}).  Note that the output object file contains the
     3697astrometric solution parameters from this analysis stage.  This
     3698process also constructs two smaller version output images: the binned
     3699image (\code{*.bin.fits}) and the thumbnail (\code{*.thm.fits}).  The
     3700binning scale for these images is specified in the recipe for the
     3701camera; the first of these for GPC would likely be binned 32x32, while
     3702the second would probably be binned 320x320.
     3703
     3704For the Phase 3 analysis stage, three actual analyses are illustrated.
     3705In the first, the photometry results files are identified by file glob
     3706and the result is an improved astrometric model for the camera and
     3707optics in our astrometry parameter table format.  The object files are
     3708also grouped into a single multi-extension file along with the
     3709astrometric calibration.
     3710
     3711In the second and third analysis examples, the collection of chip
     3712binned and thumbnail images are loaded by file glob, mosaic-ed
     3713together into a single image, and written to disk as a JPEG.  These
     3714images are used by the pipeline tracking tool, ippMonitor.  The binned
     3715image results in a full GPC image represented by 1200x1200 pixels; the
     3716thumbnail yields a 120x120 representation of the full GPC.
     3717
     3718The example above would be sufficient if we were processing a small
     3719number of images by hand or for test purposes.  However, the IPP is
     3720designed to be more flexible about the physical location of the data
     3721files that this illustration permits.  The use of nebulous allows us
     3722to use a similar naming scheme and yet place the actual data files on
     3723different hardware depending on the chip ID (among other
     3724possibilities).  To convert the filename version above to a version in
     3725which the files are stored on Nebulous simply requires changing the
     3726\code{file:} tag to \code{neb:}.  The analysis programs recognize this
     3727tag to indicate a file available from Nebulous, and make a request to
     3728Nebulous for the actual file names.  Nebulous can supply files based
     3729on a name match much like the file glob.  Nebulous also allows the
     3730storage object ID to include path-like elements, allowing a structured
     3731organization of the files within Nebulous (which does not reflect a
     3732{\em physical location} relationship).
     3733
     3734\subsubsection{Basic Detrend Creation Commands}
     3735
     3736In the following example, we examine the steps to produce master
     3737detrend images.  First, a few important points to note about this
     3738process.  The construction of a master detrend frame (bias, flat, etc)
     3739involves combining a number of individual frames of an appropriate
     3740type of exposure, possibly after some preparatory processing.  For
     3741example, in building a twilight flat-field image, 5 or 10 (or however
     3742many) raw flat-field images are first masked and bias corrected before
     3743being combined.  To build a night-time fringe-frame image, a
     3744collection of raw night-time images are bias, dark, and flat-corrected
     3745before they are combined.  In the combination, it may be necessary to
     3746apply some scaling and/or offset correction to the images to place
     3747them on a common footing.  For example, in the construction of a
     3748master flat-field image, the individual images must be normalized in a
     3749consistent fashion; in building a master fringe frame, the fringe
     3750amplitude must be used as part of the scaling applied to the input
     3751images.  In a mosaic camera, if individual chips are analysed
     3752independently, the resulting master chip images may require
     3753re-normalized to place the results on a common, consistent footing.
     3754
     3755Beyond the details of the analysis steps, there is the question of the
     3756choice of input images.  This choice is extremely dependent on the
     3757implementation for a particular camera, telescope, type of detrend
     3758image, etc.  The analysis {\em process} should not be designed to make
     3759strong assumptions about the selection of the input data.  In the IPP,
     3760the definition of the selection rules is part of the input
     3761configuration information and the scheduling rules, and can be
     3762considered outside of the discussion of the analysis commands.  The
     3763IPP provides a tool, part of the \code{dettools} suite, which examines
     3764the Metadata Database tables for raw images of the appropriate type to
     3765select input images based on selection options such as time range,
     3766filter, camera, chip, exposure type, airmass, exposure time, etc.  In
     3767the discussion below, we assume that some selection is made with
     3768\code{dettools}, resulting in a collection of input exposures and
     3769their corresponding chips.  These lists are placed in tables which are
     3770then provided as part of the input to the analysis programs below; the
     3771corresponding images used as part of these inputs are also saved in
     3772Metadata Database tables as discussed in Section~\ref{ipptools}.  In
     3773practice, the database tables provide the primary source; the list
     3774files are constructed from these tables and are simply intermediate
     3775data sources for the analysis programs.
     3776
     3777Another important distinction to clarify in the detrend processing is
     3778between the detrend {\em run}, the {\em version}, and the {\em
     3779iteration}.  These issues are discussed further in
     3780Section~\ref{ipptools}.  Briefly, though, there are the following
     3781concepts to keep in mind: The detrend {\em run} is a particular
     3782attempt to construct a master detrend image.  One run defines a
     3783collection of selection criteria for the initial set of input images.
     3784The resulting master detrend image is given an identifier, equivalent
     3785to the exposure ID.  If the same selection criteria are used multiple
     3786times (eg, for multiple experiments on the analysis recipe used to
     3787construct the image), the same detrend ID may be used for multiple
     3788detrend runs.  In this case, each new detrend run is given a different
     3789{\em version} number, equivalent to the version numbers used to track
     3790the science analysis passes.  For the detrend image construction, this
     3791concept must go one level further, however.  In order to produce a
     3792single validated master detrend image, it is necessary in general to
     3793produce multiple intermediate attempts.  The intermediate master
     3794frames are applied to the input images; the statistics of the residual
     3795images are then used to select a subset of the input images, rejecting
     3796poor quality or deviant images.  This processing is a form of
     3797image-level outlier rejection, and is particularly necessary for input
     3798images which result from observations of the sky (eg, twilight flats
     3799or night-time fringe frames); images obtained using stable calibration
     3800sources may not require this level of iterative processing.  This type
     3801of analysis can also be used to determine if a new master frame is
     3802needed (all input images internally consistent, but deviant from the
     3803current best master) or if conditions are unacceptable to produce a
     3804new master (all input images mutually inconsistent).  In order to
     3805track these multiple analysis passes, the IPP infrastructure assigns
     3806iteration numbers to the data products associated with a particular
     3807detrend run and version.
     3808
     3809One final point to address is the issue of the validity domain of a
     3810detrend image.  The end result of the detrend run is a master detrend
     3811image of a particular type, e.g., a master r' flat-field image for
     3812GPC-1.  As a result of the input selection criteria, the resulting
     3813master detrend frame will have a primary domain of validity, which
     3814consists of a particular camera, telescope, set of chips, and which
     3815may include a time range, filter, airmass range, etc.  The primary
     3816domain of validity defines those images which would be best processed
     3817with the particular master detrend image.  Beyond this primary domain
     3818of validity is a wider, relaxed domain of partial validity.
     3819
     3820Clearly, a SkyProbe flat-field image would be inapporiate in all
     3821context to be applied to a GPC image.  Likewise, a GPC-1 $r'$ flat would
     3822be inappropriate for a GPC-1 $g'$ science image.  However, in some
     3823circumstances, it is appropriate or desireable to apply a detrend
     3824frame to an image outside of its primary domain of validity.  For
     3825example, if a master flat-field image was produced using input images
     3826from a certain week, an image from a different week may be viewed as
     3827outside the primary domain; however, for some experiments or because a
     3828flat-field in the appropriate time range could not be produced, it may
     3829be acceptable to apply the out-of-date flat-field image to the science
     3830image.  In general, any image of the appropriate type, camera, filter
     3831(if a valid construct) and detector is in the partial domain of
     3832validity as a detrend image with those same values.  The detrend
     3833creation system assigns a primary domain of validity to the masters
     3834which it creates; it is the choice of the analysis routines to apply
     3835images from a more relaxed domain, if necessary or desired, or to
     3836require the primary domain and yield an error if it is not available.
     3837
     3838Below we give the series of analysis commands used to construct a
     3839master detrend frame.  In this example, we construct a master $r'$
     3840flat-field image for GPC1, using the detrend images for week 050.  In
     3841practice, we will likely build detrend frames on a nightly basis, but
     3842the choice of timescale will depend to some extent on the observing
     3843process and the stability of the system.  In this example, we
     3844construct a detrend ID using the camera, type, filter, and week
     3845number, though this choice is completely arbitrary.  We also
     3846illustrate the example for one of the input images with exposure ID
     38477654321f and for chip 24.  In this example, this is the third time
     3848this image has been used for the analysis, thus the processing results
     3849for this frame/chip are given a version number of 02.
     3850
     3851\begin{verbatim}
     3852dettools -define [selection criteria] -detID GPC1.flat.r.w050.00
     3853
     3854ppImage -recipe MKDET.PROCESS
     3855        -infile   file:/data/2006.11.01/7654321f/7654321f.24.fits
     3856        -output   file:/data/2006.11.01/7654321f/7654321f.24.PC.02.img.fits
     3857
     3858ppMerge -recipe MKDET.STACK
     3859        -inlist   file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c24.list
     3860        -output   file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c24.fits
     3861
     3862ppNorm  -recipe MKDET.NORM
     3863        -inglob   file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c??.fits
     3864
     3865ppImage -recipe MKDET.RESID
     3866        -infile   file:/data/2006.11.01/7654321f/7654321f.24.PC.02.img.fits
     3867        -output   file:/data/2006.11.01/7654321f/7654321f.24.RS.06.img.fits
     3868        -thumb    file:/data/2006.11.01/7654321f/7654321f.24.RS.06.thm.fits
     3869        -binned   file:/data/2006.11.01/7654321f/7654321f.24.RS.06.bin.fits
     3870
     3871ppImage -recipe MOSAIC
     3872        -inglob   file:/data/2006.11.01/7654321o/7654321o.??.RS.06.bin.fits
     3873        -outjpeg  file:/data/2006.11.01/7654321o/7654321o.XX.RS.06.bin.jpeg
     3874
     3875ppImage -recipe MOSAIC
     3876        -inglob   file:/data/2006.11.01/7654321o/7654321o.??.RS.06.thm.fits
     3877        -outjpeg  file:/data/2006.11.01/7654321o/7654321o.XX.RS.06.thm.jpeg
     3878
     3879ppImage -recipe MKDET.MOSAIC.
     3880        -inglob   file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c??.fits
     3881        -outjpgt  file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.thm.jpeg
     3882        -outjpgb  file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.bin.jpeg
     3883
     3884\end{verbatim}
     3885
     3886Again, this data illustrates the use of files in the UNIX file system;
     3887the substitution of \code{neb:} for \code{file:} will inform the
     3888programs to retrieve the file names via Nebulous.  In this example, we
     3889use output names for the intermediate images which are equivalent to
     3890those used for the science processing; the version numbers used for
     3891these data products are sequential over all detrend runs which use
     3892those input frames.  Also, for the detrend image products, we have
     3893added the 'v', 'n', 'c' tags to clarify which of the two-digit numbers
     3894represents a version (v), an iteration (n), and a chip ID (c).  Note
     3895that this sequence of analysis steps makes heavy use of
     3896\code{ppImage}, with different choices of the recipe and the output
     3897options to change the behavior somewhat.  However, all of the uses of
     3898\code{ppImage} represented here are consistent with the primary
     3899responsibilities of \code{ppImage}: read in the file or mosaic image
     3900into the correct level of the image hierarchy, perform a detrend
     3901analysis (including rebinning in this category), re-structure the
     3902collection of image arrays as described by the recipe, write out the
     3903image in the desired format.
     3904
     3905\section{IPPTools}
     3906
     3907PanTasks is the IPP tool which manages the sequencing of data analysis
     3908steps and, with the related tool `PControl', distributes the data
     3909processing across a cluster of computers. However, by itself, PanTasks
     3910does not determine the organization of data or the analysis sequencing
     3911for a particular pipeline.  This level of information is contained
     3912within specific PanTasks scripts.  To use the tasks defined by a set
     3913of PanTasks scripts, additional helper programs are needed.  This
     3914section discusses these programs and the PanTasks scripts used by the
     3915IPP.
     3916
     3917IPPTools is a collection of programs, Metadata Database table
     3918definitions, and PanTasks scripts used to define the actual data
     3919organization and the sequencing of operations by the IPP.  Within the
     3920IPP, the Metadata Database is used to store the analysis state, as
     3921well as result processing data points.  This section discusses the
     3922tasks needed to define each of the IPP analysis stages (Phase 1-4,
     3923detrend creation, etc) and examines the relevant MDDB tables. 
     3924
     3925\subsection{Persistent vs Ephemeral State in PanTasks}
     3926
     3927\begin{figure}
     3928\begin{center}
     3929\includegraphics[scale=0.85]{pics/ipptools.01.ps}
     3930\caption{\label{queues} PanTasks queues and MDDB tables}
     3931\end{center}
     3932\end{figure}
     3933
     3934The IPP, a fairly complex analysis system, uses PanTasks to select
     3935jobs, distribute them to the cluster, and harvest the results.  It
     3936uses the Metadata Database to record the results of a given analysis
     3937step, and to determine which jobs must be performed when.
     3938
     3939There are some subtleties in the interaction between PanTasks, the
     3940Metadata Database tables which store the system state, and the jobs
     3941which are currently being performed.  There is a choice to be made
     3942between rigorously maintaining the system state in the Metadata DB at
     3943all times or keeping an intermediate set of state tables.  Keeping the
     3944exact system state in the Metadata DB tables would require many extra
     3945queries to/from the database and may introduce additional latencies
     3946which are undesirable.  This is because any attempt by PanTasks to
     3947initiate a new job would require PanTasks to mark the corresponding
     3948data item in the Metadata DB (the item which acts as the trigger) with
     3949a `pending' state, and then mark it again as `done' when the job
     3950actually completes.  This also has the drawback that, if the system
     3951crashes (eg, hardware failure), some initial process would be required
     3952on start up to find all Metadata DB items which are in the `pending'
     3953state (examining all possible items which can be in such a state) and
     3954reset them to the `new' state.
     3955
     3956We implement an alternative in which PanTasks maintains an internal,
     3957ephemeral stack of the pending jobs, and only updates the system state
     3958entries in the Metadata DB when jobs are actually completed.  In this
     3959scenario, as far as the Metadata DB tables are concerned, data items
     3960transition only between a `new' and a `done' state.  Any jobs which
     3961are pending when the system crashes or the power is lost are simply
     3962dropped, and will be automatically re-constructed when the system
     3963restarts.  In this paradigm, no intermediate operation state is saved,
     3964and no partially completed job can be recovered.  Since the IPP is
     3965defined in terms of a fine granularity, with jobs lasting no more than
     396630 - 120 seconds, crashes under this model will not have a large
     3967impact on the data processing.
     3968
     3969Figure~\ref{queues} illustrates this ephemeral vs persistent state
     3970information and the interrelation between the metadata tables and
     3971PanTasks.  The left-hand portion of the diagram illustrates the
     3972recommended interaction between the metadata database tables and
     3973PanTasks' internal queues.  Some table in the metadata database
     3974defines a list of data items which are to be processed by some
     3975analysis job.  PanTasks uses a two-step approach to define the
     3976analysis jobs based on this list.  First, one task queries the MDDB
     3977for a list of pending items, adds the returned items to an internal
     3978PanTasks queue.  The process of adding the elements to the queue is
     3979defined so that only unique items are added: already existing items
     3980are skipped.  The entries in the queue consist of the data items of
     3981interest and an internal temporary state.  At first, this would be
     3982`pending'.  A second tasks pops `pending' entries one-by-one from this
     3983internal queue, submits a job based on the entry, and sets the
     3984temporary state in the internal queue to `running'.  The internal
     3985state is needed to prevent PanTasks from re-submitting a job for the
     3986same data item before the first job is done or assessed.  Since the
     3987job make take an arbitrary amount of time, PanTasks requires a
     3988mechanism to remember which data items it has already submitted.  When
     3989the job eventually completes, the metadata database table is updated
     3990noting the completion.  This may be done either by the job itself or
     3991by PanTasks as part of the job exit rules.  In addition, the state of
     3992the entry in the queue can be set to either `done' or the entry can be
     3993simply removed from the queue.
     3994
     3995
     3996The purpose of this interaction is to maintain the temporary state
     3997information within non-persistent elements of PanTasks rather than
     3998using the metadata database tables to store this information.  This
     3999concept has two advantages.  First, PanTasks internal queues are in
     4000memory and relatively small, thus interfacing with them is quite fast
     4001for PanTasks -- this should reduce the system latency.  Second, by
     4002keeping this information non-persistent, the system responds correctly
     4003to stopping and restarting PanTasks.  Any jobs which have not been
     4004completed will not be marked in the database, and will be restarted
     4005naturally by PanTasks.  The alternative, of writing a temporary state
     4006marker in the database would require PanTasks, on startup, to
     4007initially clean all database tables of these temporary state markers.
     4008
     4009The right-hand portion of the diagram illustrates this process using
     4010the process of copying the images from the summit as an example.  The
     4011metadata database table of interest in this case is the list of
     4012pending images, with entries supplied by a job which queries the
     4013summit data systems.  The job which is actually performed is a remote
     4014copy of the image file from the location specified by the summit data
     4015system to the appropriate location within the IPP Image Server
     4016(Nebulous).  (As an alternative to the above, the `pending images'
     4017table may be part of the summit database system, and the `get images'
     4018command may query the summit directly.  In this scenario, the `copy
     4019image' command reports to the summit data system that an individual
     4020image file has been copied.)
     4021
     4022In the rest of this document, the use of PanTasks internal queues to
     4023manage the temporary data states is glossed over and assumed part of
     4024the tasks defined in the process.
     4025
     4026
     4027\subsection{IPP Pipelines Overview}
     4028
     4029The IPP as a whole performs all of the image analysis functions
     4030required by the Pan-STARRS telescopes, including images from the full
     4031Gigapixel camera (or cameras), the test camera TC-3, and the SkyProbe
     4032camera.  The IPP is designed to be very flexible, with instrument
     4033specific details isolated in configuration files associated with the
     4034different cameras known to the system.  As a result, the organization
     4035of the top level analysis infrastructure must be sufficiently general
     4036that a wide range of cameras can be accomodated.  We have a few
     4037general principles regarding constraints on the data to be processed
     4038which are used to guide the IPP design and developement:
     4039
     4040\begin{itemize}
     4041\item {\bf Camera Focal Plane Hierarchy} The IPP analysis programs
     4042  assume that the images to be processed are obtained by a camera
     4043  which can be represented by our Camera Focal-Plane Hierarchy of data
     4044  structures.  This hierarchy is discussed in detail in the Modules
     4045  SDRS, and defines a top-level {\em Focal-Plane Array (FPA)}, which
     4046  may contain 1 or more {\em Chips}, each of which may contain one or
     4047  more {\em Cells}.  An {\em FPA} is identified as having a single
     4048  optical system feeding photons to the detectors.  A {\em Chip} is
     4049  identified as a unit of data all deriving from a single detector
     4050  (piece of silicon), while a {\em Cell} is identified as a collection
     4051  of pixels read out as a continuous cartesian grid.  Finally, a
     4052  single collection of data from an {\em FPA} may include multiple
     4053  {\em Readouts} from any or all of the {\em Cells}. 
     4054
     4055\item {\bf Exposures vs Groups} The processing presumes that the data
     4056  is organized into {\em exposures} and exposure {\em groups}.  An
     4057  exposure represents the data from a single FPA, with the possible
     4058  subdivision of the exposure into multiple readouts for some or all
     4059  of the cells.  Exposure {\em Groups} are any group of exposures
     4060  which are related together in some way; the definition of the {\em
     4061  Groups} may be provided by the observers, or they may be derived
     4062  from the characteristics of the exposures.  The use of a particular
     4063  {\em group} depends on the context of that group.  A few examples of
     4064  exposure groups:
     4065
     4066  \begin{itemize}
     4067  \item a dithered sequence of exposures to be stacked for cosmetics
     4068  and improved signal-to-noise.
     4069  \item a twilight flat-field sequence.
     4070  \item all images of the same filter within a 10 degree region to be
     4071  used to construct an sample astrometric reference. 
     4072  \end{itemize}
     4073
     4074\item {\bf Image Files (imfiles) vs Exposures}  Any single exposure
     4075  may consist of a number of different data files.  The number of {\em
     4076  imfiles} for a given exposure will depend on the camera, as will the
     4077  data organization within those image files.  Also, a particular
     4078  camera will supply files corresponding to one of the particular
     4079  Focal-Plane Hierarchy elements.  The IPP analysis must be able to
     4080  interpret the incoming data correctly.
     4081\end{itemize}
     4082
     4083As discussed elsewhere, there are several major types of analysis
     4084performed by the IPP.  For the purposes of data organization and
     4085parallel processing efficiencies, we have identified the following
     4086divisions of the analysis tasks.  These will be discuss in much more
     4087detail below.
     4088
     4089\begin{itemize}
     4090\item {\bf Science Image Analysis} : This represents the analysis
     4091  performed on the images obtained by the telescope, and generally
     4092  performed in real-time, night-by-night.  The science image analysis
     4093  tasks are further subdivided as follows:
     4094
     4095  \begin{itemize}
     4096  \item {\bf Phase 1} : The full focal-plane array is examined quickly
     4097  to determine an initial astrometric calibration.  In this step, the
     4098  OTA guide stars may be used as the astrometric reference; if none
     4099  are available, predicted bright star positions are examined.  This
     4100  step is only used for mosaic images, and may be skipped if no guide
     4101  stars are available {\em and} the astrometric calibration for the
     4102  telescope / camera is reliable (better than 10 arcseconds).
     4103
     4104  \item {\bf Phase 2} : Each image file is analysed independently: the
     4105  image is detrended (bias, dark, flat, fringe, etc), sources are then
     4106  detected to a modest level, improved astrometric calibration is
     4107  performed.
     4108
     4109  \item {\bf Phase 3} : The collection of sources measured from all of
     4110  the image files for the camera are used to determine a global
     4111  astrometric, and possibly photometric, solution for the exposure.
     4112  This step is only required for mosaic cameras.
     4113
     4114  \item {\bf Phase 4.1} : An exposure group consisting of images
     4115  obtained in a specific region of the sky are merged together.  In
     4116  this step, the images are first warped to a common pixel grid, defined by
     4117  the static sky images.  The collection of images are then used to
     4118  construct a single, cleaned image by rejecting the outliers from the
     4119  source images in the stack.  The corresponding static sky pixels are
     4120  then used to construct a difference image from the resulting stack.
     4121
     4122  \item {\bf Magic} : In this step, the difference images are examined
     4123  to find the trailed images introduced by artificial satelites.
     4124  These so-called {\em streaks} are excised from the difference
     4125  images, as well as all of the source images which were used to
     4126  generate the difference images; the public data sources are updated
     4127  with the precise, correct time.  Note that this step requires that
     4128  separate difference images be generated for each of the input
     4129  images, a step which would be skipped if {\em magic} were avoided.
     4130  Also note that, until {\em magic} is performed, the publically
     4131  available time has a limited precision (probably $\sim 1$ minute
     4132  errors).  This step is only necessary in the operational IPP system
     4133  given the restrictions from the Air Force.
     4134
     4135  \item {\bf Phase 4.2} : After {\em magic} the final difference and
     4136  the final cleaned stacked image are produced and objects in both
     4137  images are detected.  The difference sources are used to mask the
     4138  extreme outliers in the cleaned stack, which is then used to update
     4139  the Static Sky images.
     4140  \end{itemize}
     4141
     4142 \item {\bf Static Sky Image Analysis} : While the science image
     4143 analysis is performed as images are availablef, the static sky image
     4144 analysi occurs on a very different timescale.  In steady state, the
     4145 full static sky analysis will take place over the course of a full
     4146 year.  At any given time, the portion of the sky corresponding to the
     4147 location of the sun will be under-going the analysis.  In practice,
     4148 for PS-1, the static sky is produced in a somewhat different fashion
     4149 than in the steady-state model.  In PS-1, the different survey
     4150 strategies introduce very different update rates for the static sky.
     4151 At one extreme, the AP Survey will not have enough data for a
     4152 complete static sky analysis until nearly 22 months after the survey
     4153 begins.  At the other extreme, the deep survey, which observes a much
     4154 smaller portion of the sky, may best be analysed quite frequently.
     4155 These details are part of the science guidelines of the PS-1 surveys,
     4156 and are beyond the scope of this document.  Rather, the IPP Static
     4157 Sky Image Analysis must provide the capability of defining the static
     4158 sky analysis in a flexible and dynamic fashion.
     4159
     4160\item {\bf Basic Detrend Creation Analysis} : The analysis of most of
     4161  the detrend data is grouped together in a common analysis stage.
     4162  The differences between the analysis of the bias, dark, flat, and
     4163  fringe images is primarily one of how the input images are
     4164  pre-processed, what statistic is used to characterize a given input
     4165  image, how the input images are scaled before being combined, and
     4166  what normalization is applied to the resulting image.  All of these
     4167  types of detrend images can thus be processed with a single analysis
     4168  pipeline which is made aware of these minor differences.  This stage
     4169  is never the less fairly complex, and as a result is subdivided into
     4170  several compenents, as discussed below.
     4171
     4172\item {\bf Other analyses} There are a number of other tasks which the
     4173  IPP must perform that are not well-defined by the different analysis
     4174  types discussed above.  Some analysis tasks are not automatically
     4175  triggered, and are thus outside the scope of this document; these
     4176  are the tasks which are more properly considered as research
     4177  projects than analysis systems.  The other important automatic tasks
     4178  are:
     4179  \begin{itemize}
     4180    \item {\bf Summit Copy} : In this stage, the data source or data
     4181    sources are queried for new exposures and image files, which are
     4182    then copied to the IPP data area.  This stage also includes the
     4183    copying of other metadata which are not included in the image
     4184    files.
     4185   
     4186    \item {\bf Image Classification} : new images which are introduced
     4187    to the IPP are examined by this analysis stage and placed in the
     4188    appropriate table for processing.  This step includes a small
     4189    amount of accumulating statistics about the images.
     4190
     4191    \item {\bf Data File management} : a few tasks are necessary to
     4192    monitor and maintain the clustered storage system.  These tasks
     4193    include the automatic duplication and deletion of different types
     4194    of files from Nebulous, the file storage archive.  This also
     4195    includes automatic redistribution of machine assignments as
     4196    hardware is added or removed from the system.  This collection of
     4197    tasks also includes monitoring of system parameters to alert
     4198    people in case of dangerous hardware situations.
     4199
     4200    \item {\bf Irregular Calibration Data} certain types of
     4201    calibration information is extracted on different intervals from
     4202    the more regular detrend images.  These types of calibration data
     4203    include improved telescope pointing models, astrometric
     4204    calibrations, photometric calibrations, flat-field correction
     4205    frames.
     4206  \end{itemize}
     4207\end{itemize}
     4208
     4209\subsection{Tables, Tasks and Tools}
     4210
     4211The following sections discuss the database tables, the tasks within
     4212PanTasks, and the collection of programs used by PanTasks to examine
     4213and manipulate the state tables.  These later programs do not, in
     4214general, perform any in depth analysis; instead they perform actions
     4215such as selecting from one table images ready for analysis in a
     4216following processing step.  This collection of tools is grouped under
     4217the name of the {\tt ippTools}, and consists of a separate tool for
     4218each of the different major analysis steps.
     4219
     4220The {\tt ippTools} make use of {\em glueforge} to simplify the
     4221management of the database table schema.  Glueforge provides a single
     4222mechanism to generate a collection of C data structures, database
     4223tables, database access APIs, and I/O routines from a simple table
     4224description configuration file.  All APIs generated by {\em glueforge}
     4225for the same type of interaction have common naming schemes.  This
     4226technique has several important advantages.  It makes the writing of C
     4227database interactions very quick and easy.  It also makes it easy to
     4228modify the database schema without disrupting the software
     4229development.  Finally, it provides a simple, self-documenting source
     4230for data structure of multiple types which can be shared between
     4231programs or platforms.
     4232
     4233Within the following diagrams, we illustrate the database tables used
     4234to track the state of the IPP.  We also show the commands provided by
     4235{\tt ippTools} to connect the tables.  Finally, we show the IPP tasks
     4236which initiate the different analysis steps.  The following set of
     4237diagrams uses several consistent features.  The blue-and-grey
     4238rectangles define the metadata database tables.  The blue section
     4239contains the table name, while the grey section lists a minimal subset
     4240of the table columns.  The ellipses represent programs (or program
     4241portions in some cases) executed by PanTasks.  The blue filled
     4242ellipses represent the {\tt ippTools} commands which are executed
     4243locally on the computer hosting PanTasks.  The grey-blue ellipses
     4244represent the commands executed on the parallel cluster, monitored by
     4245{\tt pcontrol}.  The green ellipses represent commands executed by
     4246hand for testing and manual intervention.
     4247
     4248In most of the analysis tasks, we use a two-table approach to the data
     4249in order to avoid excessive latencies.  One table is used to track
     4250quantities which are still pending for a particular stage.  When the
     4251analysis is completed, these items are moved from the 'pending' tables
     4252to corresponding 'done' tables.  Although this introduces a somewhat
     4253higher number of tables and complexity, it will avoid the system from
     4254slowing down as the number of data items grows with time.  The pending
     4255tables are searched repeatedly by the {\tt ippTools} programs as they
     4256attempt to select new data of interest.  In contrast, the done tables
     4257are searched much less frequently. 
     4258
     4259\subsection{Summit Copy Tasks}
     4260
     4261\begin{figure}
     4262\begin{center}
     4263\includegraphics[scale=0.85]{pics/ipptools.02.ps}
     4264\caption{\label{pcopy} Summit Copy Tasks}
     4265\end{center}
     4266\end{figure}
     4267
     4268Figure~\ref{pcopy} illustrates the MDDB tables used to copy data
     4269(images and metadata tables) from the summit.  The left-hand portion
     4270of the diagram shows the tables involved in copying images from the
     4271summit system.  The table of pending image files lists the URLs of the
     4272individual image files available for transfer, along with their
     4273associated exposure ID and the camera which generated the image.  Two
     4274other entries assist in interpreting the file: the class and the class
     4275ID.  The final entry in this table is the current copy state of the
     4276file, can have the value of `ready' or `copied'.
     4277
     4278The class defines the data grouping represented by this image file and
     4279may have values of: FPA, Chip, Cell.  This value indicates that the
     4280provided image file represents the specified portion of the camera
     4281FPA.  If the value is FPA, the file represents data from a complete
     4282FPA, though the file may contain pixel data in multiple extensions or
     4283other groupings to be identified later.  If the value is chip, the
     4284file contains only data for a single chip, presumably of multiple
     4285chips available, and equivalently for Cell.  Further discussion of the
     4286FPA image hierarchy is given in the IPP documents (eg, Modules SDRS).
     4287The class ID gives the identifier used to name the class level
     4288corresponding to this file.  This value is necessary to make decisions
     4289on how to copy the data based on the chip / cell before the data is
     4290available to IPP components.  Table~\ref{classes} lists likely values
     4291for the class and class ID for some common cameras.  The system
     4292described is sufficiently flexible to allow us to transfer the GPC
     4293images by cell if we eventually decide that is more efficient.
     4294
     4295The copy process copies the file from the given URL to the appropriate
     4296IPP node and adds an entry to the table of new image files, consisting
     4297of the same information as the pending image file table, though with a
     4298new value for the URL.  This URL may be an explicit filename, a
     4299reference to an entry in the image server, or a web address, or
     4300located on the image server (marked with file:, neb:, and http:,
     4301respectively).  (TBD: other possible file storage types?  perhaps the
     4302path could be abstracted without going to the level of the image
     4303server?  eg: ref:DIR0001/file0001.fits might be in a directory which
     4304is defined in a table of directories.) After an image file is
     4305successfully copied, the corresponding state in the `pending chip'
     4306table is updated from `ready' to `copied'.
     4307
     4308\begin{table}
     4309\begin{center}
     4310\caption{Camera and Data Classes\label{classes}}
     4311\begin{tabular}{llll}
     4312\hline
     4313\hline
     4314camera   & class  & classID \\
     4315\hline
     4316GPC      & chip   & chip02 \\
     4317skyprobe & fpa    & sp01 \\
     4318Megacam  & fpa    & MegacamSpliced \\
     4319Suprime  & chip   & chip0 \\
     4320\hline
     4321\end{tabular}
     4322\end{center}
     4323\end{table}
     4324
     4325The right hand portion of this diagram illustrates the process of
     4326copying a metadata table.  The table of pending tables lists the URLs
     4327for the tables which are ready, a unique table ID for each table, and
     4328the table type.  The copy function copies the listed table and uploads
     4329the data to the IPP version of the same metadata database.  Two
     4330examples of metadata tables needed by the IPP for the basic image
     4331processing system are illustrated: the table of new exposures and the
     4332table of pending matches.  The first lists the exposures which are
     4333avilable from the summit system, and all represent entries which are
     4334available from the Image server.  the second represents the matches
     4335between exposure IDs and chips
     4336
     4337\subsection{Phase 0}
     4338
     4339\begin{figure}
     4340\begin{center}
     4341\includegraphics[scale=0.85]{pics/ipptools.03.ps}
     4342\caption{\label{phase0} Phase 0 Tasks}
     4343\end{center}
     4344\end{figure}
     4345
     4346Figure~\ref{phase0} illustrates phase 0, in which the image files are
     4347categorised, examined for summary information and basic statistics,
     4348and moved to the later phase 'pending' tables to trigger further
     4349analysis.  The command {\tt p0search -pending} examines the `new
     4350imfiles' and 'new exposure' tables.  It selects images from this table
     4351which have not yet been examined (state is `new').  These are returned
     4352to PanTasks, which sends each image file to a separate analysis node
     4353running the {\tt p0search -update} command.  With this command, the
     4354file header is examined and relevant metadata is extracted (eg, RA,
     4355DEC, times, and so forth to be defined later).  The process may also
     4356select a portion of the image pixel data to determine a rough bias and
     4357background level.  These statistics, whether derived from the header
     4358or the pixel values, are placed along with image summary information
     4359in the `raw image files' table, and the state field of the `new image
     4360files' table is set to `ready'.
     4361
     4362The {\tt p0search -update} command is also responsible for moving the
     4363exposures to the tables used for triggering the analysis process.  If
     4364the image class is FPA, the image can be advanced without waiting for
     4365any other image files.  If the class is Chip or Cell, the process must
     4366also examine the `new exposure' table for this exposure ID.  The
     4367number of class files available for this exposure is listed in this
     4368table.  The process must the select all image files matching the
     4369exposure ID with state of `ready' and compare the number avalable to
     4370the number expected.  If the two match, then a new exposure is ready.
     4371Based on the image type (from the most recently examined image file
     4372header or new exp table?), the exposure is added to the `raw exposure'
     4373table for images of that type.  The allowed types are `detrend', (all
     4374bias, dark, flat images), `object', `focus'(??), etc.  (** The
     4375different tables represent different analysis modes.  This process
     4376also adds an entry to the exp ID / image file match **).  This process
     4377also adds all science (OBJECT) exposures to the P1 exposure table (for
     4378mosaic data) or the P2 chip table (for single detector data).  These
     4379tables are used to trigger the Phase 1 and Phase 2 analysis stages.
     4380
     4381\subsection{Phase 1}
     4382
     4383\begin{figure}
     4384\begin{center}
     4385\includegraphics[scale=0.85]{pics/ipptools.04.ps}
     4386\caption{\label{phase1} Phase 1 Tasks}
     4387\end{center}
     4388\end{figure}
     4389
     4390Figure~\ref{phase1} shows the tables involved in running the Phase 1
     4391analysis stage.  There are paths for exposures to enter the analysis
     4392automatically from the Phase 0 analysis (arrow on left) or to be added
     4393manually based on a selection from the raw exposure table.  Exposures
     4394to be analysed by Phase 1 are added to the P1 exposure table with the
     4395state `new'.  Exposures may be added multiple times for processing and
     4396reprocessing. The P1 done exposure table keeps a record of the old
     4397attempts for debugging and analysis.  Each time an exposure is added
     4398to the P1 exp table, it is given a new, unique version number,
     4399allowing the system as a whole to track different analysis attempts.
     4400This method is used in all of the image analysis stages (and
     4401extrapolated to iterations in the detrend analysis steps below).  The
     4402top portion of the diagram shows the use of the command {\tt p1search
     4403-define} to select and submit an exposure or a group of exposures,
     4404potentially selected on the basis of a query from the raw science
     4405exposure table.
     4406
     4407The P1 pending exposure table is examined by {\tt p1search -pending}
     4408to select the new exposures, which are sent to PanTasks.  PanTasks
     4409initiates a separate analysis job (p1astro) for each exposure, which
     4410are sent to the parallel processing nodes.  Within the analysis job,
     4411the chips (image files) associated with the exposure are select from
     4412the raw image file table.  The analysis examines the contents of these
     4413files, either extract the guide star information from the image files
     4414(GS table extension) or searches for and centroids the pixels on
     4415appropriate bright stars.  The analysis results in astrometric
     4416calibration terms which are written to the astrometric calibration
     4417file for this exposure.  The location of the astrometric calibration
     4418file and the statistics of the measurement are written back to the P1
     4419exposure table.  The images associated with exposures which are
     4420successfully processed by P1 are then added to the P2 image table,
     4421which is used to trigger the Phase 2 analysis.  This last step is
     4422performed by the command {\tt p1search -done}, which is executed
     4423regularly to search for completed Phase 1 jobs.
     4424
     4425\subsection{Phase 2}
     4426
     4427\begin{figure}
     4428\begin{center}
     4429\includegraphics[scale=0.85]{pics/ipptools.05.ps}
     4430\caption{\label{phase2} Phase 2 Tasks}
     4431\end{center}
     4432\end{figure}
     4433
     4434Figure~\ref{phase2} shows the tables involved in running the P2
     4435analysis stage.  There are paths for images to enter the analysis
     4436automatically from the P1 analysis (arrow on left) or to be added
     4437manually based on a selection from the raw exposure and raw image file
     4438tables.  Image files to be analysed by Phase 2 are added to the P2
     4439pending imfiles table with the state `new'.  When images are added to
     4440this table, a single entry is also added to the P2 exposure table
     4441listing the P1 and P2 versions for this exposure.  These version
     4442numbers must be integers starting with 1.  If this image did not have
     4443a P1 analysis, the P1 version is set to 0.  Exposures may be added
     4444multiple times for processing and reprocessing. The P2 image table
     4445keeps a record of the old attempts for debugging and analysis.  As
     4446with P1, each time a collection of associated images from an exposure
     4447is added to the P2 image table, it is given a new, unique version
     4448number, allowing the system as a whole to track different analysis
     4449attempts.  Note that these version numbers are unique for each {\em
     4450exposure} processed by Phase 2, not just for any image file.  The top
     4451portion of the diagram illustrates the behavior of the commands {\tt
     4452p2search -define} and {\tt p2search -quick}.  The first may be used to
     4453re-submit the images for an exposure or a group of exposures,
     4454potentially selected on the basis of a query from the raw science
     4455exposure and raw image file tables.  The second version sends images
     4456files directly to PanTasks for processing; these entries will not be
     4457included in the processing tables, and is used only for testing
     4458purposes.
     4459
     4460The P2 pending image table is examined with the command {\tt p2search
     4461  -pending} to select the `new' images.  These images are used by
     4462PanTasks to generate P2 analysis jobs, running the analysis command
     4463{\tt ppImage}.  The P2 analysis uses the input url to find and load
     4464the image file.  The url may be a file on disk, an entry in the image
     4465server, Nebulous, etc.  The master detrend images matching the
     4466specific science image and the conditions are selected by examining
     4467the table of master detrend frames.  The specific detrend image files
     4468are selected by using the master detrend ID to select the matching the
     4469entries in the table of master detrend files.  After the analysis, the
     4470output image, mask, and FITS table of objects, including the
     4471astrometry calibration, are written back to the P2 image table, along
     4472with summary statistics from the P2 analysis.  The state is also
     4473updated (to `done').
     4474
     4475The completed images are examined by the command {\tt p2search -done},
     4476and when all image files for a single exposure are completed, this
     4477command migrates them to the P2 done table.  This process is also
     4478responsible for populating the P3 pending tables so exposures may be
     4479processing by Phase 3.
     4480
     4481\subsection{Phase 3}
     4482
     4483\begin{figure}
     4484\begin{center}
     4485\includegraphics[scale=0.85]{pics/ipptools.06.ps}
     4486\caption{\label{phase3} Phase 3 Tasks}
     4487\end{center}
     4488\end{figure}
     4489
     4490Figure~\ref{phase3} illustrates the tables and commands involved in
     4491the Phase 3 analysis.  The P3 pending exposure table lists the
     4492exposure ID, the P3 analysis version, the P2 analysis version to be
     4493used as input to this P3 analysis, and the recipe to be used.  The
     4494command {\tt p3search -pending} extracts exposures from this table and
     4495provides them to PanTasks for processing.  PanTasks launches a Phase 3
     4496analysis (the command {\tt psastro}?) for each exposure.  In this
     4497analysis, the P2 exposure and image tables are used, in conjunction
     4498with the P2 version information, to select the P2 output measured
     4499objects and the astrometric calibrations from P2 and P1.  These
     4500measured objects are matched with the reference catalog objects, and
     4501calibrated astrometry {\em and eventually photometry} is produced for
     4502the full exposure.  The location of the resulting astometry
     4503calibration table is stored back in the P3 exposure table.  If the
     4504recipe file specifies, the 2-D photometric and background / fringe
     4505corrections may also be performed at this stage.  Since these analyses
     4506require reference data, the recipe may also be used to skip these
     4507analysis if such reference data is unavailable or unreliable.  At the
     4508end of Phase 3, the objects from the exposure are inserted into the
     4509photometry database (this is not shown).
     4510
     4511The astrometric calibration portion of Phase 3 is principally needed
     4512for a mosaic camera.  For single-chip cameras, Phase 3 may be used to
     4513perform the photometric calibration and simply pass the astrometric
     4514results along to the output file to be listed in the P3 exposure
     4515table.  In this way, later stages of the analysis (ie, Phase 4) can
     4516use the P3 exposure table as input for all cameras, even if all the
     4517funcionality of Phase 3 is not necessary for that camera.  This would
     4518be the case for the skyprobe camera, for example.
     4519
     4520\subsection{Phase 4}
     4521
     4522\begin{figure}
     4523\begin{center}
     4524\includegraphics[scale=0.85]{pics/ipptools.07.ps}
     4525\caption{\label{phase4} Phase 4 Tasks}
     4526\end{center}
     4527\end{figure}
     4528
     4529At the end of Phase 3, the images are ready for Phase 4.  The Phase 3
     4530diagram shows the output line adding the exposures to be processed by
     4531Phase 4 to a Phase 4 table.  However, this line is just for
     4532illustration purposes.  The rules for initiating a Phase 4 run are
     4533somewhat more complicated than those for running Phases 1-3.  Groups
     4534of exposures which have an appropriate overlap should be chosen for
     4535the Phase 4 analysis.  In the steady-state period of PS-4, it may be
     4536straightforward to choose the exposure groups: they would simply be
     4537the exposures obtained nearly simultaneously by the four separate
     4538cameras.  The circumstance for PS-1 will be much more complicated (and
     4539even PS-4 will probably be more complex than it seems at first
     4540glance).  For example, in PS-1, we will not have a static sky for most
     4541of the AP Survery.  In this circumstance, we cannot run P4, at least
     4542until after the complete AP Reference catalog is built, and
     4543potentially all exposures re-run through Phase 3.  It may be useful
     4544for the AP Survey data to split the Phase 4 analysis into two stages:
     4545image combination and image differencing.  It may even be the case
     4546that only the combination portion of Phase 4 is performed on the AP
     4547Survey data.
     4548
     4549More generally, the image groups selected for Phase 4 analysis may be
     4550chosen on the basis of a query of the AP Database (DVO) with some
     4551rules. 
     4552
     4553\note{Phase 4 run can be defined by selecting an observation group, a
     4554  set of exposures, or a set of rules related to a spatial region (eg,
     4555  region, time range, and filter}.
     4556
     4557\note{Phase 4 discussion (and diagram) needs more work}
     4558
     4559\subsection{Analysis Version and Recipes}
     4560
     4561Note that each of the stages P1-P4 refer to the processing version
     4562from the previous stage.  This allows the processing stage to request
     4563the correct version of the results from the previous stage, and makes
     4564it possible to run and re-run the analysis at any stage without
     4565deleting the earlier results.  As different analysis attempts are
     4566performed for a given image, the versions branch out.
     4567
     4568Also note that at every stage, the entries include a recipe
     4569identifier.  This is used to select the analysis recipe which should
     4570be used for this version.  By default, the recipe should be set to the
     4571current best recipe (use a default name for this?).  This feature
     4572allows the user to run test analyses with variations on the recipe
     4573without altering the analysis system.  For example, it is possible to
     4574use a different flat-field set by specifying alternate rules for the
     4575flat-field selection in a recipe file.  If it is necessary to run the
     4576P1-P3 analysis with the raw master flats, for example, the user simply
     4577defines that selection in the recipe file and submits the images of
     4578interest to P1 (or P2, etc), with the corresponding entry for the
     4579recipe.
     4580
     4581The recipe file may also be used to specify alternative analysis paths
     4582and desitinations.  For example, it is not necessary that all analysis
     4583stops with P4: the recipe file may be used to halt the analysis at P2
     4584or P3.  In addition, the recipe file may be used to specify an
     4585alternative destination for the output results.  For example, to
     4586generate the photometric flat-field correction frame from a collection
     4587of dithered images, the user may not want the photometry results in
     4588the main DVO database.  By using the recipe to set an alternative DVO
     4589database target, and by specifying the use of the raw master flat
     4590rather than the corrected one, the analysis of the dithered images is
     4591kept isolated from the other photometry database results.  The
     4592resulting photometry may be used to construct the new, corrected
     4593flat-field images, and the processing of the same images using the new
     4594flat-field images may be sent to the master DVO database. 
     4595
     4596\subsection{Basic Detrend Creation}
     4597
     4598\begin{figure}
     4599\begin{center}
     4600\includegraphics[scale=0.85]{pics/ipptools.08.ps}
     4601\caption{\label{detrend} Detrend Creation Tasks}
     4602\end{center}
     4603\end{figure}
     4604
     4605Figure~\ref{detrend} illustrates the tables needed for the generic
     4606detrend construction process, using the flat-field construction as an
     4607example.  This diagram is somewhat more complex than the preceeding
     4608versions.  In this diagram, both single jobs and multiple jobs are
     4609represented by the process elements (the blue ellipses).  In some
     4610cases, more that one task will be needed to perform the function
     4611illustrated by a single process task.  The complexity of this diagram
     4612is enhanced by the need for multiple iterations and both single chip
     4613and full mosaic processing.  At the moment, the distinction between
     4614mosaic and single chip cameras is not specifically discussed.
     4615Finally, the triggers which initiate a specific detrend analysis are
     4616glossed over.
     4617
     4618The detrend analysis is initiated by choosing a type of detrend image
     4619to be constructed and by specifying the criteria which will be used to
     4620select the input raw detrend frames for the construction.  For
     4621example, these criteria could specify that all twilight flat images
     4622over a certain period of days, perhaps with restrictions on the flux
     4623levels or the time-from-sunset of the images.  The detrend analysis
     4624run is given an ID (det ID) which will also be used to identify the
     4625resulting master detrend frame. 
     4626
     4627Given the definition of a master detrend run, the input exposures are
     4628selected from the raw detrend exposure table, and written to the input
     4629detrend exposure table.  In the next step, the corresponding image
     4630files are selected from the table of raw image files.  Since there
     4631will be a different set of input raw images for each attempt at
     4632creating a master detrend image, and since any given attempt may use
     4633some of the same input images as any other attempt, a separate table
     4634of input raw images is constructed. 
     4635
     4636Each of the input raw images may be pre-processed before it may be
     4637used to construct the detrend frame.  For example, the input
     4638flat-field images should (probably) be dark- and bias-corrected before
     4639they are stacked.  The information about these input processed images
     4640is written to the input images table.  If no processing is needed,
     4641this step simply copies the appropriate information to the table, and
     4642points back to the raw image, rather than a processed version. 
     4643
     4644The input processed images are combined (stacked) to create a master
     4645detrend image for the particular data element defined by the image
     4646class (chip/cell/fpa).  At this stage, not all input images should
     4647necessarily be included in the stack.  If residual statistics have
     4648been measured for the input images (say, using a prior stack), then
     4649some of the input image may be excluded.  The table of residual images
     4650is used to guide this process.  The information describing the
     4651resulting master image is written to the master images table. 
     4652
     4653The statistics of the master detrend images must examined so that any
     4654necessary renormalizations may be performed.  For example, after
     4655stacking the individual flat images, the resulting stacks must be
     4656renomalized to account for the different ranges of input image fluxes.
     4657This analysis is least-squares solution in which an appropriate scale
     4658is determined for each input exposure and a separate gain is
     4659determined for each of the chips or cells in the camera.  This
     4660analysis can only performed after all image stacks (ie, for all chips)
     4661have been constructed.  The resulting information is written to the
     4662table of master detrend frames. 
     4663
     4664Once the master detrend is constructed, the master detrend images may
     4665be used to construct residual images for each of the input images.
     4666These residual statistics, as well as the locations of the residual
     4667images and other related data products (jpeg thumbnails?) are written
     4668to the residual image table.  Note the red arrow which by-passes the
     4669stack construction and merge steps and skips directly to the residual
     4670analysis.  In some cases, it may be useful to have the input images
     4671confronted with an existing detrend image, and the resulting residual
     4672values used to guide the rest of the process.  For example, in the
     4673flat-field analysis, applying an earlier flat can result in a very
     4674good first-pass rejection of poor input images.  The logic to make
     4675this leap must be part of PanTasks, since each of the individual
     4676blocks represent complete processing jobs.
     4677
     4678Finally, the residual statistics from the complete mosaic (all input
     4679images, all chips) are used to assess the quality of the newly
     4680constructed master detrend image, and to potentially modify the
     4681selection of input images.  This latter process is performed by
     4682marking the state of the residual images from this iteration.  The
     4683stacking process always examines the state information for the
     4684residual images from the previous iteration, if it exists, when
     4685constructing the master stack.  Once a master detrend frame has been
     4686judged of high enough quality, the state of the entry for the frame in
     4687the master detrend frames table is set to an appropriate value to tell
     4688the other routines that this image should be used as a master detrend.
     4689The exact choice of which master detrend frame is used for a given
     4690science image depends on the recipe along with information such as the
     4691time period used or the conditions used.
     4692
     4693Note that, although this discussion focuses on the construction of
     4694flat-field images, the same structure should be capable of
     4695constructing the biases, dark, fringes, etc.  In some cases, as noted
     4696above, the `process' stage is a null operation.
     4697
     4698\begin{figure}
     4699\begin{center}
     4700\includegraphics[scale=0.85]{pics/ipptools.09.ps}
     4701\caption{\label{detprocess} Detrend Creation : Process Tasks}
     4702\end{center}
     4703\end{figure}
     4704
     4705\begin{figure}
     4706\begin{center}
     4707\includegraphics[scale=0.85]{pics/ipptools.10.ps}
     4708\caption{\label{detresid} Detrend Creation : Residual Tasks}
     4709\end{center}
     4710\end{figure}
     4711
     4712\begin{figure}
     4713\begin{center}
     4714\includegraphics[scale=0.85]{pics/ipptools.11.ps}
     4715\caption{\label{detstack} Detrend Creation : Stack and Norm}
     4716\end{center}
     4717\end{figure}
    32564718
    32574719\section{Interfaces}
Note: See TracChangeset for help on using the changeset viewer.